Creating a Learning Profile by Using Face and Emotion Recognition

The aim of this work is to employ face recognition for creating learning profiles of the analysed persons who are students in this study. Generating education profiles will help experts in the diagnosis of Attention Deficit Hyperactivity Disorder (ADHD), which is a serious problem in children. Children with ADHD often have the ability and potential to learn. However, it may be difficult to reveal their capabilities and skills. Accordingly, a suffering child may have a hard time succeeding in real life when he/she is ignored and expected to mix with other children. The unrealized gap and deficiency may lead to other problems and more complicated situation with unpredictable consequences. Thanks to the system developed in this study, and the like, which will help in diagnosing the ADHD disease, and hence suffering individuals will be able to recognize their deficiencies, understand their ability to learn and adapt when approached differently in a way which suits his/her situation. This personalized handling of infected students will be an excellent guide to advance their potential and integration within the society carefully and smoothly. The system analyzes the face of a student to inspire his/her emotional state. The reported test results demonstrate how the system works well and produces high accuracy under a variety of severe conditions such as skewed angle, less illumination, accessories etc.


I. INTRODUCTION
One of the most important aspects in the education system is recognizing the potential and skills of a student to caste his/her focus and mentoring as well as set the expectations from the student accordingly.Many people are unaware of their abilities due to the misguidance from the education system which is classical in most countries and favors unproblematic students, leaving out students who might excel once they receive little reasonable attention.Indeed, many gifted children may fail because they cannot timely and precisely discover their potential and interests; this leads to losing self confidence and deviating in unpleasant directions.
In fact, it is hard for an individual to find out what he/she could be good at in real life, especially if the education system does not involve enough experts who are capable of realizing the skills, potential and deficiencies of every student to put him/her on the right path for success which matches his/her potential and skills the best.However, having a system which monitors and evaluates a person while attending and following the lessons will also help the person to know self capabilities and deficiencies.This will also help in identifying topics which attract the attention of the student more than others.Teachers who know their students' potential and preferred skills, and parents who understand their children are able to spot their challenges and assist them in developing and expanding their knowledge.The situation is the same for adults.Knowing a person's personal learning profile will help him/her attain greater success, and hence it will be possible to benefit from each individual the best way where he/she fits and feels more comfortable to produce and contribute; rather than forcing people to occupy positions where they will be less productive with all the consequences on personality, mostly leading to failure with all the associated loses which may be hard to recover and catch up to resume productivity.
Children with ADHD often have the ability to grasp and master a topic when approached the right way.However, because this deficiency can be difficult to identify with high accuracy and even to get the parents to accept such a hard situation, neglected suffering children may struggle to achieve in real life.As a result they will lose the opportunity to contribute to the community and demonstrate their unique potential.Equally important, the community will lose.Motivated by the need to identify children with special needs, we developed an automated system which depends on emotion detection from facial expressions to spot potentially problematic persons who may need further attention and follow up in order to caste, improve and direct their focus along the right path leading to success.The system has been tested using videos from a school environment and the achieved results reported in this paper demonstrate its applicability and effectiveness.The reported results are limited as dictated by the restriction to share more details related the videos of the participating children.
The rest of this paper is organized as follows.Section 2 covers the related work.Section 3 includes the method.Section 4, reports the results.Section 5 discusses the results with all the associated aspects.Section 6 is conclusions.

II. RELATED WORK A. Face Recognition
The study by Zhao and Ye (2010) [1] criticised the usage of "username + password" as static identity verification in distance education systems leading to low reliability as a main disadvantage of distance education.Facial recognition certification methods adopted in this research, based on the two-dimensional hidden Markov model (P2D-HMM) ensure that distance learning is safe and righteous to support the development of distance education.Sun, Pei and Zhou (2008) [2] realized networking and multimedia as the development trend of modern educational technology.Based on this, some programs have been offered by the distance education system for years.One of the aims of this project is to investigate whether students who receive distance education are satisfied with this environment.In this study, to predict the affective status of students in the modern distance education mechanism, a distance education system based on affective computing which takes facial features as input has been proposed and implemented.Rekha and Ramaprased (2017) [3] automated the attendance system by combining Matlab GUI and face recognition technology using the Eigen Face database and PCA algorithm.
Lukas, et al. (2016) [4] realized authentication as one of the important technologies in information systems.They proposed a method for confirming student attendance in the classroom using facial recognition by combining Discrete Wavelet Transforms (DWT) and Discrete Cosine Transform (DCT) to obtain the characteristics of the face of the student.In the experiments where 16 students participated in the classroom setting, In the process, 121 of 148 effective programs for face recognition were completed.Rougeaux (2020) [5] developed a method for defining the position of at least one eyeball within an image.The method includes the following steps: (a) capturing a time series of image frames illuminated in a predetermined temporal manner by at least two spaced-apart light sources by at least one imaging sensor; (b) processing image frames to determine their specular reflection locations; and (c) operating the time-series evolution of the location of the specular reflections to isolate corneal reflections from the determined specular reflection locations.Zhang et al. (2004) [6] presented a new approach for facial recognition by increasing the classifiers based on statistical local features.The face image is scanned with a scalable subwindow from which Local Binary Model (LBP) histograms are obtained to identify local features of a face image.The AdaBoost algorithm is used to distinguish pattern histograms for classification and to learn the similarity of each facial image pair.As a result of the resulting tests for the proposed method, a recognition rate of 97.9% was obtained.Hampapur, et al. (2003) [7] used computer-controlled pan-tilt-zoom cameras operated by a 3-dimensional wide basic stereo monitoring system.A computer face recognition system was used to detect individuals, and 3D location tracking was used to determine locations of individuals.Yang, et al. (2004) [8] stated that face screening based on Gabor features received considerable interest and success.However, Gabor features adopted by most systems today are unnecessary and high dimensional.This article proposes both a low-dimensional and distinctive facial recognition method using AdaBoost Gabor features.
Dedeoglu, Kanade and August (2004) [9] offered a solution to the issue of a very high zoom factor of a human face video.A graphic model that encodes (1) spatial-temporal consistencies, and (2) image creation and distortion processes was used to formulate the task of hallucination and samplingbased learning.He et al. (2005) [10] studied a face recognition method which is based on appearance called Laplacian Face approach.Using Preserve Locality Projections (LPP), face images are mapped to a face subfield for analysis.Dalal and Triggs (2005) [11] proposed a solution to the problem of feature sets for robust visual object recognition by adopting linear SVM-based human perception as a test case.The conducted experiments demonstrated that Directed Gradient Histograms (HOG) Guidelines are substantially higher than the current feature sets for human perception after analysis of the available edge and gradient-based descriptors; this has been shown to produce good results.Kumar, Banerjee and Vemuri (2009) [12] presented a new face classification system in which face images are represented as spatial arrangements of image patches.They investigated a smooth nonlinear functional mapping for the respective patches so that patches of the same face are close to each other.This was achieved using Volterra cores, which can produce successively better approximations for a non-linear smooth functionality.
Qiang et al. (2022) [28] recently provided a detailed survey paper on face recognition, steps of face recognition, tools and applications that were used in disease analysis based on facial feature recognition.They discussed the existing works on metabolic, genetic, neuromuscular and other diseases that used face recognition technology for detecting the disease and/or disease progress and listed possible future scopes of face recognition systems in various disease detection and monitoring.Du et al. (2022) [29] also discussed a literature review on face recognition focusing on a complete end-toend face recognition system with deep learning algorithms.They discussed the face detection, alignment, representation and various methods used in these processes with their scopes and limitations.[13] proposed a new approach for facial representation and recognition by analyzing images by considering size and orientation together.Face images are first divided into different scales and orientation responses by integrating multi-scale and flexible Gabor filters.Local binary pattern analysis is subsequently used to characterize neighboring relationships not only in the field of the view, but also on different scales and responses to orientation.552 Danielson, et al. (2018) [14] determined that 23% of ADHD children in the United States could not receive treatment because they could not be diagnosed.Such a result from one of the most developed countries illustrates the challenges associated with the problem.Indeed, ADHD is not a trivial problem.It requires sophisticated systems capable of producing stable and accurate results.Thus, the need for the system developed in this study.

Lei et al. (2010)
A study from Tel Aviv University showed that there is a direct correlation between ADHD and inability to suppress involuntary eye movement and face recognition [15].Though might be recognized as an interesting discovery, this needs further validation on a variety of cohorts to demonstrate its wide applicability and robustness.Wenstrand and Sachs (2006) [16] described a system which includes an image and eyes detection processes that work together to detect the light received through person's eyes.
Dan (2020) [30] compared the emotional facial expression of adolescents diagnosed with ADHD to other adolescents without ADHD by combining existing researches.They compared the results of neutral, happy, sad, fearful and angry facial expressions of 10 to 19 years old people with and without ADHD showing how the neural studies differed from the facial expression ones.Levantini et al. (2020) [31] provided a review on ADHD detection from tracking eye movement.They discussed how eye fixation, dwell, saccade, anti-saccade, smooth pursuit, pupil diameter, and blink tracking of people performing specific tasks were helpful detecting ADHD symptoms in children.Their investigations also provided the limitations of existing research works in this area, the possible scopes and challenges of ADHD detection from eye movements.Staff et al. (2021) [32] proposed a different type of study on Morphed Facial Emotion Recognition Task (MFERT) on both ADHD children and control group.They showed that the children with ADHD were able to accurately recognize emotions when the intensity of the emotions increased and discussed the possibilities of similar advanced studies to analyze the affects on children with ADHD.

C. Minimizing Error Rate in Face Recognition Method
For a developed system to be more trustworthy, it must recognize with high accuracy the face of an investigate kid, despite the surrounding difficulties, including illumination, deviation, etc.Researchers have developed several techniques which are capable of supporting this request.Kumar and Reddy (2012) [17] concentrated on solving the very low resolution (VLR) problem in face recognition.Face recognition methods from public face databases were used with feature extraction techniques based on relationship learning and Euclidean distance.Wagner et al. (2011) [18] addressed the issue of concurrently coping with shifts in lighting, image misalignment, and occlusion in test images.A conceptually simple face recognition system was proposed, with high robustness and stability against illumination changes, image misalignment, and partial occlusion.The method aligns a test face picture to a sequence of pre-training images using sparse representation methods.
Yin and Liu (2017) [19] concentrated on Multitasking learning (MTL) for facial recognition.They proposed a multitasking convolutional neural network (CNN), with identity classification as the primary task and pose, illumination, and expression (PIE) predictions as side projects.Wang, et al. (2014) [20] discussed low resolution face recognition (LR FR), various exposures, illumination, expression, and other topics .The aim is to recognize faces in low-resolution or low-quality photographs.The definition outline, device design, and process categorization are all included in the summary of (LR.FR).

D. Emotion Recognition
Emotion recognition is an important aspect to consider in identifying persons suffering from ADHD.Thus, several relevant systems for emotion detection have been investigated.For instance, Tarnowski et al. (2017) [21] described the results of recognizing seven emotional states (neutral, joy, sorrow, surprise, rage, fear, and disgust) based on facial expressions.Features were determined for a 3D dimensional face model.The k-NN (k-nearest neighbor classifier) and MLP (two-layer neural network classifier) networks were used to categorize the features.
Dubey and Singh (2016) [22] provided an overview of facial expression recognition's uses and applications.They covered the basics of the facial emotion recognition system, as well as its applications, a comparison of common facial expression recognition techniques, and stages of an automatic facial expression recognition system.Schurgin, et al. (2014) [23] realized that some areas of the face contain more useful information than others when determining whether a face shows a specific emotion.When analyzing various emotions, the authors investigated whether people differently utilize different areas of the face to express emotions.Still to be investigated further is the link between emotions and each of ethnicity and cultural aspects.The connection is anticipated to be strong and hence we need to dig out further and extend this study to take the discoveries into account.
Khan (2023) [33] reviewed the emotion recognition researches from last decade that used machine learning and deep learning methods starting from data pre-processing, feature extraction and finally classification.

E. Learning Profile
Learning profile is not a stable aspect.It differs even for the same person based on a variety of factors.For instance, a student may reacts positively to a subject that he/she likes, while the same student may not reflect high confidence and concentration when exposed to a subject of no interest.Accordingly, it is essential not to judge a student based on a single perspective.Rather, multiple perspectives, various environments, different crowds, different time periods, etc. should be utilized when a student is to be subject to the process of learning profile extraction.
Barnard-Brak, Paton and Lan (2010) [24] based their study on the existence of some speculation that students have different profiles of self-regulated learning habits.These profiles, on the other hand, tend to be linked to substantially different academic outcomes.The aim of the study of Barnard-Brak, Paton and Lan was to find out whether there exist learner profiles for self-regulated learning skills and strategies.
Zaina and Bressan (2008) [25] demonstrated the design of a framework that evaluates learning profiles based on student preference categories.Felder-Silverman Learning Style Model is used to build categories.A relationship has been formed between the observed and the learning objects used to automatically construct learning scenarios based on learning profiles of students.

III. METHOD A. Image Processing, Face and Eye Detection
The most important and foremost part of face recognition is face detection.The method of face detection in images is complicated due to variations across human faces, too many different parameters such as posture, speech, location and orientation, skin tone, the presence of glasses or facial hair, differences in camera gain, lighting conditions, and image resolution.This technique can be called an object detection system and can be used in many fields, such as protection, biometrics, law enforcement, entertainment, personal safety, etc. Face detection takes place in two separate stages.The first step is to classify whether images contain faces.This takes a binary value of yes or no for any arbitrary image as an input and an output indicating whether there are any faces present in the image.A neural network has been implemented and trained to classify images as faces or not faces.The second step is to locate the face.This takes an image as input and the output is a bounding box with the position of any face or faces within that image (specifically: x, y, width, height).
A data set was created by collecting various pictures from online public platforms.These pictures are generally composed of photographs of students during online education.Later, after various analyses of these photographs, some methods were applied on them for facial despatch.
There are many different methods for face recognition.These methods have been researched and analyzed to find the appropriate method for this project.As a result of the conducted study, the OpenCV method, which is the most suitable method for the purpose, was used.OpenCV (Open Source Computer Vision Library) is an open source software library for computer vision and machine learning, designed to provide a shared infrastructure for applications for computer vision and speed up the use of machine perception in consumer products.The OpenCV library, which is used in many different software languages, was used with the python software in this project.Besides this library, we used and installed other libraries.This part was very important to improve the algorithm.The considered libraries include dlib, face recognition libraries.

OpenCV (Open Source Computer Vision
) is an open-source image manipulation library.It is a platform-independent library, so it can run on Windows, Linux, FreeBSD, Android, Mac OS, and iOS platforms.OpenCV applications can be easily developed with Perl and Ruby programming languages, C ++, C, Python, Java, Matlab, EmguCV library, and different wrappers developed by Visual Basic.Net, and Visual C ++ languages and communities.
There are more than 2,500 algorithms for image processing and machine learning.With these algorithms, operations such as face recognition, distinguishing objects, detecting human movements, object classification, license plate recognition, processing on three-dimensional images, image comparison, optical character recognition OCR (Optical Character Recognition) can be performed easily.

C. Face Recognition
Training the system to be capable of face detection is the most crucial phase in the face recognition procedure.Faces of people can be collected and examined using a camera or from photographs in a face recognition system.The method evaluates the distance between the eyes, nose, mouth, and jaw margins, as well as the complete face anatomy.When the user approaches the camera, these measurements are saved in a database and utilized for comparisons.The margin of error in comparisons is minimized since each face has unique traits.The human face has about 80 nodes.
With facial recognition technology, the distance between the eyes, the width of the nose, the depth of the eye sockets, the 554 shape of the cheekbones, the length of the jawline, etc. are measured.The algorithm is focused on the golden triangle between the lips and temples.This region will never be altered, regardless of weight gain, aging facial characteristics, hair and beard development, or the use of spectacles.First of all, the contours of the faces are trained with the algorithm created, and then that face is recognized in seconds from the camera.There are various parameters such as angle, light, accessory, aging, weight gain that affects face recognition algorithms.While preparing the algorithm, these parameters should be taken into account in order to minimize the margin of error.
Facial features are structured in the algorithm where physical or geometric approaches are used.It is based on different classifications according to their first position on the face such as the eye, nose, and mouth, their distance from each other, and their angles when comparing the exposure photograph obtained initially after the face is detected.This strategy is based on the recognition of landmarks in variations that may result from any shadows, changes in illumination, or exposures.In addition, in the view-based method, the entire face template is utilized.

D. The Haar-Like features algorithm
The Haar-Like features algorithm was used in the next step to locate human faces in a frame or image.This algorithm helped us because some of the common characteristics of the human face are shared by all human faces, just as the eye area is darker than neighbouring pixels and the nose region is brighter than the eyes area.
The coordinates x, y, w, h that form a rectangular box are given in the image to display the location of the face or the region of interest in the image.Since this method trains on the face successfully, it is one of the methods used to reduce the error margin.

E. Emotion Recognition:
Given how difficult it is for humans to evaluate psychological conditions such as facial expression, it is simple to imagine how difficult it is for a computer to do this.However, a face may have many expressions at the same moment.For instance, there are occasions where the facial expression when smiling might be upsetting.Furthermore, the intricacy of the situation can be better grasped by analyzing the variances in these claims amongst persons.Therefore, the smallest facial expressions and details are very important.
Computer recognition of facial expression includes two basic stages: face detection from the picture and evaluation of the facial expression.The first stage, recognizing a face in a photograph, is commonly accomplished using a more advanced artificial neural network technique.When evaluating the psychological condition of the face subsequently, "dual local samples operator" can be employed for facial expression, for example, "Chi square statistics" and distance criterion.

IV. RESULT
According to the information obtained as a result of the analyzed studies, it was determined that there are many parameters in face detection, face recognition, and emotion recognition technology.These include ambient light, facial aging, facial measurements, individuality, mobility, etc.These parameters evaluated very well according to the data set to be used.As a result of the studies, after the image data analysis, image processing, face recognition, eye detection, data set creation, face recognition and emotion recognition processes were performed.Finally, these analyses were tested on online course records.The results were recorded and successful analysis results were obtained.
Figure 1 shows a sample face detection result where a person is in dark and wearing eye glasses.Figure 2 shows boundary boxes for both face and eyes in red and green respectively.A sample set of images created with the Haar Cascade model for face recognition is shown in Figure 3. Figure 4 shows a sample result of recognizing a face correctly even if accessories were present.
Shown in Figure 5 are the results obtained in each emotion change of the 40-minute lesson.In line with these results, all the emotions of a student were identified and analyzed.Finally, there is a personalized analysis in Figure 5.These analyses can be used in the diagnosis of ADHD with the help of an expert.For example, a student in Figure 6 had a lot of emotional changes during a lesson, which may be one of the symptoms of ADHD.The aim of this project is to create learning profiles of people using facial recognition methods and also to create data for experts to understand whether they have disorders such as ADHD (Attentionm Deficit Hyperactivity Disorder).Face recognition and emotion recognition will be used for this purpose and will be analyzed at the end.The first step for these studies is face detection.Face detection in images is complicated due to the variations between human faces, exposure, posture, location and orientation, skin tone, presence of glasses or facial hair, differences in camera gain, lighting conditions, and image resolution.
For face detection and eye detection, OpenCV with python were used; this combination forms the most suitable method for the purpose.OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine 555 learning software library with the goal of establishing a shared infrastructure for computer vision applications and accelerating the adoption of machine perception in consumer goods.This strategy was chosen because it offers far too many benefits for the project.When we consider the benefits of OpenCV, it provides access to over 2,500 cutting-edge and traditional algorithms.It features a vast user base, allowing users to ask for advice and aid other creators.OpenCV was created to provide algorithmic efficiency for real-time program processing, as well as to take advantage of hardware acceleration and multi-core system deployment.However, there are several downsides to using this library.Given these drawbacks, it may be stated to be slower in image processing techniques than Matlab, which might cause some handling issues.Another problem is that OpenCV is written in C, thus it must release memory every time it is allocated.
The utilized version of the algorithm was downloaded from various libraries such as dlib, face recognition.In the next step, the Haar-Like features algorithm was used, which is the algorithm used to locate human faces in a frame or image.During detection, the Haar-Like features pass the window on an image and do the convolution operation with the filters to find out if the features are in the image.The Face recognition method started after image processing, face detection, eye detection, and the training algorithm.After a detailed literature review and research for this stage, an algorithm was developed in Python.In this algorithm using OpenCV, face training was The results of these algorithms can be seen on the created dataset.When checking the result, we clearly can say the algorithm worked successfully on the dataset which is created at the beginning.As a result of the studies mentioned above, five different algorithms have been created.The first of these is face detection on the photos in the dataset created at the beginning and the other is face detection made live on the webcam.The important aspect in face detection is how accurate the prepared method is.Some methods can have a lot of errors.Light, angle, scale, background, expression, position, orientation, skin color, gender, age, pixel values, accessories such as glasses, and facial hair, etc. are affected by many different parameters.Deep learning is used to ensure that these parameters do not affect face detection.
As a result of all these studies, online course registrations were obtained in order to create a demo of the project.And these algorithms invoked on these online course recordings have been tested.The emotion recognition program has worked successfully on videos.Even in very long videos, the changes in people's faces within seconds are analyzed and recorded live.

VI. CONCLUSION
The aim of this work is to employ face recognition for creating learning profiles of the analyzed persons who are students in this study.Generating educational profiles will help experts in the diagnosis of Attention Deficit Hyperactivity Disorder (ADHD), which is a concerning issue in children.Face recognition and emotion recognition technology is developing day by day.With this study, it is aimed to use and develop these applications in important areas and to increase the accuracy rate.As a result, analyses that can be used by experts were obtained.In order to minimize the margin of error, all phases were modeled from the beginning, face and eye detection models were performed before face recognition modeling was performed.The effect of external factors such as bad angle, light and accessories on the margin of error is minimized.These models have also been tested on real human face data.Personal data protection ethical permissions have been obtained in the use of this data.
He discussed both 2D and 3D facial emotion detections with the existing datasets, the techniques used, performances of the existing models, and comparisons between different emotion detection techniques including SVM, KNN, LDA, HMM, CNN, LSTM and their variations to classify 2 to 23 emotion classes.The performance scores clearly showed that the deep learning models performed better in facial emotion recognition.Naga et al. (2023) [34] also provided a literature review on facial emotion detection starting with discussing 11 popular datasets, different steps of facial emotion recognition, existing technologies, tool, and applications, and concluded with the comparisons of existing methods and current challenges in the research domain.Most recent facial emotion detection researches are based on CNNbased models and their variations for more accurate emotion recognition tasks.Sarvakar et al. (2023) [35] proposed a CNN-based facial emotion detection model and compared the 553 performances with decision tree, simple CNN and feedforward NN on two datasets-CK+ and SFEW for a 7 class emotion classification task.The CNN had six 2D convolutional layers, two max pooling layers and two fully connected layers.The proposed CNN achieved 54% accuracy outperforming the other three models by 23% to 36%.Recently, Karatay et al. (2022 & 2023) [26] [27] proposed CNN-LSTM and CNN-Transformer based facial emotion recognition model to detect emotions from FABO and CK+ datasets.After extracting face and body keypoints using OpenPose tool, they applied basic pre-processing to feed the pre-processed data into a CNN model.The CNN outputs were then used in a LSTM and a transformer model to generate the final emotion classification.Different variations of keypoints, models, and other characteristics showed 83% to 100% accuracy in emotion detection showing the efficiency of various combinations of features and models.

Fig. 1 .
Fig. 1. Results for detection in the dark and with glasses

Fig. 4 .
Fig. 4. Successful Face recognition result for people using accessory