Abstract
Machines would require the ability to perceive and adapt to affects for achieving artificial sociability. Most autonomous systems use Automated Facial Expression Classification (AFEC) and Automated Affect Interpretation (AAI) to achieve sociability. Varying lighting conditions, occlusion, and control over physiognomy can influence the real life performance of vision-based AFEC systems. Physiological signals provide complementary information for AFEC and AAI. We employed transient facial thermal features for AFEC and AAI. Infrared thermal images with participants' normal expression and intentional expressions of happiness, sadness, disgust, and fear were captured. Facial points that undergo significant thermal changes with a change in expression termed as Facial Thermal Feature Points (FTFPs) were identified. Discriminant analysis was invoked on principal components derived from the Thermal Intensity Values (TIVs) recorded at the FTFPs. The cross-validation and person-independent classification respectively resulted in 66.28% and 56.0% success rates. Classification significance tests suggest that (1) like other physiological cues, facial skin temperature also provides useful information about affective states and their facial expression; (2) patterns of facial skin temperature variation can complement other cues for AFEC and AAI; and (3) infrared thermal imaging may help achieve artificial sociability in robots and autonomous systems.
- Baldwin, J. F., Case, S. J., and Martin, T. P. 1998. Machine interpretation of facial expressions. BT Technol. J. 16, 156--164. Google Scholar
- Bartneck, C. 2001. How convincing is Mr. Data's smile: Affective expressions of machines. User Model. User-Adapt. Interact. 11, 279--295. Google Scholar
- Belmont Report. 1979. Ethical principles and guidelines for the protection of human subjects of research. United States Government, National Institute of Health, Washington D.C., http://ohsr.od.nih.gov/guidelines/belmont.html.Google Scholar
- Bolle, R. M., Connell, J. H., Pankanti, S., Ratha, N. K., and Senior A. W. 2004. Guide to Biometrics. Springer, Verlag, Berlin, Germany. Google Scholar
- Boulic, L. E. R. and Thalmann, D. 1998. Interacting with virtual humans through body actions. IEEE Computer Graphics and Applications 98, 8--11. Google Scholar
- Brooks, R. A. 2002. Flesh and Machines: How Robots Will Change Us. Pantheon Books, New York, NY. Google Scholar
- Cantronic Systems Inc. 2001. IR 860 User Manual. Cantronic Systems Inc., Coquitlam, Canada.Google Scholar
- Cantronix Systems Inc. 2003. IR 860 Product Data Sheet. Cantronic Systems Inc., Coquitlam, Canada.Google Scholar
- Cohen, I., Sebe, N., Garg, A., Chen, L. S. and Huang, T. S. 2003. Facial expression recognition from video sequences: temporal and static modeling. Comput. Vision Image Understand. 91, 160--187. Google Scholar
- Cohen, J. 1977. Statistical Power Analysis for Behavioral Sciences. Academic Press, New York.Google Scholar
- Costanza, M. C. and Affifi, A. A. 1979. Comparison of stopping rules in forward stepwise discriminant analysis. J. Amer. Statist. Assoc. 74, 777--785.Google Scholar
- Critchley, D., Daly, E. M., Bullmore, E. T., Williams, S. C. R., Amelsvoort, T. V., Robertson, D. M., Rowe, A., Phillips, M., Mcalonan, G., Howllin, P., and Murphy, D. G. M. 2000. The functional neuroanantomy of social behavior: changes in cerebral blood flow when people with autistic disorder process facial expressions. Brain 123, 203--212.Google Scholar
- Dautenhahn, K. 1995. Getting to know each other---Artificial social intelligence for autonomous robots. Robotics Autonom. Syst. 16, 333--356.Google Scholar
- Dhew. 1979. Department of Health, Education and Welfare Publication No. (OS) 78-0014. US Government Printing Office, Washington D.C.Google Scholar
- Donato, G., Bartlett, M. S., Hager J. C., Ekman P. and Sejnowski, T. J. 1999. Classifying facial expressions. IEEE Trans. Patt. Anal. Machine Intell. 21, 974--989. Google Scholar
- Duda, R. O., Hart, P. E. and Stork, D. G. 2001. Pattern Classification. Wiley Interscience, New York. Google Scholar
- Ekman, P. 1982. Emotion in the Human Face. Cambridge University Press, Cambridge, UK.Google Scholar
- Ekman, P. 1993. Facial expression and emotion. Amer. Psychol. 48, 384--392.Google Scholar
- Ekman, P. and Friesen, W. V. 1978. Facial Action Coding System: A Technique for the Measurement off Facial Movement. Consulting Psychology Press, Mountain View, CA.Google Scholar
- Essa, I. and Pentland, A. 1997. Coding, analysis, interpretation and recognition of facial expressions. IEEE Trans. Patt. Anal. Machine Intell. 19, 757--763. Google Scholar
- Eveland, C. K., Socolinsky, D. A., and Wolf, L. B. 2003. Tracking human faces in infrared video. Image Vision Comput. 21, 579--590.Google Scholar
- Fasel, B. and Luettin, J. 2003. Automatic facial expression analysis: a survey. Patt. Recognit. 36, 259--275.Google Scholar
- Field, A. 2000. Discovering Statistics Using SPSS for Windows. Sage Publications, London, UK. Google Scholar
- Gao, Y., Leung, M. K. H., Hui, S. C., and Tananda, M. W. 2003. Facial expression recognition form line-based caricatures. IEEE Trans. Syst. Man Cyber. 33, 407--412. Google Scholar
- George, D. and Mallery, P. 1995. SPSS/PC+ Step by Step: A Simple Guide and Reference. Wadsworth Publishing, London, UK. Google Scholar
- Huang, C. and Huang, Y. 1997. Facial expression recognition using model-based feature extraction and action parameters classification. J. Visual Comm. Image Represent. 8, 278-- 290.Google Scholar
- Huberty, C. J. 1984. Issues in the use and interpretation of discriminant analysis. Psycholog. Bull. 95, 156--171.Google Scholar
- Huberty, C. J. 1994. Applied Discriminant Analysis. Wiley, New York, NY.Google Scholar
- Jones, C. H., Ring, E. F. J., and Clark, R. P. 1988. Medical thermography. In Applications of Thermal Imaging, Burnay, S. G., Williams, T. L. and Jones, C. H. N., Eds. Adam Hilger, Bristol, UK, 156--187.Google Scholar
- Kearney, G. D. and Mckenzie, S. 1993. Machine interpretation of emotion: Design of memory-based expert systems for interpreting facial expressions in terms of signaled emotions. Cognitive Science 17, 589--622.Google Scholar
- Khan, M. M., Ward, R. D., and Ingleby, M. 2004. Automated classification and recognition of facial expressions. In Proceedings of the IEEE Conference on Cybernetics and Intelligent Systems, Singapore, (Dec), 202--206.Google Scholar
- Khan, M. M., Ward, R. D., and Ingleby, M. 2005. The distinguishing facial expressions by thermal imaging using facial thermal feature points. In Proceedings of the 19th British HCI Group Annual Conference (HCI'05), Edinburgh, (Sept), L. Mackinnon, O. Bertelsen and N. Bryan-Kinns Eds. The British Computer Society, London, UK. 10--14.Google Scholar
- Kim, H. K., Bang, S. W., and Kim S. R. 2004. Emotion recognition system using short-term monitoring of physiological signals. Medical Biological Engin. Comput. 42, 419--427.Google Scholar
- Klein, J., Moon, Y., and Picard, R. W. 2002. This computer responds to user frustration: Theory, design and results. Interact. Comput. 14, 119--140.Google Scholar
- Kurse, P. W. 2001. Uncooled Thermal Imaging: Analysis, Systems and Applications. SPIE Press, Bellingham, WA.Google Scholar
- Lisetti, C. S. and Schiano, D. J. 2000. Automatic facial expression interpretation: Where human-computer interaction, artificial intelligence and cognitive science intersect. Pragmatics Cognition 8, 185--235.Google Scholar
- Matsuzaki, H. and Mizote, M. 1996. Measurement of facial temperature fluctuations by thermal image analysis. Progress in Biophysics and Molecular Biology 65 Supplement 1, 185-- 186.Google Scholar
- Mcgimpsey, J. G., Vaidya, A., Biagioni, P. A., and Lamey, P.-J. 2000. Role of thermography is the assessment of infraorbital nerve injury after malar fractures. British J. Oral Maxillofacial Surg. 38, 581--584.Google Scholar
- Nanavati, S., Thieme, M., Nanavati, R. 2002. Biometrics: Identity Verification in a Networked World. John Wiley & Sons, New York, NY. Google Scholar
- Norman, D. A., Ortony, A., and Russell, D. M. 2003. Affect and machine design: Lessons for the development of autonomous machines. IBM Syst. J. 42, 38--44. Google Scholar
- Ogasawara, T., Kitagawa, Y., Ogawa, T., Yamada, T., Kawamura, Y., and Sano, K. 2001. MR imaging and thermography of facial angioedema: A case report. Oral Surg. Oral Medicine, Oral Pathol. 92, 473--476.Google Scholar
- Pantic, M. and Rothkrantz, L. J. M. 2000. Automatic analysis of facial expressions: The state of the art. IEEE Trans. Patt. Anal. Machine Understand. 22, 1424--1445. Google Scholar
- Pavlidis, I. 2004. Lie detection using thermal imaging. In Proceedings of SPIE Thermosense XXVI (April), Orlando, FL, The International Society for Optical Engineering, Bellingham, WA, 270--279.Google Scholar
- Picard, R. W. 2000. Affective Computing. MIT Press, Cambridge, MA. Google Scholar
- Pizzagalli, D., Koenig, T., Regard, M., and Lehmann, D. 1998. Faces and emotion: brain electric field sources during covert emotional processing. Neuropsychologia 36, 323--332.Google Scholar
- Posamentier, M. T. and Abdi, H. 2003. Processing faces and facial expressions. Neuropsychology Rev. 13, 113--143.Google Scholar
- Prokoski, F. J. and Iedel, R. 1999. Infrared identification of faces and body parts. In Biometrics: Personal Identification in Networked Society. Jain, A. K., Bolle, R. M., and Pankanti, S., Eds. Kulwer Academic Press, Boston, MA, 191--212.Google Scholar
- Redford, P. 2000. Conference Report: Social goals and emotions. Psychologist 13, 290--291.Google Scholar
- Schwarz, S., Hofmann, M. H., Gutzen, C., Schlax S., and Emde, G. V. 2002. VIEWER: A program for visualizing, recording, and analyzing animal behavior. Comput. Methods Programs Biomedicine 67, 55--66.Google Scholar
- Sharma, S. 1996. Applied Multivariate Techniques. Wiley, New York, NY. Google Scholar
- Sugimoto, Y., Yoshitomi, Y., and Tomita, S. 2000. A method of detecting transitions of emotional states using a thermal facial image based on a synthesis of facial expressions. Robotics Autonom. Syst. 31, 147--160.Google Scholar
- Swets, D. and Weng, J. 1998. Using discriminant eigenfeatures for image retrieval. IEEE Trans. Patt. Anal. Machine Intell. 20, 39--51. Google Scholar
- Turner, J. R. and Thayer, J. F. 2001. Introduction to Variance Analysis. Sage Publications, London, UK.Google Scholar
- Ward, R. D. and Marsden, P. H. 2004. Affective computing: problems, reactions and intentions. Interacting with Computers 16, 707--713.Google Scholar
- Yoshitomi, Y., Kim, S. I., Kawano, T., and Kitazoe, T. 2000. Effects of sensor fusion for recognition of emotional states using voice, face image and thermal image of face. In Proceedings of the IEEE International Workshop on Robotics and Human Interactive Communication, Osaka, Japan, (Sept), 178--183.Google Scholar
Index Terms
Automated Facial Expression Classification and affect interpretation using infrared measurement of facial skin temperature variations
Recommendations
Classifying pretended and evoked facial expressions of positive and negative affective states using infrared measurement of skin temperature
Earlier researchers were able to extract the transient facial thermal features from thermal infrared images (TIRIs) to make binary distinctions between the expressions of affective states. However, effective human-computer interaction would require ...
Expression-invariant face recognition by facial expression transformations
In this paper, we present a method of expression-invariant face recognition that transforms input face image with an arbitrary expression into its corresponding neutral facial expression image. When a new face image with an arbitrary expression is ...
Face and Facial Expression Recognition
ICAART 2015: Proceedings of the International Conference on Agents and Artificial Intelligence - Volume 2Face and facial expression recognition is a broad research domain in machine learning domain. Non-negative
matrix factorization (NMF) is a very recent technique for data decomposition and image analysis. Here we
propose face identification system as ...






Comments