skip to main content
article

Automated Facial Expression Classification and affect interpretation using infrared measurement of facial skin temperature variations

Published:01 September 2006Publication History
Skip Abstract Section

Abstract

Machines would require the ability to perceive and adapt to affects for achieving artificial sociability. Most autonomous systems use Automated Facial Expression Classification (AFEC) and Automated Affect Interpretation (AAI) to achieve sociability. Varying lighting conditions, occlusion, and control over physiognomy can influence the real life performance of vision-based AFEC systems. Physiological signals provide complementary information for AFEC and AAI. We employed transient facial thermal features for AFEC and AAI. Infrared thermal images with participants' normal expression and intentional expressions of happiness, sadness, disgust, and fear were captured. Facial points that undergo significant thermal changes with a change in expression termed as Facial Thermal Feature Points (FTFPs) were identified. Discriminant analysis was invoked on principal components derived from the Thermal Intensity Values (TIVs) recorded at the FTFPs. The cross-validation and person-independent classification respectively resulted in 66.28% and 56.0% success rates. Classification significance tests suggest that (1) like other physiological cues, facial skin temperature also provides useful information about affective states and their facial expression; (2) patterns of facial skin temperature variation can complement other cues for AFEC and AAI; and (3) infrared thermal imaging may help achieve artificial sociability in robots and autonomous systems.

References

  1. Baldwin, J. F., Case, S. J., and Martin, T. P. 1998. Machine interpretation of facial expressions. BT Technol. J. 16, 156--164. Google ScholarGoogle Scholar
  2. Bartneck, C. 2001. How convincing is Mr. Data's smile: Affective expressions of machines. User Model. User-Adapt. Interact. 11, 279--295. Google ScholarGoogle Scholar
  3. Belmont Report. 1979. Ethical principles and guidelines for the protection of human subjects of research. United States Government, National Institute of Health, Washington D.C., http://ohsr.od.nih.gov/guidelines/belmont.html.Google ScholarGoogle Scholar
  4. Bolle, R. M., Connell, J. H., Pankanti, S., Ratha, N. K., and Senior A. W. 2004. Guide to Biometrics. Springer, Verlag, Berlin, Germany. Google ScholarGoogle Scholar
  5. Boulic, L. E. R. and Thalmann, D. 1998. Interacting with virtual humans through body actions. IEEE Computer Graphics and Applications 98, 8--11. Google ScholarGoogle Scholar
  6. Brooks, R. A. 2002. Flesh and Machines: How Robots Will Change Us. Pantheon Books, New York, NY. Google ScholarGoogle Scholar
  7. Cantronic Systems Inc. 2001. IR 860 User Manual. Cantronic Systems Inc., Coquitlam, Canada.Google ScholarGoogle Scholar
  8. Cantronix Systems Inc. 2003. IR 860 Product Data Sheet. Cantronic Systems Inc., Coquitlam, Canada.Google ScholarGoogle Scholar
  9. Cohen, I., Sebe, N., Garg, A., Chen, L. S. and Huang, T. S. 2003. Facial expression recognition from video sequences: temporal and static modeling. Comput. Vision Image Understand. 91, 160--187. Google ScholarGoogle Scholar
  10. Cohen, J. 1977. Statistical Power Analysis for Behavioral Sciences. Academic Press, New York.Google ScholarGoogle Scholar
  11. Costanza, M. C. and Affifi, A. A. 1979. Comparison of stopping rules in forward stepwise discriminant analysis. J. Amer. Statist. Assoc. 74, 777--785.Google ScholarGoogle Scholar
  12. Critchley, D., Daly, E. M., Bullmore, E. T., Williams, S. C. R., Amelsvoort, T. V., Robertson, D. M., Rowe, A., Phillips, M., Mcalonan, G., Howllin, P., and Murphy, D. G. M. 2000. The functional neuroanantomy of social behavior: changes in cerebral blood flow when people with autistic disorder process facial expressions. Brain 123, 203--212.Google ScholarGoogle Scholar
  13. Dautenhahn, K. 1995. Getting to know each other---Artificial social intelligence for autonomous robots. Robotics Autonom. Syst. 16, 333--356.Google ScholarGoogle Scholar
  14. Dhew. 1979. Department of Health, Education and Welfare Publication No. (OS) 78-0014. US Government Printing Office, Washington D.C.Google ScholarGoogle Scholar
  15. Donato, G., Bartlett, M. S., Hager J. C., Ekman P. and Sejnowski, T. J. 1999. Classifying facial expressions. IEEE Trans. Patt. Anal. Machine Intell. 21, 974--989. Google ScholarGoogle Scholar
  16. Duda, R. O., Hart, P. E. and Stork, D. G. 2001. Pattern Classification. Wiley Interscience, New York. Google ScholarGoogle Scholar
  17. Ekman, P. 1982. Emotion in the Human Face. Cambridge University Press, Cambridge, UK.Google ScholarGoogle Scholar
  18. Ekman, P. 1993. Facial expression and emotion. Amer. Psychol. 48, 384--392.Google ScholarGoogle Scholar
  19. Ekman, P. and Friesen, W. V. 1978. Facial Action Coding System: A Technique for the Measurement off Facial Movement. Consulting Psychology Press, Mountain View, CA.Google ScholarGoogle Scholar
  20. Essa, I. and Pentland, A. 1997. Coding, analysis, interpretation and recognition of facial expressions. IEEE Trans. Patt. Anal. Machine Intell. 19, 757--763. Google ScholarGoogle Scholar
  21. Eveland, C. K., Socolinsky, D. A., and Wolf, L. B. 2003. Tracking human faces in infrared video. Image Vision Comput. 21, 579--590.Google ScholarGoogle Scholar
  22. Fasel, B. and Luettin, J. 2003. Automatic facial expression analysis: a survey. Patt. Recognit. 36, 259--275.Google ScholarGoogle Scholar
  23. Field, A. 2000. Discovering Statistics Using SPSS for Windows. Sage Publications, London, UK. Google ScholarGoogle Scholar
  24. Gao, Y., Leung, M. K. H., Hui, S. C., and Tananda, M. W. 2003. Facial expression recognition form line-based caricatures. IEEE Trans. Syst. Man Cyber. 33, 407--412. Google ScholarGoogle Scholar
  25. George, D. and Mallery, P. 1995. SPSS/PC+ Step by Step: A Simple Guide and Reference. Wadsworth Publishing, London, UK. Google ScholarGoogle Scholar
  26. Huang, C. and Huang, Y. 1997. Facial expression recognition using model-based feature extraction and action parameters classification. J. Visual Comm. Image Represent. 8, 278-- 290.Google ScholarGoogle Scholar
  27. Huberty, C. J. 1984. Issues in the use and interpretation of discriminant analysis. Psycholog. Bull. 95, 156--171.Google ScholarGoogle Scholar
  28. Huberty, C. J. 1994. Applied Discriminant Analysis. Wiley, New York, NY.Google ScholarGoogle Scholar
  29. Jones, C. H., Ring, E. F. J., and Clark, R. P. 1988. Medical thermography. In Applications of Thermal Imaging, Burnay, S. G., Williams, T. L. and Jones, C. H. N., Eds. Adam Hilger, Bristol, UK, 156--187.Google ScholarGoogle Scholar
  30. Kearney, G. D. and Mckenzie, S. 1993. Machine interpretation of emotion: Design of memory-based expert systems for interpreting facial expressions in terms of signaled emotions. Cognitive Science 17, 589--622.Google ScholarGoogle Scholar
  31. Khan, M. M., Ward, R. D., and Ingleby, M. 2004. Automated classification and recognition of facial expressions. In Proceedings of the IEEE Conference on Cybernetics and Intelligent Systems, Singapore, (Dec), 202--206.Google ScholarGoogle Scholar
  32. Khan, M. M., Ward, R. D., and Ingleby, M. 2005. The distinguishing facial expressions by thermal imaging using facial thermal feature points. In Proceedings of the 19th British HCI Group Annual Conference (HCI'05), Edinburgh, (Sept), L. Mackinnon, O. Bertelsen and N. Bryan-Kinns Eds. The British Computer Society, London, UK. 10--14.Google ScholarGoogle Scholar
  33. Kim, H. K., Bang, S. W., and Kim S. R. 2004. Emotion recognition system using short-term monitoring of physiological signals. Medical Biological Engin. Comput. 42, 419--427.Google ScholarGoogle Scholar
  34. Klein, J., Moon, Y., and Picard, R. W. 2002. This computer responds to user frustration: Theory, design and results. Interact. Comput. 14, 119--140.Google ScholarGoogle Scholar
  35. Kurse, P. W. 2001. Uncooled Thermal Imaging: Analysis, Systems and Applications. SPIE Press, Bellingham, WA.Google ScholarGoogle Scholar
  36. Lisetti, C. S. and Schiano, D. J. 2000. Automatic facial expression interpretation: Where human-computer interaction, artificial intelligence and cognitive science intersect. Pragmatics Cognition 8, 185--235.Google ScholarGoogle Scholar
  37. Matsuzaki, H. and Mizote, M. 1996. Measurement of facial temperature fluctuations by thermal image analysis. Progress in Biophysics and Molecular Biology 65 Supplement 1, 185-- 186.Google ScholarGoogle Scholar
  38. Mcgimpsey, J. G., Vaidya, A., Biagioni, P. A., and Lamey, P.-J. 2000. Role of thermography is the assessment of infraorbital nerve injury after malar fractures. British J. Oral Maxillofacial Surg. 38, 581--584.Google ScholarGoogle Scholar
  39. Nanavati, S., Thieme, M., Nanavati, R. 2002. Biometrics: Identity Verification in a Networked World. John Wiley & Sons, New York, NY. Google ScholarGoogle Scholar
  40. Norman, D. A., Ortony, A., and Russell, D. M. 2003. Affect and machine design: Lessons for the development of autonomous machines. IBM Syst. J. 42, 38--44. Google ScholarGoogle Scholar
  41. Ogasawara, T., Kitagawa, Y., Ogawa, T., Yamada, T., Kawamura, Y., and Sano, K. 2001. MR imaging and thermography of facial angioedema: A case report. Oral Surg. Oral Medicine, Oral Pathol. 92, 473--476.Google ScholarGoogle Scholar
  42. Pantic, M. and Rothkrantz, L. J. M. 2000. Automatic analysis of facial expressions: The state of the art. IEEE Trans. Patt. Anal. Machine Understand. 22, 1424--1445. Google ScholarGoogle Scholar
  43. Pavlidis, I. 2004. Lie detection using thermal imaging. In Proceedings of SPIE Thermosense XXVI (April), Orlando, FL, The International Society for Optical Engineering, Bellingham, WA, 270--279.Google ScholarGoogle Scholar
  44. Picard, R. W. 2000. Affective Computing. MIT Press, Cambridge, MA. Google ScholarGoogle Scholar
  45. Pizzagalli, D., Koenig, T., Regard, M., and Lehmann, D. 1998. Faces and emotion: brain electric field sources during covert emotional processing. Neuropsychologia 36, 323--332.Google ScholarGoogle Scholar
  46. Posamentier, M. T. and Abdi, H. 2003. Processing faces and facial expressions. Neuropsychology Rev. 13, 113--143.Google ScholarGoogle Scholar
  47. Prokoski, F. J. and Iedel, R. 1999. Infrared identification of faces and body parts. In Biometrics: Personal Identification in Networked Society. Jain, A. K., Bolle, R. M., and Pankanti, S., Eds. Kulwer Academic Press, Boston, MA, 191--212.Google ScholarGoogle Scholar
  48. Redford, P. 2000. Conference Report: Social goals and emotions. Psychologist 13, 290--291.Google ScholarGoogle Scholar
  49. Schwarz, S., Hofmann, M. H., Gutzen, C., Schlax S., and Emde, G. V. 2002. VIEWER: A program for visualizing, recording, and analyzing animal behavior. Comput. Methods Programs Biomedicine 67, 55--66.Google ScholarGoogle Scholar
  50. Sharma, S. 1996. Applied Multivariate Techniques. Wiley, New York, NY. Google ScholarGoogle Scholar
  51. Sugimoto, Y., Yoshitomi, Y., and Tomita, S. 2000. A method of detecting transitions of emotional states using a thermal facial image based on a synthesis of facial expressions. Robotics Autonom. Syst. 31, 147--160.Google ScholarGoogle Scholar
  52. Swets, D. and Weng, J. 1998. Using discriminant eigenfeatures for image retrieval. IEEE Trans. Patt. Anal. Machine Intell. 20, 39--51. Google ScholarGoogle Scholar
  53. Turner, J. R. and Thayer, J. F. 2001. Introduction to Variance Analysis. Sage Publications, London, UK.Google ScholarGoogle Scholar
  54. Ward, R. D. and Marsden, P. H. 2004. Affective computing: problems, reactions and intentions. Interacting with Computers 16, 707--713.Google ScholarGoogle Scholar
  55. Yoshitomi, Y., Kim, S. I., Kawano, T., and Kitazoe, T. 2000. Effects of sensor fusion for recognition of emotional states using voice, face image and thermal image of face. In Proceedings of the IEEE International Workshop on Robotics and Human Interactive Communication, Osaka, Japan, (Sept), 178--183.Google ScholarGoogle Scholar

Index Terms

  1. Automated Facial Expression Classification and affect interpretation using infrared measurement of facial skin temperature variations

            Recommendations

            Comments

            Login options

            Check if you have access through your login credentials or your institution to get full access on this article.

            Sign in

            Full Access

            PDF Format

            View or Download as a PDF file.

            PDF

            eReader

            View online with eReader.

            eReader
            About Cookies On This Site

            We use cookies to ensure that we give you the best experience on our website.

            Learn more

            Got it!