skip to main content
research-article

A Deep Learning System for Recognizing Facial Expression in Real-Time

Published:05 June 2019Publication History
Skip Abstract Section

Abstract

This article presents an image-based real-time facial expression recognition system that is able to recognize the facial expressions of several subjects on a webcam at the same time. Our proposed methodology combines a supervised transfer learning strategy and a joint supervision method with center loss, which is crucial for facial tasks. A newly proposed Convolutional Neural Network (CNN) model, MobileNet, which has both accuracy and speed, is deployed in both offline and in a real-time framework that enables fast and accurate real-time output. Evaluations towards two publicly available datasets, JAFFE and CK+, are carried out respectively. The JAFFE dataset reaches an accuracy of 95.24%, while an accuracy of 96.92% is achieved on the 6-class CK+ dataset, which contains only the last frames of image sequences. At last, the average run-time cost for the recognition of the real-time implementation is around 3.57ms/frame on a NVIDIA Quadro K4200 GPU.

References

  1. Ognjen Rudovic, Jaeryoung Lee, Miles Dai, Bjorn Schuller, and Rosalind Picard. 2018. Personalized machine learning for robot perception of affect and engagement in autism therapy. Retrieved from arXiv preprint arXiv:1802.01186.Google ScholarGoogle Scholar
  2. Ying Qiu, Yang Liu, Juan Arteaga-Falconi, Haiwei Dong, and Abdulmotaleb El Saddik. 2019. EVM-CNN: Real-time contactless heart rate estimation from facial video. IEEE Trans. Multimedia (2019).Google ScholarGoogle Scholar
  3. Abdulmotaleb El Saddik. 2018. Digital twins: The convergence of multimedia technologies. IEEE MultiMedia 25, 2 (2018), 87--92.Google ScholarGoogle ScholarCross RefCross Ref
  4. Albert Mehrabian. 2008. Communication without words. Communication Theory, C. David Mortensen (Ed.). Transaction Publishers, New Brunswick, 193--200.Google ScholarGoogle Scholar
  5. Paul Ekman and Wallace V. Friesen. 2003. Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues. ISHK.Google ScholarGoogle Scholar
  6. Ligang Zhang and Dian Tjondronegoro. 2011. Facial expression recognition using facial movement features. IEEE Trans. Affect. Comput. 2, 4 (2011), 219--229. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Zhengyou Zhang, Michael Lyons, Michael Schuster, and Shigeru Akamatsu. 1998. Comparison between geometry-based and Gabor-wavelets-based facial expression recognition using multi-layer perceptron. In Proceedings of the 3rd International Conference on Face 8 Gesture Recognition. 454--459. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Hong-Bo Deng, Lian-Wen Jin, Li-Xin Zhen, Jian-Cheng Huang.2005. A new facial expression recognition method based on local Gabor filter bank and PCA plus LDA. Int. J. Inform. Technol. 11, 11 (2005), 86--96.Google ScholarGoogle Scholar
  9. Feifei Zhang, Qirong Mao, Xiangjun Shen, Yongzhao Zhan, and Ming Dong. 2018. Spatially coherent feature learning for pose-invariant facial expression recognition. ACM Trans. Multimedia Comput., Commun. Appl. 14, 1s (2018), 27. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Shu Liao, Wei Fan, Albert C. S. Chung, and Dit-Yan Yeung. 2006. Facial expression recognition using advanced local binary patterns, Tsallis entropies and global appearance features. In Proceedings of the IEEE International Conference on Image Processing. 665--668.Google ScholarGoogle ScholarCross RefCross Ref
  11. Pranav Kumar, S. L. Happy, and Aurobinda Routray. 2016. A real-time robust facial expression recognition system using HOG features. In Proceedings of the International Conference on Computing, Analytics and Security Trends. 289--293.Google ScholarGoogle ScholarCross RefCross Ref
  12. Rahul Islam, Karan Ahuja, Sandip Karmakar, and Ferdous Barbhuiya. 2016. SenTion: A framework for sensing facial expressions. Retrieved from arXiv preprint arXiv:1608.04489.Google ScholarGoogle Scholar
  13. Huei-Fang Yang, Bo-Yao Lin, Kuang-Yu Chang, and Chu-Song Chen. 2018. Joint estimation of age and expression by combining scattering and convolutional networks. ACM Trans. Multimedia Comput., Commun. Appl. 14, 1 (2018), 9--1. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Veena Mayya, Radhika M. Pai, and M. M. Manohara Pai. 2016. Automatic facial expression recognition using DCNN. Procedia Comput. Sci. 93 (2016), 453--461.Google ScholarGoogle Scholar
  15. Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. Retrieved from arXiv preprint arXiv:1704.04861.Google ScholarGoogle Scholar
  16. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems, Vol. 1. 1097--1105. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. Retrieved from arXiv preprint arXiv:1409.1556.Google ScholarGoogle Scholar
  18. Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 815--823.Google ScholarGoogle ScholarCross RefCross Ref
  19. Patrick Lucey, Jeffrey F. Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews. 2010. The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. 94--101.Google ScholarGoogle ScholarCross RefCross Ref
  20. Michael Lyons, Shigeru Akamatsu, Miyuki Kamachi, and Jiro Gyoba. 1998. Coding facial expressions with Gabor wavelets. In Proceedings of the 3rd International Conference on Face 8 Gesture Recognition. 200--205. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Maja Pantic, Michel Valstar, Ron Rademaker, and Ludo Maat. 2005. Web-based database for facial expression analysis. In Proceedings of the IEEE International Conference on Multimedia and Expo. 5.Google ScholarGoogle ScholarCross RefCross Ref
  22. Oliver Langner, Ron Dotsch, Gijsbert Bijlstra, Daniel H. J. Wigboldus, Skyler T. Hawk, and A. D. Van Knippenberg. 2010. Presentation and validation of the Radboud Faces Database. Cognit. Emot. 24, 8 (2010), 1377--1388.Google ScholarGoogle ScholarCross RefCross Ref
  23. Yuxiang Jiang, Haiwei Dong, and Abdulmotaleb El Saddik. 2018. Baidu Meizu deep learning competition: Arithmetic operation recognition using end-to-end learning OCR techniques. IEEE Access 6 (2018), 60128--60136.Google ScholarGoogle ScholarCross RefCross Ref
  24. Nima Tajbakhsh, Jae Y. Shin, Suryakanth R. Gurudu, R. Todd Hurst, Christopher B. Kendall, Michael B. Gotway, and Jianming Liang. 2016. Convolutional neural networks for medical image analysis: Full training or fine tuning? IEEE Trans. Medical Imag. 35, 5 (2016), 1299--1312.Google ScholarGoogle ScholarCross RefCross Ref
  25. Bo-Kyeong Kim, Jihyeon Roh, Suh-Yeon Dong, and Soo-Young Lee. 2016. Hierarchical committee of deep convolutional neural networks for robust facial expression recognition. J. Multimod. User Interfaces 10, 2 (2016), 173--189.Google ScholarGoogle ScholarCross RefCross Ref
  26. Hong-Wei Ng, Viet Dung Nguyen, Vassilios Vonikakis, and Stefan Winkler. 2015. Deep learning for emotion recognition on small datasets using transfer learning. In Proceedings of the ACM International Conference on Multimodal Interaction. 443--449. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. 2016. A discriminative feature learning approach for deep face recognition. In Proceedings of the European Conference on Computer Vision. 499--515.Google ScholarGoogle ScholarCross RefCross Ref
  28. Charles Darwin and Phillip Prodger. 1998. The Expression of the Emotions in Man and Animals. Oxford University Press, USA.Google ScholarGoogle Scholar
  29. Paul Ekman and Erika L. Rosenberg. 1997. What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press, USA.Google ScholarGoogle Scholar
  30. Di Huang, Caifeng Shan, Mohsen Ardabilian, Yunhong Wang, and Liming Chen. 2011. Local binary patterns and its application to facial image analysis: a survey. IEEE Trans. Syst., Man, Cyber., Part C (Appl. Rev.) 41, 6 (2011), 765--781. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Yongqiang Yao, Di Huang, Xudong Yang, Yunhong Wang, and Liming Chen. 2018. Texture and geometry scattering representation-based facial expression recognition in 2D+3D videos. ACM Trans. Multimedia Comput., Commun. Appl. 14, 1s (2018), 18. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Zhiding Yu and Cha Zhang. 2015. Image based static facial expression recognition with multiple deep network learning. In Proceedings of the ACM International Conference on Multimodal Interaction. 435--442. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Peter Burkert, Felix Trier, Muhammad Zeshan Afzal, Andreas Dengel, and Marcus Liwicki. 2015. DeXpression: Deep convolutional neural network for expression recognition. Retrieved from arXiv preprint arXiv:1509.05371.Google ScholarGoogle Scholar
  34. Yichuan Tang. 2013. Deep learning using linear support vector machines. Retrieved from arXiv preprint arXiv:1306.0239.Google ScholarGoogle Scholar
  35. Ian J. Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al. 2013. Challenges in representation learning: A report on three machine learning contests. In Proceedings of the International Conference on Neural Information Processing. 117--124.Google ScholarGoogle ScholarCross RefCross Ref
  36. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1--9.Google ScholarGoogle ScholarCross RefCross Ref
  37. Ali Mollahosseini, David Chan, and Mohammad H. Mahoor. 2016. Going deeper in facial expression recognition using deep neural networks. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision. 1--10.Google ScholarGoogle Scholar
  38. Paul Viola and Michael J. Jones. 2004. Robust real-time face detection. Int. J. Comput. Vis. 57, 2 (2004), 137--154. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22, 10 (2010), 1345--1359. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278--2324.Google ScholarGoogle ScholarCross RefCross Ref
  41. George E. Dahl, Tara N. Sainath, and Geoffrey E. Hinton. 2013. Improving deep neural networks for LVCSR using rectified linear units and dropout. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. 8609--8613.Google ScholarGoogle Scholar
  42. S. L. Happy and Aurobinda Routray. 2015. Automatic facial expression recognition using features of salient facial patches. IEEE Trans. Affect. Comput. 6, 1 (2015), 1--12.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Rohit Verma and Mohamed-Yahia Dabbagh. 2013. Fast facial expression recognition based on local binary patterns. In Proceedings of the 26th IEEE Canadian Conference on Electrical and Computer Engineering. 1--4.Google ScholarGoogle ScholarCross RefCross Ref
  44. Ping Liu, Shizhong Han, Zibo Meng, and Yan Tong. 2014. Facial expression recognition via a boosted deep belief network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1805--1812. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Frank Y. Shih, Chao-Fa Chuang, and Patrick S. P. Wang. 2008. Performance comparisons of facial expression recognition in JAFFE database. Int. J. Pattern Recog. Artific. Intell. 22, 3 (2008), 445--459.Google ScholarGoogle ScholarCross RefCross Ref
  46. Fei Cheng, Jiangsheng Yu, and Huilin Xiong. 2010. Facial expression recognition in JAFFE dataset based on Gaussian process classification. IEEE Trans. Neural Netw. 21, 10 (2010), 1685--1690. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Yogachandran Rahulamathavan, Raphael C.-W. Phan, Jonathon A. Chambers, and David J. Parish. 2013. Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans. Affect. Comput. 4, 1 (2013), 83--92. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Andre Teixeira Lopes, Edilson de Aguiar, and Thiago Oliveira-Santos. 2015. A facial expression recognition system using convolutional networks. In Proceedings of the 28th SIBGRAPI Conference on Graphics, Patterns and Images. 273--280. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Kamlesh Mistry, Li Zhang, Siew Chin Neoh, Ming Jiang, Alamgir Hossain, and Benoît Lafon. 2014. Intelligent appearance and shape based facial emotion recognition for a humanoid robot. In Proceedings of the 8th International Conference on Software, Knowledge, Information Management and Applications. 1--8.Google ScholarGoogle ScholarCross RefCross Ref
  50. Mundher Al-Shabi, Wooi Ping Cheah, and Tee Connie. 2016. Facial expression recognition using a hybrid CNN-SIFT aggregator. Retrieved from arXiv preprint arXiv:1608.02833.Google ScholarGoogle Scholar
  51. Pooya Khorrami, Thomas Le Paine, and Thomas S. Huang. 2015. Do deep neural networks learn facial action units when doing expression recognition? In Proceedings of the IEEE International Conference on Computer Vision Workshops. 19--27. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation. 265--283. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. A Deep Learning System for Recognizing Facial Expression in Real-Time

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Multimedia Computing, Communications, and Applications
          ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 15, Issue 2
          May 2019
          375 pages
          ISSN:1551-6857
          EISSN:1551-6865
          DOI:10.1145/3339884
          Issue’s Table of Contents

          Copyright © 2019 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 5 June 2019
          • Accepted: 1 January 2019
          • Revised: 1 November 2018
          • Received: 1 September 2018
          Published in tomm Volume 15, Issue 2

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Research
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format .

        View HTML Format
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!