skip to main content
research-article

Hypomimia Recognition in Parkinson’s Disease With Semantic Features

Authors Info & Claims
Published:26 October 2021Publication History
Skip Abstract Section

Abstract

Parkinson’s disease is the second most common neurodegenerative disorder, commonly affecting elderly people over the age of 65. As the cardinal manifestation, hypomimia, referred to as impairments in normal facial expressions, stays covert. Even some experienced doctors may miss these subtle changes, especially in a mild stage of this disease. The existing methods for hypomimia recognition are mainly dominated by statistical variable-based methods with the help of traditional machine learning algorithms. Despite the success of recognizing hypomimia, they show a limited accuracy and lack the capability of performing semantic analysis. Therefore, developing a computer-aided diagnostic method for semantically recognizing hypomimia is appealing. In this article, we propose a Semantic Feature based Hypomimia Recognition network, named SFHR-NET, to recognize hypomimia based on facial videos. First, a Semantic Feature Classifier (SF-C) is proposed to adaptively adjust feature maps salient to hypomimia, which leads the encoder and classifier to focus more on areas of hypomimia-interest. In SF-C, the progressive confidence strategy (PCS) ensures more reliable semantic features. Then, a two-stream framework is introduced to fuse the spatial data stream and temporal optical stream, which allows the encoder to semantically and progressively characterize the rigid process of hypomimia. Finally, to improve the interpretability of the model, Gradient-weighted Class Activation Mapping (Grad-CAM) is integrated to generate attention maps that cast our engineered features into hypomimia-interest regions. These highlighted regions provide visual explanations for decisions of our network. Experimental results based on real-world data demonstrate the effectiveness of our method in detecting hypomimia.

REFERENCES

  1. [1] Wood A., Lupyan G., Sherrin S., and Niedenthal. P.2016. Altering sensorimotor feedback disrupts visual discrimination of facial expressions. Psychonomic Bulletin & Review. 23, 4 (2016), 11501156. https://doi.org/10.3758/s13423-015-0974-5Google ScholarGoogle ScholarCross RefCross Ref
  2. [2] Bandini Andrea, Orlandi Silvia, Escalante Hugo Jair, Giovannelli Fabio, Cincotta Massimo, Reyes-Garcia Carlos A., Vanni Paola, Zaccara Gaetano, and Manfredi Claudia. 2017. Analysis of facial expressions in Parkinson’s disease through video-based automatic methods. Journal of Neuroscience Methods 281 (2017), 720. https://doi.org/10.1016/j.jneumeth.2017.02.006Google ScholarGoogle ScholarCross RefCross Ref
  3. [3] Bengio Yoshua, Courville Aaron, and Vincent Pascal. 2013. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 8 (2013), 17981828. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. [4] Berretti Stefano, Amor Boulbaba Ben, Daoudi Mohamed, and Bimbo Alberto Del. 2011. 3D facial expression recognition using SIFT descriptors of automatically detected keypoints. The Visual Computer 27, 11 (2011), 10211036. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. [5] Bologna Matteo, Fabbrini Giovanni, Marsili Luca, Defazio Giovanni, Thompson Philip D., and Berardelli Alfredo. 2013. Facial bradykinesia. Journal of Neurology, Neurosurgery & Psychiatry 84, 6 (2013), 681685. https://doi.org/10.1136/jnnp-2012-303993 https://jnnp.bmj.com/content/84/6/681.full.pdf.Google ScholarGoogle ScholarCross RefCross Ref
  6. [6] Capecci M., Ciabattoni L., Foresi G., Monteri A., and Pepa L.. 2019. A machine-learning based emotion recognition system in patients with Parkinson’s disease. In 2019 IEEE 9th International Conference on Consumer Electronics (ICCE-Berlin). 2021. https://doi.org/10.1109/ICCE-Berlin47944.2019.8966224Google ScholarGoogle ScholarCross RefCross Ref
  7. [7] Dauer William and Przedborski Serge. 2003. Parkinson’s disease: Mechanisms and models. Neuron 39, 6 (2003), 889909. https://doi.org/10.1016/S0896-6273(03)00568-3Google ScholarGoogle ScholarCross RefCross Ref
  8. [8] Du Liang, Tan Jingang, Yang Hongye, Feng Jianfeng, Xue Xiangyang, Zheng Qibao, Ye Xiaoqing, and Zhang Xiaolin. 2019. SSF-DAN: Separated semantic feature based domain adaptation network for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).Google ScholarGoogle ScholarCross RefCross Ref
  9. [9] Goetz Christopher G., Fahn Stanley, Martinez-Martin Pablo, Poewe Werner, Sampaio Cristina, Stebbins Glenn T., Stern Matthew B., Tilley Barbara C., Dodel Richard, Dubois Bruno, Holloway Robert, Jankovic Joseph, Kulisevsky Jaime, Lang Anthony E., Lees Andrew, Leurgans Sue, LeWitt Peter A., Nyenhuis David, Olanow C. Warren, Rascol Olivier, Schrag Anette, Teresi Jeanne A., Hilten Jacobus J. Van, and LaPelle Nancy. 2007. Movement disorder society-sponsored revision of the unified Parkinson’s disease rating scale (MDS-UPDRS): Process, format, and clinimetric testing plan. Movement Disorders 22, 1 (2007), 4147. arXiv: https://onlinelibrary.wiley.com/doi/pdf/10.1002/mds.21198.Google ScholarGoogle ScholarCross RefCross Ref
  10. [10] Grammatikopoulou Athina, Grammalidis Nikos, Bostantjopoulou Sevasti, and Katsarou Zoe. 2019. Detecting hypomimia symptoms by selfie photo analysis: For early Parkinson’s disease detection. In Proceedings of the 12th ACM International Conference on Pervasive Technologies Related to Assistive Environments (Rhodes, Greece) (PETRA’19). Association for Computing Machinery, New York, NY, USA, 517-522. https://doi.org/10.1145/3316782.3322756 Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. [11] Gray. L. Tickle-Degnen and H. M.2010. A meta-analysis of performance on emotion recognition tasks in Parkinson’s disease. Neuropsychology 24, 2 (2010), 176-191. https://doi.org/10.1037/a0018104Google ScholarGoogle ScholarCross RefCross Ref
  12. [12] He Kaiming, Zhang Xiangyu, Ren Shaoqing, and Sun Jian. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Ho Mary Wen-Reng, Chien Sarina Hui-Lin, Lu Ming-Kuei, Chen Jui-Cheng, Aoh Yu, Chen Chun-Ming, Lane Hsien-Yuan, and Tsai Chon-Haw. 2020. Impairments in face discrimination and emotion recognition are related to aging and cognitive dysfunctions in Parkinson’s disease with dementia. Scientific Reports 10, 1 (2020), 18.Google ScholarGoogle ScholarCross RefCross Ref
  14. [14] Horn Berthold K. P. and Schunck Brian G.. 1981. Determining optical flow. In Techniques and Applications of Image Understanding, Vol. 281. International Society for Optics and Photonics, 319331. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. [15] Jakab Tomas, Gupta Ankush, Bilen Hakan, and Vedaldi Andrea. 2020. Self-supervised learning of interpretable keypoints from unlabelled videos. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarGoogle Scholar
  16. [16] Ji S., Xu W., Yang M., and Yu K.. 2013. 3D convolutional neural networks for human action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 1 (2013), 221231. https://doi.org/10.1109/TPAMI.2012.59 Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. [17] Jin Bo, Qu Yue, Zhang Liang, and Gao Zhan. 2020. Diagnosing Parkinson’s disease through facial expression recognition: Video analysis. Journal of Medical Internet Research 22 (2020).Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Lilly M. L., Hermanns Melinda, Dallam Deborah L., and Soomro Amal. 2020. Parkinson’s disease: Addressing health care practitioners’ automatic responses to hypomimia. Journal of the American Association of Nurse Practitioners (2020).Google ScholarGoogle Scholar
  19. [19] Ricciardi Lucia, Visco-Comandini Frederica, Erro Roberto, Morgante Francesca, Bologna Matteo, Fasano Alfonso, Ricciardi Diego, Edwards Mark J., and Kilner James. 2017. Facial emotion recognition and expression in Parkinson’s disease: An emotional mirror mechanism? PLOS ONE 12, 1 (01 2017), 116. https://doi.org/10.1371/journal.pone.0169110Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Luo Xiaofan, Wong Fukoeng, and Hu Haifeng. 2020. FIN: Feature integrated network for object detection. ACM Trans. Multim. Comput. Commun. Appl. 16, 2 (2020), 48:1-18. https://doi.org/10.1145/3381086 Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. [21] Mahendran Aravindh and Vedaldi Andrea. 2016. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision 120, 3 (2016), 233255. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. [22] Fernandez Pedro D. Marrero, Pena Fidel A. Guerrero, Ren Tsang, and Cunha Alexandre. 2019. Feratt: Facial expression recognition with attention net. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Miao Yu, Dong Haiwei, Al’Jaam Jihad Mohamad, and El-Saddik Abdulmotaleb. 2019. A deep learning system for recognizing facial expression in real-time. ACM Trans. Multim. Comput. Commun. Appl. 15, 2 (2019), 33:1-20. https://doi.org/10.1145/3311747 Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. [24] Mullin Stephen and Schapira Anthony H. V.. 2015. Pathogenic mechanisms of neurodegeneration in Parkinson’s disease. Neurologic Clinics 33, 1 (2015), 117. https://doi.org/10.1016/j.ncl.2014.09.010Movement Disorders.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Noori Farzan Majeed, Riegler Michael, Uddin Md. Zia, and Tørresen Jim. 2020. Human activity recognition from multiple sensors data using multi-fusion representations and CNNs. ACM Trans. Multim. Comput. Commun. Appl. 16, 2 (2020), 45:1–45:19. https://doi.org/10.1145/3377882Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. [26] Pepa L., Capecci M., and Ceravolo M. G.. 2019. Smartwatch based emotion recognition in Parkinson’s disease. In 2019 IEEE 23rd International Symposium on Consumer Technologies (ISCT). 2324. https://doi.org/10.1109/ISCE.2019.8901033Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Rajnoha M., Mekyska J., Burget R., Eliasova I., Kostalova M., and Rektorova I.. 2018. Towards identification of hypomimia in Parkinson’s disease based on face recognition methods. In 2018 10th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT). 14. https://doi.org/10.1109/ICUMT.2018.8631249Google ScholarGoogle Scholar
  28. [28] Raza Chand, Anjum Rabia, and Shakeel Noor ul Ain. 2019. Parkinson’s disease: Mechanisms, translational models and management strategies. Life Sciences 226 (2019), 7790. https://doi.org/10.1016/j.lfs.2019.03.057Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Ricciardi Lucia, Baggio Paola, Ricciardi Diego, Morabito Bruno, Pomponi Massimiliano, Bentivoglio Anna, Bernabei Roberto, Maestri Roberto, Frazzitta Giuseppe, and Volpe Daniele. 2015. Rehabilitation of hypomimia in Parkinson’s disease: A feasibility study of two different approaches. Neurological Sciences 37 (11 2015). https://doi.org/10.1007/s10072-015-2421-9Google ScholarGoogle Scholar
  30. [30] Ricciardi L., Angelis A. De, Marsili L., Faiman I., Pradhan P., Pereira E. A., Edwards M. J., Morgante F., and Bologna M.. 2020. Hypomimia in Parkinson’s disease: An axial sign responsive to levodopa. European Journal of Neurology 27, 12 (2020), 24222429. https://doi.org/10.1111/ene.14452 arXiv: https://onlinelibrary.wiley.com/doi/pdf/10.1111/ene.14452.Google ScholarGoogle ScholarCross RefCross Ref
  31. [31] Argaud S., Delplanque S., Houvenaghel J.-F., Auffret M., Duprez J., Verin M., Grandjean D., and Sauleau. P.2016. Does facial amimia impact the recognition of facial emotions? An EMG study in Parkinson’s disease. PLOS ONE 11, 7 (2016). https://doi.org/10.1371/journal.pone.0160329Google ScholarGoogle ScholarCross RefCross Ref
  32. [32] Sariyanidi Evangelos, Zampella Casey J., Schultz Robert T., and Tunc Birkan. 2020. Can facial pose and expression be separated with weak perspective camera?. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarGoogle Scholar
  33. [33] Seliverstov Yury, Diagovchenko Dmitrii, Kravchenko Michael, Babin Mikhail, Fedotova Ekaterina, and Belyaev Mikhail. 2018. Hypomimia detection with a smartphone camera as a possible self-screening tool for Parkinson disease (P3.047). Neurology 90, 15 Supplement (2018). arXiv: https://n.neurology.org/content https://n.neurology.org/content/90/15_Supplement/P3.047.Google ScholarGoogle Scholar
  34. [34] Selvaraju Ramprasaath R., Cogswell Michael, Das Abhishek, Vedantam Ramakrishna, Parikh Devi, and Batra Dhruv. 2017. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).Google ScholarGoogle ScholarCross RefCross Ref
  35. [35] Shi Lei, Zhang Yifan, Cheng Jian, and Lu Hanqing. 2019. Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarGoogle ScholarCross RefCross Ref
  36. [36] Simonyan Karen and Zisserman Andrew. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).Google ScholarGoogle Scholar
  37. [37] Tian Zhi, Shen Chunhua, Chen Hao, and He Tong. 2019. FCOS: Fully convolutional one-stage object detection. In Proceedings of the IEEE International Conference on Computer Vision. 96279636.Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Ventura Maria I., Baynes Kathleen, Sigvardt Karen A., Unruh April M., Acklin Sarah S., Kirsch Heidi E., and Disbrow Elizabeth A.. 2012. Hemispheric asymmetries and prosodic emotion recognition deficits in Parkinson’s disease. Neuropsychologia 50, 8 (2012), 19361945. https://doi.org/10.1016/j.neuropsychologia.2012.04.018Google ScholarGoogle ScholarCross RefCross Ref
  39. [39] Vinokurov Nomi, Arkadir David, Linetsky Eduard, Bergman Hagai, and Weinshall Daphna. 2016. Quantifying hypomimia in Parkinson patients using a depth camera. In Pervasive Computing Paradigms for Mental Health, Serino Silvia, Matic Aleksandar, Giakoumis Dimitris, Lopez Guillaume, and Cipresso Pietro (Eds.). Springer International Publishing, Cham, 6371. Google ScholarGoogle ScholarCross RefCross Ref
  40. [40] Yan Rui, Xie Lingxi, Tang Jinhui, Shu Xiangbo, and Tian Qi. 2020. HiGCIN: Hierarchical graph-based cross inference network for group activity recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020).Google ScholarGoogle ScholarCross RefCross Ref
  41. [41] Yin X. and Liu X.. 2018. Multi-task convolutional neural network for pose-invariant face recognition. IEEE Transactions on Image Processing 27, 2 (2018), 964975. https://doi.org/10.1109/TIP.2017.2765830Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. [42] Zhao Jia-Xing, Liu Jiang-Jiang, Fan Deng-Ping, Cao Yang, Yang Jufeng, and Cheng Ming-Ming. 2019. EGNet: Edge guidance network for salient object detection. In Proceedings of the IEEE International Conference on Computer Vision. 87798788.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Hypomimia Recognition in Parkinson’s Disease With Semantic Features

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Multimedia Computing, Communications, and Applications
          ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 17, Issue 3s
          October 2021
          324 pages
          ISSN:1551-6857
          EISSN:1551-6865
          DOI:10.1145/3492435
          Issue’s Table of Contents

          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 26 October 2021
          • Accepted: 1 May 2021
          • Revised: 1 April 2021
          • Received: 1 December 2020
          Published in tomm Volume 17, Issue 3s

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Full Text

        View this article in Full Text.

        View Full Text

        HTML Format

        View this article in HTML Format .

        View HTML Format
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!