Abstract
Parkinson’s disease is the second most common neurodegenerative disorder, commonly affecting elderly people over the age of 65. As the cardinal manifestation, hypomimia, referred to as impairments in normal facial expressions, stays covert. Even some experienced doctors may miss these subtle changes, especially in a mild stage of this disease. The existing methods for hypomimia recognition are mainly dominated by statistical variable-based methods with the help of traditional machine learning algorithms. Despite the success of recognizing hypomimia, they show a limited accuracy and lack the capability of performing semantic analysis. Therefore, developing a computer-aided diagnostic method for semantically recognizing hypomimia is appealing. In this article, we propose a Semantic Feature based Hypomimia Recognition network, named SFHR-NET, to recognize hypomimia based on facial videos. First, a Semantic Feature Classifier (SF-C) is proposed to adaptively adjust feature maps salient to hypomimia, which leads the encoder and classifier to focus more on areas of hypomimia-interest. In SF-C, the progressive confidence strategy (PCS) ensures more reliable semantic features. Then, a two-stream framework is introduced to fuse the spatial data stream and temporal optical stream, which allows the encoder to semantically and progressively characterize the rigid process of hypomimia. Finally, to improve the interpretability of the model, Gradient-weighted Class Activation Mapping (Grad-CAM) is integrated to generate attention maps that cast our engineered features into hypomimia-interest regions. These highlighted regions provide visual explanations for decisions of our network. Experimental results based on real-world data demonstrate the effectiveness of our method in detecting hypomimia.
- [1] 2016. Altering sensorimotor feedback disrupts visual discrimination of facial expressions. Psychonomic Bulletin & Review. 23, 4 (2016), 1150–1156. https://doi.org/10.3758/s13423-015-0974-5Google Scholar
Cross Ref
- [2] . 2017. Analysis of facial expressions in Parkinson’s disease through video-based automatic methods. Journal of Neuroscience Methods 281 (2017), 7–20. https://doi.org/10.1016/j.jneumeth.2017.02.006Google Scholar
Cross Ref
- [3] . 2013. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 8 (2013), 1798–1828. Google Scholar
Digital Library
- [4] . 2011. 3D facial expression recognition using SIFT descriptors of automatically detected keypoints. The Visual Computer 27, 11 (2011), 1021–1036. Google Scholar
Digital Library
- [5] . 2013. Facial bradykinesia. Journal of Neurology, Neurosurgery & Psychiatry 84, 6 (2013), 681–685. https://doi.org/10.1136/jnnp-2012-303993 https://jnnp.bmj.com/content/84/6/681.full.pdf.Google Scholar
Cross Ref
- [6] . 2019. A machine-learning based emotion recognition system in patients with Parkinson’s disease. In 2019 IEEE 9th International Conference on Consumer Electronics (ICCE-Berlin). 20–21. https://doi.org/10.1109/ICCE-Berlin47944.2019.8966224Google Scholar
Cross Ref
- [7] . 2003. Parkinson’s disease: Mechanisms and models. Neuron 39, 6 (2003), 889–909. https://doi.org/10.1016/S0896-6273(03)00568-3Google Scholar
Cross Ref
- [8] . 2019. SSF-DAN: Separated semantic feature based domain adaptation network for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).Google Scholar
Cross Ref
- [9] . 2007. Movement disorder society-sponsored revision of the unified Parkinson’s disease rating scale (MDS-UPDRS): Process, format, and clinimetric testing plan. Movement Disorders 22, 1 (2007), 41–47. arXiv: https://onlinelibrary.wiley.com/doi/pdf/10.1002/mds.21198.Google Scholar
Cross Ref
- [10] . 2019. Detecting hypomimia symptoms by selfie photo analysis: For early Parkinson’s disease detection. In Proceedings of the 12th ACM International Conference on Pervasive Technologies Related to Assistive Environments (Rhodes, Greece) (
PETRA’19 ). Association for Computing Machinery, New York, NY, USA, 517-522. https://doi.org/10.1145/3316782.3322756 Google ScholarDigital Library
- [11] 2010. A meta-analysis of performance on emotion recognition tasks in Parkinson’s disease. Neuropsychology 24, 2 (2010), 176-191. https://doi.org/10.1037/a0018104Google Scholar
Cross Ref
- [12] . 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
Cross Ref
- [13] . 2020. Impairments in face discrimination and emotion recognition are related to aging and cognitive dysfunctions in Parkinson’s disease with dementia. Scientific Reports 10, 1 (2020), 1–8.Google Scholar
Cross Ref
- [14] . 1981. Determining optical flow. In Techniques and Applications of Image Understanding, Vol. 281. International Society for Optics and Photonics, 319–331. Google Scholar
Digital Library
- [15] . 2020. Self-supervised learning of interpretable keypoints from unlabelled videos. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
- [16] . 2013. 3D convolutional neural networks for human action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 1 (2013), 221–231. https://doi.org/10.1109/TPAMI.2012.59 Google Scholar
Digital Library
- [17] . 2020. Diagnosing Parkinson’s disease through facial expression recognition: Video analysis. Journal of Medical Internet Research 22 (2020).Google Scholar
Cross Ref
- [18] . 2020. Parkinson’s disease: Addressing health care practitioners’ automatic responses to hypomimia. Journal of the American Association of Nurse Practitioners (2020).Google Scholar
- [19] . 2017. Facial emotion recognition and expression in Parkinson’s disease: An emotional mirror mechanism? PLOS ONE 12, 1 (
01 2017), 1–16. https://doi.org/10.1371/journal.pone.0169110Google ScholarCross Ref
- [20] . 2020. FIN: Feature integrated network for object detection. ACM Trans. Multim. Comput. Commun. Appl. 16, 2 (2020), 48:1-18. https://doi.org/10.1145/3381086 Google Scholar
Digital Library
- [21] . 2016. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision 120, 3 (2016), 233–255. Google Scholar
Digital Library
- [22] . 2019. Feratt: Facial expression recognition with attention net. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.Google Scholar
Cross Ref
- [23] . 2019. A deep learning system for recognizing facial expression in real-time. ACM Trans. Multim. Comput. Commun. Appl. 15, 2 (2019), 33:1-20. https://doi.org/10.1145/3311747 Google Scholar
Digital Library
- [24] . 2015. Pathogenic mechanisms of neurodegeneration in Parkinson’s disease. Neurologic Clinics 33, 1 (2015), 1–17. https://doi.org/10.1016/j.ncl.2014.09.010
Movement Disorders. Google ScholarCross Ref
- [25] . 2020. Human activity recognition from multiple sensors data using multi-fusion representations and CNNs. ACM Trans. Multim. Comput. Commun. Appl. 16, 2 (2020), 45:1–45:19. https://doi.org/10.1145/3377882Google Scholar
Digital Library
- [26] . 2019. Smartwatch based emotion recognition in Parkinson’s disease. In 2019 IEEE 23rd International Symposium on Consumer Technologies (ISCT). 23–24. https://doi.org/10.1109/ISCE.2019.8901033Google Scholar
Cross Ref
- [27] . 2018. Towards identification of hypomimia in Parkinson’s disease based on face recognition methods. In 2018 10th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT). 1–4. https://doi.org/10.1109/ICUMT.2018.8631249Google Scholar
- [28] . 2019. Parkinson’s disease: Mechanisms, translational models and management strategies. Life Sciences 226 (2019), 77–90. https://doi.org/10.1016/j.lfs.2019.03.057Google Scholar
Cross Ref
- [29] . 2015. Rehabilitation of hypomimia in Parkinson’s disease: A feasibility study of two different approaches. Neurological Sciences 37 (
11 2015). https://doi.org/10.1007/s10072-015-2421-9Google Scholar - [30] . 2020. Hypomimia in Parkinson’s disease: An axial sign responsive to levodopa. European Journal of Neurology 27, 12 (2020), 2422–2429. https://doi.org/10.1111/ene.14452 arXiv: https://onlinelibrary.wiley.com/doi/pdf/10.1111/ene.14452.Google Scholar
Cross Ref
- [31] 2016. Does facial amimia impact the recognition of facial emotions? An EMG study in Parkinson’s disease. PLOS ONE 11, 7 (2016). https://doi.org/10.1371/journal.pone.0160329Google Scholar
Cross Ref
- [32] . 2020. Can facial pose and expression be separated with weak perspective camera?. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
- [33] . 2018. Hypomimia detection with a smartphone camera as a possible self-screening tool for Parkinson disease (P3.047). Neurology 90, 15 Supplement (2018). arXiv: https://n.neurology.org/content https://n.neurology.org/content/90/15_Supplement/P3.047.Google Scholar
- [34] . 2017. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).Google Scholar
Cross Ref
- [35] . 2019. Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
Cross Ref
- [36] . 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).Google Scholar
- [37] . 2019. FCOS: Fully convolutional one-stage object detection. In Proceedings of the IEEE International Conference on Computer Vision. 9627–9636.Google Scholar
Cross Ref
- [38] . 2012. Hemispheric asymmetries and prosodic emotion recognition deficits in Parkinson’s disease. Neuropsychologia 50, 8 (2012), 1936–1945. https://doi.org/10.1016/j.neuropsychologia.2012.04.018Google Scholar
Cross Ref
- [39] . 2016. Quantifying hypomimia in Parkinson patients using a depth camera. In Pervasive Computing Paradigms for Mental Health, , , , , and (Eds.). Springer International Publishing, Cham, 63–71. Google Scholar
Cross Ref
- [40] . 2020. HiGCIN: Hierarchical graph-based cross inference network for group activity recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020).Google Scholar
Cross Ref
- [41] . 2018. Multi-task convolutional neural network for pose-invariant face recognition. IEEE Transactions on Image Processing 27, 2 (2018), 964–975. https://doi.org/10.1109/TIP.2017.2765830Google Scholar
Digital Library
- [42] . 2019. EGNet: Edge guidance network for salient object detection. In Proceedings of the IEEE International Conference on Computer Vision. 8779–8788.Google Scholar
Cross Ref
Index Terms
Hypomimia Recognition in Parkinson’s Disease With Semantic Features
Recommendations
Effective semantic features for facial expressions recognition using SVM
Most traditional facial expression-recognition systems track facial components such as eyes, eyebrows, and mouth for feature extraction. Though some of these features can provide clues for expression recognition, other finer changes of the facial ...
Tag-Based Semantic Features for Scene Image Classification
Neural Information ProcessingAbstractThe existing image feature extraction methods are primarily based on the content and structure information of images, and rarely consider the contextual semantic information. Regarding some types of images such as scenes and objects, the ...
Recognition of the parkinson’s disease using a hybrid feature selection approach
Accurate and efficient recognition of Parkinson’s disease is one of the prominent issues in the field of healthcare. To address this problem, different methods have been proposed in the literature. However, existing methods are lacking in accurately ...






Comments