Abstract
In this article, we focus on the dynamic facial emotion recognition from online video. We combine deep neural networks with transfer learning theory and propose a novel model named DT-EFER. In detail, DT-EFER uses GoogLeNet to extract the deep features of key images from video clips. Then to solve the dynamic facial emotion recognition scenario, the framework introduces transfer learning theory. Thus, to improve the recognition performance, model DT-EFER focuses on the differences between key images instead of those images themselves. Moreover, the time complexity of this model is not high, even if previous exemplars are introduced here. In contrast to other exemplar-based models, experiments based on two datasets, namely, BAUM-1s and Extended Cohn–Kanade, have shown the efficiency of the proposed DT-EFER model.
- [1] . 2016. Incremental enhanced \( \alpha \) expansion move for large data: A probability regularization perspective. International Journal of Machine Learning and Cybernetics 8, 5 (2016), 1–17.Google Scholar
- [2] . 2016. Bayesian enhanced \( \alpha \) expansion move clustering with loose link constraints. Neurocomputing 194 (2016), 288–300.
DOI: Google ScholarDigital Library
- [3] . 2019. EEG-based multi-modal emotion recognition using bag of deep features: An optimal feature selection approach. Sensors 19, 23 (2019).Google Scholar
Cross Ref
- [4] . 2019. Deep features-based speech emotion recognition for smart affective services. Multimedia Tools and Applications 78, 5 (2019), 5571–5589.Google Scholar
Digital Library
- [5] , Wenhao Ying, and Lu Zhao. 2020. Fast Enhanced Exemplar-Based Clustering for Incomplete EEG Signals. Computational and Mathematical Methods in Medicine 2020, Article ID 4147807 (2020), 11 pages. Google Scholar
Cross Ref
- [6] . 2020. Mimamo net: Integrating micro-and macro-motion for video emotion recognition. In Proceedings of the AAAI Conference on Artificial Intelligence. 2621–2628.Google Scholar
Cross Ref
- [7] . 2021. A convolution bidirectional long short-term memory neural network for driver emotion recognition. IEEE Transactions on Intelligent Transportation Systems 22, 7 (2021), 4570–4578.
DOI: Google ScholarDigital Library
- [8] . 2016. Video-based emotion recognition using CNN-RNN and C3D hybrid networks. In Proceedings of the 18th ACM International Conference on Multimodal Interaction. 445–450.Google Scholar
Digital Library
- [9] . 2021. Privacy protection for medical data sharing in smart healthcare. ACM Transactions on Multimedia Computing Communications and Applications 16, 3s (2021), 1–18.Google Scholar
Digital Library
- [10] . 2007. Clustering by passing messages between data points. Science 315, 5814 (2007), 972–976.
DOI: Google ScholarCross Ref
- [11] . 2019. Eyemotion: Classifying facial expressions in VR using eye-tracking cameras. In Proceedings of the 2019 IEEE Winter Conference on Applications of Computer Vision. 1626–1635.
DOI: Google ScholarCross Ref
- [12] . 2020. Facial expression recognition under partial occlusion from virtual reality headsets based on transfer learning. In Proceedings of the 2020 IEEE 6th International Conference on Multimedia Big Data. 70–75.
DOI: Google ScholarCross Ref
- [13] . 2012. Transfer spectral clustering. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 789–803.Google Scholar
Cross Ref
- [14] . 2016. Emonets: Multimodal deep learning approaches for emotion recognition in video. Journal on Multimodal User Interfaces 10, 2 (2016), 99–111.Google Scholar
Cross Ref
- [15] . 2017. Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Transactions on Affective Computing 10, 2 (2017), 223–236.Google Scholar
Cross Ref
- [16] . 2018. A brief review of facial emotion recognition based on visual information. Sensors 18, 2 (2018), 401.Google Scholar
Cross Ref
- [17] . 2020. Deep facial expression recognition: A survey. IEEE Transactions on Affective Computing (2020), 1–1.
DOI: Google ScholarCross Ref
- [18] . 2010. The extended cohn-kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops. 94–101.Google Scholar
Cross Ref
- [19] . 2018. Facial emotion recognition: A survey and real-world user experiences in mixed reality. Sensors 18, 2 (2018), 416–440.
DOI: Google ScholarCross Ref
- [20] . 2019. Video facial emotion recognition based on local enhanced motion history image and CNN-CTSLSTM networks. Journal of Visual Communication and Image Representation 59 (2019), 176–185. Google Scholar
Digital Library
- [21] . 2015. A survey on data stream clustering and classification. Knowledge and Information Systems 45, 3 (2015), 535–569.Google Scholar
Digital Library
- [22] . 2021. Survey on emotional body gesture recognition. IEEE Transactions on Affective Computing 12, 2 (2021), 505–523.
DOI: Google ScholarDigital Library
- [23] . 2017. Audio-visual emotion recognition in video clips. IEEE Transactions on Affective Computing 10, 1 (2017), 60–75.Google Scholar
Digital Library
- [24] . 2015. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1–9.Google Scholar
Cross Ref
- [25] . 2021. Pinball loss twin support vector clustering. ACM Transactions on Multimedia Computing, Communications, and Applications 17, 2s (2021).Google Scholar
Digital Library
- [26] . 2021. Classification of alzheimer’s disease using ensemble of deep neural networks trained through transfer learning. IEEE Journal of Biomedical and Health Informatics 26, 4 (2021), 1453–1463.
DOI: Google ScholarCross Ref
- [27] . 2021. Neural style palette: A multimodal and interactive style transfer from a single style image. IEEE Transactions on Multimedia 23 (2021), 2245–2258.Google Scholar
Digital Library
- [28] . 2018. Intelligent facial emotion recognition based on stationary wavelet entropy and jaya algorithm. Neurocomputing 272 (2018), 668–676.Google Scholar
Digital Library
- [29] . 2022. Two-stage fuzzy fusion based-convolution neural network for dynamic emotion recognition. IEEE Transactions on Affective Computing 13, 2 (2022), 805–817.Google Scholar
- [30] . 2021. Physiological signals based affective computing: A systematic review. Acta Automatica Sinica 47, x (2021).
DOI: Google ScholarCross Ref
- [31] . 2013. Clustering based on enhanced expansion move. IEEE Transaction on Knowledge and Data Engineering 25, 10 (2013), 2206–2216.
DOI: 109/TKDE.2012.202Google Scholar - [32] . 2017. Seizure classification from EEG signals using transfer learning, semi-supervised learning and TSK fuzzy system. IEEE Transactions on Neural Systems and Rehabilitation Engine 25, 12 (2017), 2270–2284.
DOI: Google ScholarCross Ref
- [33] . 2019. Exemplar-based data stream clustering toward internet of things. The Journal of Supercomputing 76, 7 (2019).Google Scholar
- [34] . 2020. Constructing a personalized cross-day EEG-based emotion-classification model using transfer learning. IEEE Journal of Biomedical and Health Informatics 24, 5 (2020), 1255–1264.
DOI: Google ScholarCross Ref
- [35] . 2018. Facial expression recognition via learning deep sparse autoencoders. Neurocomputing 273 (2018), 643–649. Google Scholar
Digital Library
- [36] . 2020. Driver emotion recognition for intelligent vehicles: A survey. ACM Computing Surveys 53, 3 (2020), 1–30.
DOI: Google ScholarDigital Library
- [37] . 2017. BAUM-1: A spontaneous audio-visual face database of affective and mental states. IEEE Transactions on Affective Computing 8, 3 (2017), 300–313.Google Scholar
Digital Library
- [38] . 2020. Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review. Information Fusion 59 (2020), 103–126.Google Scholar
Cross Ref
- [39] . 2016. Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratified cross validation. IEEE Access 4 (2016), 8375–8385.Google Scholar
Cross Ref
- [40] . 2019. A novel double-index-constrained, multi-view, fuzzy-clustering algorithm and its application for detecting epilepsy electroencephalogram signals. IEEE Access 7 (2019), 103823–103832.Google Scholar
Cross Ref
Index Terms
Dynamic Transfer Exemplar based Facial Emotion Recognition Model Toward Online Video
Recommendations
Facial Emotion Recognition with Varying Poses and/or Partial Occlusion Using Multi-stage Progressive Transfer Learning
Image AnalysisAbstractThis paper describes the use of multi-stage Progressive Transfer Learning (MSPTL) to improve the performance of automated Facial Emotion Recognition (FER). Our proposed FER solution is designed to work with 2D images, and is able to classify ...
Automatic face analysis system based on face recognition and facial physiognomy
ICHIT'06: Proceedings of the 1st international conference on Advances in hybrid information technologyAn automatic face analysis system is proposed which uses face recognition and facial physiognomy. It first detects human's face, extracts its features, and classifies the shape of facial features. It will analyze the person's facial physiognomy and then ...
Toward Personalized Emotion Recognition: A Face Recognition Based Attention Method for Facial Emotion Recognition
2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)This paper aims to address the subject-dependent challenge of the facial emotion recognition (FER) task. To accomplish this, we propose a novel face recognition based attention FER (FRA-FER) framework which propagates subtle face recognition (FR) features ...






Comments