Abstract
Emotional color transfer aims to change the evoked emotion of a source image to that of a target image by adjusting color distribution. Most of existing emotional color transfer methods only consider the low-level visual features of an image and ignore the facial expression features when the image contains a human face, which would cause incorrect emotion evaluation for the given image. In addition, previous emotional color transfer methods may easily result in ambiguity between the emotion of resulting image and target image. For example, if the background of the target image is dark while the facial expression is happiness, then previous methods would directly transfer dark color to the source image, neglecting the facial emotion in the image. To solve this problem, we propose a new facial-expression-aware emotional color transfer framework. Given a target image with facial expression features, we first predict the facial emotion label of the image through the emotion classification network. Then, facial emotion labels are matched with pre-trained emotional color transfer models. Finally, we use the matched emotion model to transfer the color of the target image to the source image. Considering none of the existing emotion image databases, which focus on images that contain face and background, we built an emotion database for our new emotional color transfer framework that is called “Face-Emotion database.” Experiments demonstrate that our method can successfully capture and transfer facial emotions, outperforming state-of-the-art methods.
- [1] . 2008. Automatic mood-transferring between color images. IEEE Comput. Graph. Appl. 28, 2 (2008), 52–61. Google Scholar
Digital Library
- [2] . 2013. Affective image adjustment with a single word. Visual Comput. 29, 11 (2013), 1121–1133. Google Scholar
Digital Library
- [3] . 2014. Emotion affective color transfer. Int. J. Softw. Eng. Appl. 8, 3 (2014), 227–232.Google Scholar
- [4] . 2017. Patch2vec: Globally consistent image patch representation. In Computer Graphics Forum, Vol. 36. 183–194.Google Scholar
- [5] . 2016. Muscular movement model-based automatic 3D/4D facial expression recognition. IEEE Trans. Multimedia 18, 7 (2016), 1438–1450. Google Scholar
Digital Library
- [6] . 2017. Multimodal 2D+3D facial expression recognition with deep fusion convolutional neural network. IEEE Trans. Multimedia 19, 12 (2017), 2816–2831.Google Scholar
Cross Ref
- [7] . 2015. A mixed bag of emotions: Model, predict, and transfer emotion distributions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 860–868.Google Scholar
Cross Ref
- [8] . 2010. Emotion related structures in large image databases. In Proceedings of the ACM International Conference on Image and Video Retrieval. 398–405. Google Scholar
Digital Library
- [9] . 2011. The geneva affective picture database (GAPED): A new 730-picture database focusing on valence and normative significance. Behav. Res. Methods 43, 2 (2011), 468–477.Google Scholar
Cross Ref
- [10] . 1999. International affective picture system (IAPS): Instruction manual and affective ratings. The Center for Research in Psychophysiology, University of Florida, Gainesville, FL.Google Scholar
- [11] . 1994. Measuring emotion: The self-assessment manikin and the semantic differential. J. Behav. Ther. Exper. Psych. 25, 1 (1994), 49–59.Google Scholar
- [12] . 2001. Color transfer between images. IEEE Comput. Graph. Appl. 21, 5 (2001), 34–41. Google Scholar
Digital Library
- [13] . 2002. Transferring color to greyscale images. In Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques. 277–280. Google Scholar
Digital Library
- [14] . 2006. Color transfer in correlated color space. In Proceedings of the ACM International Conference on Virtual Reality Continuum and Its Applications. 305–309. Google Scholar
Digital Library
- [15] . 2012. Example-based color transfer for gradient meshes. IEEE Trans. Multimedia 15, 3 (2012), 549–560. Google Scholar
Digital Library
- [16] . 2014. Video color conceptualization using optimization. Sci. China Info. Sci. 57, 7 (2014), 1–11.Google Scholar
Cross Ref
- [17] . 2016. Cartoon and texture decomposition-based color transfer for fabric images. IEEE Trans. Multimedia 19, 1 (2016), 80–92. Google Scholar
Digital Library
- [18] . 2016. Sufficient image appearance transfer combining color and texture. IEEE Trans. Multimedia 19, 4 (2016), 702–711. Google Scholar
Digital Library
- [19] . 2016. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2414–2423.Google Scholar
Cross Ref
- [20] . 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision. 2223–2232.Google Scholar
Cross Ref
- [21] . 2020. Learning representations for high-dynamic-range image color transfer in a self-supervised Way. IEEE Trans. Multimedia 23 (2020), 176–188.Google Scholar
Cross Ref
- [22] . 2015. Image color transfer to evoke different emotions based on color combinations. Signal Image Video Process. 9, 8 (2015), 1965–1973.Google Scholar
Cross Ref
- [23] . 2019. Emotion-based color transfer of images using adjustable color combinations. Soft Comput. 23, 3 (2019), 1007–1020. Google Scholar
Digital Library
- [24] . 2018. Emotional image color transfer via deep learning. Pattern Recogn. Lett. 110 (2018), 16–22.Google Scholar
Cross Ref
- [25] . 1986. A new pan-cultural facial expression of emotion. Motivat. Emot. 10, 2 (1986), 159–168.Google Scholar
Cross Ref
- [26] . 2009. Facial expression recognition based on local binary patterns: A comprehensive study. Image Vision Comput. 27, 6 (2009), 803–816. Google Scholar
Digital Library
- [27] . 2014. Deeply learning deformable facial action parts model for dynamic expression analysis. In Proceedings of the Asian Conference on Computer Vision. 143–157.Google Scholar
- [28] . 2018. Grey wolf optimisation-based feature selection and classification for facial emotion recognition. IET Biometrics 7, 5 (2018), 490–499.Google Scholar
Cross Ref
- [29] . 2017. Joint image emotion classification and distribution learning via deep convolutional neural network. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’17). 3266–3272. Google Scholar
Digital Library
- [30] . 2018. Expnet: Landmark-free, deep, 3D facial expressions. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition (FG’18). 122–129.Google Scholar
Cross Ref
- [31] . 2008. Emotional valence categorization using holistic image features. In Proceedings of the IEEE International Conference on Image Processing. 101–104.Google Scholar
Cross Ref
- [32] . 2016. Recognizing emotions from abstract paintings using non-linear matrix completion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5240–5248.Google Scholar
Cross Ref
- [33] . 2016. Multi-scale blocks based image emotion classification using multiple instance learning. In Proceedings of the IEEE International Conference on Image Processing (ICIP). 634–638.Google Scholar
Cross Ref
- [34] . 2018. Discrete probability distribution prediction of image emotions with shared sparse learning. IEEE Trans. Affect. Comput. 11, 4 (2018), 574–587.Google Scholar
Digital Library
- [35] . 2018. Building emotional machines: Recognizing image emotions through deep neural networks. IEEE Trans. Multimedia 20, 11 (2018), 2980–2992.Google Scholar
Digital Library
- [36] . 2020. Sentiment analysis using deep learning approaches: An overview. Sci. China Info. Sci. 63, 1 (2020), 1–36.Google Scholar
- [37] . 1992. An argument for basic emotions. Cogn. Emot. 6, 3–4 (1992), 169–200.Google Scholar
Cross Ref
- [38] . 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the International Conference on Machine Learning. 448–456. Google Scholar
Digital Library
- [39] . 2013. Interpretable aesthetic features for affective image classification. In Proceedings of the IEEE International Conference on Image Processing. 3230–3234.Google Scholar
Cross Ref
- [40] . 2010. Affective image classification using features inspired by psychology and art theory. In Proceedings of the ACM International Conference on Multimedia. 83–92. Google Scholar
Digital Library
- [41] . 2010. The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 94–101.Google Scholar
Cross Ref
Index Terms
Facial-expression-aware Emotional Color Transfer Based on Convolutional Neural Network
Recommendations
Facial-expression-aware emotional color transfer
PG '18: Proceedings of the 26th Pacific Conference on Computer Graphics and Applications: PostersEmotional color transfer aims to change the evoked emotion of the source image to that of the target image by adjusting color distribution. Most of existing emotional color transfer methods ignore the facial expression features in the image. Therefore, ...
Emotional facial expression interface: effects of displayed facial designs
Ergo'IA '10: Proceedings of the Ergonomie et Informatique Avancee ConferenceIn supporting the visual-mediated emotional recognition, little research has centred on analysing the effect of presenting different combinatorial facial designs. Moreover, in the theoretical field of emotions, research on the recognition of emotional ...
Cross-Domain Color Facial Expression Recognition Using Transductive Transfer Subspace Learning
Facial expression recognition across domains, e.g., training and testing facial images come from different facial poses, is very challenging due to the different marginal distributions between training and testing facial feature vectors. To deal with ...






Comments