Abstract
Image aesthetics assessment aims to endow computers with the ability to judge the aesthetic values of images, and its potential has been recognized in a variety of applications. Most previous studies perform aesthetics assessment purely based on image content. However, given the fact that aesthetic perceiving is a human cognitive activity, it is necessary to consider users’ perception of an image when judging its aesthetic quality. In this article, we regard users’ social behavior as the reflection of their perception of images and harness these additional clues to improve image aesthetics assessment. Specifically, we first merge the raw social interactions between users and images into clusters as the social labels of images, so the collective social behavioral information associated with an image can be well represented over a structured and compact space. Then, we develop a novel deep multi-task network to jointly learn social labels in different modalities from social images and apply it to common web images. In this manner, our approach is readily generalized to web images without social behavioral information. Finally, we introduce a high-level fusion sub-network to the aesthetics model, in which the social and visual representations of images are well balanced for aesthetics assessment. Experimental results on two benchmark datasets well verify the effectiveness of our approach and highlight the benefits of different types of social behavioral information for image aesthetics assessment.
- Dolores Albarracin and Robert S. Wyer Jr. 2000. The cognitive impact of past behavior: Influences on beliefs, attitudes, and future behavioral decisions.J. Person. Social Psychol. 79, 1 (2000), 5.Google Scholar
Cross Ref
- Samiul Azam and Marina Gavrilova. 2016. Soft biometric: Give me your favorite images and I will tell your gender. In Proceedings of the 15th IEEE International Conference on Cognitive Informatics 8 Cognitive Computing. 535--541.Google Scholar
Cross Ref
- Chaoran Cui, Huiui Liu, Tao Lian, Liqiang Nie, Lei Zhu, and Yilong Yin. 2019. Distribution-oriented aesthetics assessment with semantic-aware hybrid network. IEEE Trans. Multim. 21, 5 (2019), 1209--1220.Google Scholar
Digital Library
- Chaoran Cui, Wenya Yang, Cheng Shi, Meng Wang, Xiushan Nie, and Yilong Yin. 2020. Personalized image quality assessment with social-sensed aesthetic preference. Inf. Sci. 512 (2020), 780--794.Google Scholar
Digital Library
- Peng Cui, Shao-Wei Liu, Wen-Wu Zhu, Huan-Bo Luan, Tat-Seng Chua, and Shi-Qiang Yang. 2014. Social-sensed image search. ACM Trans. Inf. Syst. 32, 2 (2014), 8.Google Scholar
Digital Library
- Peng Cui, Wenwu Zhu, Tat-Seng Chua, and Ramesh Jain. 2016. Social-sensed multimedia computing. IEEE Multim. 23, 1 (2016), 92--96.Google Scholar
Digital Library
- Ritendra Datta, Dhiraj Joshi, Jia Li, and James Z. Wang. 2006. Studying aesthetics in photographic images using a computational approach. In Proceedings of the 8th European Conference on Computer Vision. 288--301.Google Scholar
- Yubin Deng, Chen Change Loy, and Xiaoou Tang. 2017. Image aesthetic assessment: An experimental survey. IEEE Sig. Proc. Mag. 34, 4 (2017), 80--106.Google Scholar
Cross Ref
- Sagnik Dhar, Vicente Ordonez, and Tamara L. Berg. 2011. High level describable attributes for predicting aesthetics and interestingness. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1657--1664.Google Scholar
- Bin-Bin Gao, Chao Xing, Chen-Wei Xie, Jianxin Wu, and Xin Geng. 2017. Deep label distribution learning with label ambiguity. IEEE Trans. Image Proc. 26, 6 (2017), 2825--2838.Google Scholar
Digital Library
- Yuan Gao, Jiayi Ma, Mingbo Zhao, Wei Liu, and Alan L. Yuille. 2019. NDDR-CNN: Layerwise feature fusing in multi-task CNNs by neural discriminative dimensionality reduction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3205--3214.Google Scholar
- Bo Geng, Linjun Yang, Chao Xu, Xian-Sheng Hua, and Shipeng Li. 2011. The role of attractiveness in web image search. In Proceedings of the 19th ACM International Conference on Multimedia. 63--72.Google Scholar
Digital Library
- Ross Girshick. 2015. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision. 1440--1448.Google Scholar
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770--778.Google Scholar
Cross Ref
- Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7132--7141.Google Scholar
Cross Ref
- Xin Jin, Le Wu, Xiaodong Li, Siyu Chen, Siwei Peng, Jingying Chi, Shiming Ge, Chenggen Song, and Geng Zhao. 2018. Predicting aesthetic score distribution through cumulative Jensen-Shannon divergence. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence. 77--84.Google Scholar
- Dhiraj Joshi, Ritendra Datta, Elena Fedorovskaya, Quang-Tuan Luong, James Z. Wang, Jia Li, and Jiebo Luo. 2011. Aesthetics and emotions in images. IEEE Sig. Proc. Mag. 28, 5 (2011), 94--115.Google Scholar
Cross Ref
- Yueying Kao, Ran He, and Kaiqi Huang. 2017. Deep aesthetic quality assessment with semantic information. IEEE Trans. Image Proc. 26, 3 (2017), 1482--1495.Google Scholar
Digital Library
- Rei Kawakami, Ryota Yoshihashi, Seiichiro Fukuda, Shaodi You, Makoto Iida, and Takeshi Naemura. 2019. Cross-connected networks for multi-task learning of detection and segmentation. In Proceedings of the IEEE International Conference on Image Processing. IEEE, 3636--3640.Google Scholar
Cross Ref
- Yan Ke, Xiaoou Tang, and Feng Jing. 2006. The design of high-level features for photo quality assessment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 419--426.Google Scholar
- Alex Kendall, Yarin Gal, and Roberto Cipolla. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7482--7491.Google Scholar
- Aditya Khosla, Atish Das Sarma, and Raffay Hamid. 2014. What makes an image popular? In Proceedings of the 23rd International Conference on World Wide Web. 867--876.Google Scholar
Digital Library
- Shu Kong, Xiaohui Shen, Zhe Lin, Radomir Mech, and Charless Fowlkes. 2016. Photo aesthetics ranking network with attributes and content adaptation. In Proceedings of the 14th European Conference on Computer Vision. 662--679.Google Scholar
Cross Ref
- Michal Kucer, Alexander C. Loui, and David W. Messinger. 2018. Leveraging expert feature knowledge for predicting image aesthetics. IEEE Trans. Image Proc. 27, 10 (2018), 5100--5112.Google Scholar
Cross Ref
- Anan Liu, Yuting Su, Weizhi Nie, and Mohan Kankanhalli. 2017. Hierarchical clustering multi-task learning for joint human action grouping and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39, 1 (2017), 102--114.Google Scholar
Digital Library
- Anan Liu, Ning Xu, Hanwang Zhang, Weizhi Nie, Yuting Su, and Yongdong Zhang. 2018. Multi-level policy and reward reinforcement learning for image captioning. In Proceedings of the 27th International Joint Conference on Artificial Intelligence. 821--827.Google Scholar
Digital Library
- Shaowei Liu, Peng Cui, Wenwu Zhu, and Shiqiang Yang. 2015. Learning socially embedded visual representation from scratch. In Proceedings of the 23rd ACM International Conference on Multimedia. 109--118.Google Scholar
Digital Library
- Shaowei Liu, Peng Cui, Wenwu Zhu, Shiqiang Yang, and Qi Tian. 2014. Social embedding image distance learning. In Proceedings of the 22nd ACM International Conference on Multimedia. 617--626.Google Scholar
Digital Library
- Shikun Liu, Edward Johns, and Andrew J. Davison. 2019. End-to-end multi-task learning with attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1871--1880.Google Scholar
- Pietro Lovato, Manuele Bicego, Cristina Segalin, Alessandro Perina, Nicu Sebe, and Marco Cristani. 2014. Faved! biometrics: Tell me which image you like and I’ll tell you who you are. IEEE Trans. Inf. Forens. Secur. 9, 3 (2014), 364--374.Google Scholar
Digital Library
- Xin Lu, Zhe Lin, Hailin Jin, Jianchao Yang, and James Z. Wang. 2015. Rating image aesthetics using deep learning. IEEE Trans. Multim. 17, 11 (2015), 2021--2034.Google Scholar
Cross Ref
- Xin Lu, Zhe Lin, Xiaohui Shen, Radomir Mech, and James Z. Wang. 2015. Deep multi-patch aggregation network for image style, aesthetics, and quality estimation. In Proceedings of the IEEE International Conference on Computer Vision. 990--998.Google Scholar
- Yiwen Luo and Xiaoou Tang. 2008. Photo and video quality evaluation: Focusing on the subject. In Proceedings of the European Conference on Computer Vision. 386--399.Google Scholar
Digital Library
- Shuang Ma, Jing Liu, and Chang Wen Chen. 2017. A-Lamp: Adaptive layout-aware multi-patch deep convolutional neural network for photo aesthetic assessment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4535--4544.Google Scholar
Cross Ref
- Long Mai, Hailin Jin, and Feng Liu. 2016. Composition-preserving deep photo aesthetics assessment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 497--506.Google Scholar
Cross Ref
- Luca Marchesotti, Florent Perronnin, Diane Larlus, and Gabriela Csurka. 2011. Assessing the aesthetic quality of photographs using generic image descriptors. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, 1784--1791.Google Scholar
Digital Library
- Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. 2016. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3994--4003.Google Scholar
Cross Ref
- Naila Murray, Luca Marchesotti, and Florent Perronnin. 2012. AVA: A large-scale database for aesthetic visual analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2408--2415.Google Scholar
Digital Library
- Radu Andrei Negoescu and Daniel Gatica-Perez. 2008. Analyzing flickr groups. In Proceedings of the International Conference on Content-based Image and Video Retrieval. 417--426.Google Scholar
Digital Library
- Liqiang Nie, Yongqi Li, Fuli Feng, Xuemeng Song, Meng Wang, and Yinglong Wang. 2020. Large-scale question tagging via joint question-topic embedding learning. ACM Trans. Inf. Syst. 38, 2 (2020), 1--23.Google Scholar
Digital Library
- Liqiang Nie, Meng Liu, and Xuemeng Song. 2019. Multimodal learning toward micro-video understanding. Synth. Lect. Image, Vid. Multim. Proc. 9, 4 (2019), 1--186.Google Scholar
Cross Ref
- Masashi Nishiyama, Takahiro Okabe, Imari Sato, and Yoichi Sato. 2011. Aesthetic quality classification of photographs based on color harmony. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 33--40.Google Scholar
Digital Library
- Jian Ren, Xiaohui Shen, Zhe Lin, Radomir Mech, and David J. Foran. 2017. Personalized image aesthetics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 638--647.Google Scholar
- Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. Arxiv Preprint Arxiv:1706.05098 (2017).Google Scholar
- Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. 2017. Sluice networks: Learning what to share between loosely related tasks. Stat 1050 (2017), 23.Google Scholar
- Cristina Segalin, Alessandro Perina, Marco Cristani, and Alessandro Vinciarelli. 2017. The pictures we like are our image: Continuous mapping of favorite pictures into self-assessed and attributed personality traits. IEEE Trans. Affect. Comput. 8, 2 (2017), 268--285.Google Scholar
Digital Library
- Kekai Sheng, Weiming Dong, Chongyang Ma, Xing Mei, Feiyue Huang, and Bao-Gang Hu. 2018. Attention-based multi-Patch aggregation for image aesthetic assessment. In Proceedings of the 26th ACM International Conference on Multimedia. 879--886.Google Scholar
Digital Library
- Hossein Talebi and Peyman Milanfar. 2018. NIMA: Neural image assessment. IEEE Trans. Image Proc. 27, 8 (2018), 3998--4011.Google Scholar
Cross Ref
- Xiaoou Tang, Wei Luo, and Xiaogang Wang. 2013. Content-based photo quality assessment. IEEE Trans. Multim. 15, 8 (2013), 1930--1943.Google Scholar
Digital Library
- Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 2016. YFCC100M: The new data in multimedia research. Commun. ACM 59, 2 (2016), 64--73.Google Scholar
Digital Library
- Roelof van Zwol, Adam Rae, and Lluis Garcia Pueyo. 2010. Prediction of favourite photos using social, visual, and textual signals. In Proceedings of the 18th ACM International Conference on Multimedia. 1015--1018.Google Scholar
Digital Library
- Yanxiang Wang, Hari Sundaram, and Lexing Xie. 2012. Social event detection with interaction graph modeling. In Proceedings of the 20th ACM International Conference on Multimedia. 865--868.Google Scholar
Digital Library
- Yinwei Wei, Xiang Wang, Weili Guan, Liqiang Nie, Zhouchen Lin, and Baoquan Chen. 2020. Neural multimodal cooperative learning toward micro-video understanding. IEEE Trans. Image Proc. 29 (2020), 1--14.Google Scholar
Cross Ref
- Ning Xu, Anan Liu, Yongkang Wong, Yongdong Zhang, Weizhi Nie, Yuting Su, and Mohan Kankanhalli. 2019. Dual-stream recurrent neural network for video captioning. IEEE Trans. Circ. Syst. Vid. Technol. 29, 8 (2019), 2482--2493.Google Scholar
Cross Ref
- Yi-Liang Zhao, Qiang Chen, Shuicheng Yan, Tat-Seng Chua, and Daqing Zhang. 2013. Detecting profilable and overlapping communities with user-generated multimedia contents in LBSNs. ACM Trans. Multim. Comput. Commun. Applic. 10, 1 (2013), 3.Google Scholar
Index Terms
Social-sensed Image Aesthetics Assessment
Recommendations
Personalized Image Aesthetics Assessment
CIKM '17: Proceedings of the 2017 ACM on Conference on Information and Knowledge ManagementAutomatically assessing image quality from an aesthetic perspective is of great interest to the high-level vision research community. Existing methods are typically non-personalized and quantify image aesthetics with a universal label. However, given ...
Personalized image quality assessment with Social-Sensed aesthetic preference
AbstractPersonalization is emerging as a key research objective for image aesthetics assessment, and how to incorporate personal preferences into aesthetics models is a crucial issue to be solved. Prior studies usually require users to ...
Technological Development of Image Aesthetics Assessment
Image and GraphicsAbstractQuantitative research on aesthetics is a classic interdisciplinary research. With the rapid development of deep learning, various approaches have been made in image aesthetics assessment (IAA). Starting from the concept of image aesthetics, this ...






Comments