skip to main content
research-article

Aesthetic Attribute Assessment of Images Numerically on Mixed Multi-attribute Datasets

Published:20 March 2023Publication History
Skip Abstract Section

Abstract

With the continuous development of social software and multimedia technology, images have become a kind of important carrier for spreading information and socializing. How to evaluate an image comprehensively has become the focus of recent researches. The traditional image aesthetic assessment methods often adopt single numerical overall assessment scores, which has certain subjectivity and can no longer meet the higher aesthetic requirements. In this article, we construct an new image attribute dataset called aesthetic mixed dataset with attributes (AMD-A) and design external attribute features for fusion. Besides, we propose an efficient method for image aesthetic attribute assessment on mixed multi-attribute dataset and construct a multitasking network architecture by using the EfficientNet-B0 as the backbone network. Our model can achieve aesthetic classification, overall scoring, and attribute scoring. In each sub-network, we improve the feature extraction through ECA channel attention module. As for the final overall scoring, we adopt the idea of the teacher-student network and use the classification sub-network to guide the aesthetic overall fine-grain regression. Experimental results, using the MindSpore, show that our proposed method can effectively improve the performance of the aesthetic overall and attribute assessment.

REFERENCES

  1. [1] Azimi Javad, Zhang Ruofei, Zhou Yang, Navalpakkam Vidhya, Mao Jianchang, and Fern Xiaoli. 2012. The impact of visual appearance on user response in online display advertising. In 21st International Conference on World Wide Web. 457458.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. [2] Bell Sean, Zitnick C. Lawrence, Bala Kavita, and Girshick Ross. 2016. Inside-Outside Net: Detecting objects in context with skip pooling and recurrent neural networks. In IEEE Conference on Computer Vision and Pattern Recognition. 28742883.Google ScholarGoogle ScholarCross RefCross Ref
  3. [3] Brachmann Anselm and Redies Christoph. 2017. Computational and experimental approaches to visual aesthetics. Front. Computat. Neurosci. 11 (2017), 102.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Chang Kuang-Yu, Lu Kung-Hung, and Chen Chu-Song. 2017. Aesthetic critiques generation for photos. In IEEE International Conference on Computer Vision. 35143523.Google ScholarGoogle Scholar
  5. [5] Datta Ritendra, Joshi Dhiraj, Li Jia, and Wang James Z.. 2006. Studying aesthetics in photographic images using a computational approach. In European Conference on Computer Vision. Springer, 288301.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. [6] Deng Yubin, Loy Chen Change, and Tang Xiaoou. 2018. Aesthetic-driven image enhancement by adversarial learning. In 26th ACM International Conference on Multimedia. 870878.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. [7] Fang Yuming, Zhu Hanwei, Zeng Yan, Ma Kede, and Wang Zhou. 2020. Perceptual quality assessment of smartphone photography. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. 36773686.Google ScholarGoogle ScholarCross RefCross Ref
  8. [8] Han Qi, Zhao Kai, Xu Jun, and Cheng Ming-Ming. 2020. Deep Hough transform for semantic line detection. In European Conference on Computer Vision (ECCV). 249265. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Hinton Geoffrey, Vinyals Oriol, Dean Jeff, et al. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 2, 7 (2015).Google ScholarGoogle Scholar
  10. [10] Huang Gao, Liu Zhuang, Maaten Laurens Van Der, and Weinberger Kilian Q.. 2017. Densely connected convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition. 47004708.Google ScholarGoogle ScholarCross RefCross Ref
  11. [11] Jin Xin, Wu Le, Zhou Xinghui, Zhao Geng, Zhang Xiaokun, Li Xiaodong, and Ge Shiming. 2018. Predicting aesthetic radar map using a hierarchical multi-task network. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV). Springer, 4150.Google ScholarGoogle ScholarCross RefCross Ref
  12. [12] Jin Xin, Zhou Xinghui, Li Xiaodong, Zhang Xiaokun, Sun Hongbo, Li Xiqiao, and Liu Ruijun. 2019. Incremental learning of multi-tasking networks for aesthetic radar map prediction. IEEE Access 7 (2019), 183647183655.Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Kairanbay Magzhan, See John, and Wong Lai-Kuan. 2019. Beauty is in the eye of the beholder: Demographically oriented analysis of aesthetics in photographs. ACM Trans. Multim. Comput., Commun. Applic. 15, 2s (2019), 121.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. [14] Kang Chen, Valenzise Giuseppe, and Dufaux Frédéric. 2020. EVA: An explainable visual aesthetics dataset. In Joint Workshop on Aesthetic and Technical Quality Assessment of Multimedia and Media Analytics for Societal Trends. 513.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. [15] Kao Yueying, Wang Chong, and Huang Kaiqi. 2015. Visual aesthetic quality assessment with a regression model. In IEEE International Conference on Image Processing (ICIP). IEEE, 15831587.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. [16] Ke Yan, Tang Xiaoou, and Jing Feng. 2006. The design of high-level features for photo quality assessment. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06). IEEE, 419426.Google ScholarGoogle Scholar
  17. [17] Kong Shu, Shen Xiaohui, Lin Zhe, Mech Radomir, and Fowlkes Charless. 2016. Photo aesthetics ranking network with attributes and content adaptation. In European Conference on Computer Vision. Springer, 662679.Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Kong Tao, Yao Anbang, Chen Yurong, and Sun Fuchun. 2016. HyperNet: Towards accurate region proposal generation and joint object detection. In IEEE Conference on Computer Vision and Pattern Recognition. 845853.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Liang Lingyu, Lin Luojun, Jin Lianwen, Xie Duorui, and Li Mengru. 2018. SCUT-FBP5500: A diverse benchmark dataset for multi-paradigm facial beauty prediction. In 24th International Conference on Pattern Recognition (ICPR). IEEE, 15981603.Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Lin Tsung-Yi, Dollár Piotr, Girshick Ross, He Kaiming, Hariharan Bharath, and Belongie Serge. 2017. Feature pyramid networks for object detection. In IEEE Conference on Computer Vision and Pattern Recognition. 21172125.Google ScholarGoogle ScholarCross RefCross Ref
  21. [21] Liu Jiang-Jiang, Hou Qibin, and Cheng Ming-Ming. 2020. Dynamic feature integration for simultaneous detection of salient object, edge, and skeleton. IEEE Trans. Image Process. 29 (2020), 86528667.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. [22] Liu Wei, Anguelov Dragomir, Erhan Dumitru, Szegedy Christian, Reed Scott, Fu Cheng-Yang, and Berg Alexander C.. 2016. SSD: Single shot multibox detector. In European Conference on Computer Vision. Springer, 2137.Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Luo Yiwen and Tang Xiaoou. 2008. Photo and video quality evaluation: Focusing on the subject. In European Conference on Computer Vision. Springer, 386399.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. [24] Ma Shuang, Liu Jing, and Chen Chang Wen. 2017. A-Lamp: Adaptive layout-aware multi-patch deep convolutional neural network for photo aesthetic assessment. In IEEE Conference on Computer Vision and Pattern Recognition. 45354544.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Marchesotti Luca, Perronnin Florent, Larlus Diane, and Csurka Gabriela. 2011. Assessing the aesthetic quality of photographs using generic image descriptors. In International Conference on Computer Vision. IEEE, 17841791.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. [26] MindSpore. 2022. Retrieved from https://www.mindspore.cn/.Google ScholarGoogle Scholar
  27. [27] Murray Naila, Marchesotti Luca, and Perronnin Florent. 2012. AVA: A large-scale database for aesthetic visual analysis. In IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 24082415.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. [28] Nishiyama Masashi, Okabe Takahiro, Sato Imari, and Sato Yoichi. 2011. Aesthetic quality classification of photographs based on color harmony. In IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 3340.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. [29] Wu Pengfei Zhu, Peihua Li, Wangmeng Zuo, Qilong Wang, Banggu, and Hu. Qinghua2020. ECA-Net: Efficient channel attention for deep convolutional neural networks. (2020), 1153111539.Google ScholarGoogle Scholar
  30. [30] She Dongyu, Lai Yu-Kun, Yi Gaoxiong, and Xu Kun. 2021. Hierarchical layout-aware graph convolutional network for unified aesthetics assessment. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. 84758484.Google ScholarGoogle ScholarCross RefCross Ref
  31. [31] Sperry R. W., Gazzaniga M. S., and Bogen J. E.. 1969. Interhemispheric relationships: The neocortical commissures; syndromes of hemisphere disconnection. Handb. Clin. Neurol. 4 (1969), 145153.Google ScholarGoogle Scholar
  32. [32] Tan Mingxing and Le Quoc. 2019. EfficientNet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning. PMLR, 61056114.Google ScholarGoogle Scholar
  33. [33] Tong Hanghang, Li Mingjing, Zhang Hong-Jiang, He Jingrui, and Zhang Changshui. 2004. Classification of digital photos taken by photographers or home users. In Pacific-Rim Conference on Multimedia. Springer, 198205.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. [34] Wang Ranran, Zhang Yin, Fortino Giancarlo, Guan Qingxu, Liu Jiangchuan, and Song Jeungeun. 2022. Software escalation prediction based on deep learning in the cognitive internet of vehicles. IEEE Trans. Intell. Transport. Syst. (2022).Google ScholarGoogle Scholar
  35. [35] Wang Ranran, Zhang Yin, Peng Limei, Fortino Giancarlo, and Ho Pin-Han. 2022. Time-varying-aware network traffic prediction via deep learning in IIoT. IEEE Trans. Industr. Inform. (2022).Google ScholarGoogle Scholar
  36. [36] Xu Liming, Zeng Xianhua, Li Weisheng, and Zheng Bochuan. 2022. MFGAN: Multi-modal feature-fusion for CT metal artifact reduction using GANs. ACM Trans. Multim. Comput., Commun. Applic. (2022).Google ScholarGoogle Scholar
  37. [37] Zhang Yin, Li Yujie, Wang Ranran, Lu Jianmin, Ma Xiao, and Qiu Meikang. 2020. PSAC: Proactive sequence-aware content caching via deep learning at the network edge. IEEE Trans. Netw. Sci. Eng. 7, 4 (2020), 21452154.Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Zhen Peining, Wang Shuqi, Zhang Suming, Yan Xiaotao, Wang Wei, Ji Zhigang, and Chen Hai-Bao. 2022. Towards accurate oriented object detection in aerial images with adaptive multi-level feature fusion. ACM Trans. Multim. Comput., Commun. Applic. (2022).Google ScholarGoogle Scholar
  39. [39] Zhou Quan, Wu Xiaofu, Zhang Suofei, Kang Bin, Ge Zongyuan, and Latecki Longin Jan. 2022. Contextual ensemble network for semantic segmentation. Pattern Recog. 122 (2022), 108290.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. [40] Zhou Quan, Yang Wenbing, Gao Guangwei, Ou Weihua, Lu Huimin, Chen Jie, and Latecki Longin Jan. 2019. Multi-scale deep context convolutional neural networks for semantic segmentation. World Wide Web 22, 2 (2019), 555570.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. [41] Zhou Wei, Xia Zhiwu, Dou Peng, Su Tao, and Hu Haifeng. 2022. Double attention based on graph attention network for image multi-label classification. ACM Trans. Multim. Comput., Commun. Applic. (2022).Google ScholarGoogle Scholar

Index Terms

  1. Aesthetic Attribute Assessment of Images Numerically on Mixed Multi-attribute Datasets

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Multimedia Computing, Communications, and Applications
      ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 18, Issue 3s
      October 2022
      381 pages
      ISSN:1551-6857
      EISSN:1551-6865
      DOI:10.1145/3567476
      • Editor:
      • Abdulmotaleb El Saddik
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 20 March 2023
      • Online AM: 6 February 2023
      • Accepted: 23 June 2022
      • Revised: 24 May 2022
      • Received: 4 December 2021
      Published in tomm Volume 18, Issue 3s

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
    • Article Metrics

      • Downloads (Last 12 months)106
      • Downloads (Last 6 weeks)30

      Other Metrics

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!