skip to main content
research-article

Increasing Image Memorability with Neural Style Transfer

Published:05 June 2019Publication History
Skip Abstract Section

Abstract

Recent works in computer vision and multimedia have shown that image memorability can be automatically inferred exploiting powerful deep-learning models. This article advances the state of the art in this area by addressing a novel and more challenging issue: “Given an arbitrary input image, can we make it more memorable?” To tackle this problem, we introduce an approach based on an editing-by-applying-filters paradigm: given an input image, we propose to automatically retrieve a set of “style seeds,” i.e., a set of style images that, applied to the input image through a neural style transfer algorithm, provide the highest increase in memorability. We show the effectiveness of the proposed approach with experiments on the publicly available LaMem dataset, performing both a quantitative evaluation and a user study. To demonstrate the flexibility of the proposed framework, we also analyze the impact of different implementation choices, such as using different state-of-the-art neural style transfer methods. Finally, we show several qualitative results to provide additional insights on the link between image style and memorability.

References

  1. Peter P. Aitken. 1974. Judgments of pleasingness and interestingness as functions of visual complexity.J. Exper. Psychol. 103, 2 (1974), 240.Google ScholarGoogle ScholarCross RefCross Ref
  2. Afsheen Rafaqat Ali and Mohsen Ali. 2017. Automatic image transformation for inducing affect. In Proceedings of the British Machine Vision Conference (BMVC’17).Google ScholarGoogle ScholarCross RefCross Ref
  3. Daniel E. Berlyne. 1960. Conflict, Arousal, and Curiosity. McGraw-Hill Book Company.Google ScholarGoogle Scholar
  4. Daniel E. Berlyne. 1963. Complexity and incongruity variables as determinants of exploratory choice and evaluative ratings.Canadian J. Psychol./Revue 17, 3 (1963), 274.Google ScholarGoogle Scholar
  5. Zoya Bylinskii, Phillip Isola, Constance Bainbridge, Antonio Torralba, and Aude Oliva. 2015. Intrinsic and extrinsic effects on image memorability. Vision Res. 116 (2015), 165--178.Google ScholarGoogle ScholarCross RefCross Ref
  6. Alex J. Champandard. 2016. Semantic style transfer and turning two-bit doodles into fine artworks. arXiv preprint arXiv:1603.01768 (2016).Google ScholarGoogle Scholar
  7. Russell Eisenman. 1966. Pleasing and interesting visual complexity: Support for Berlyne. Percept. Motor Skills 23, 3 suppl. (1966), 1167--1170.Google ScholarGoogle Scholar
  8. Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16).Google ScholarGoogle ScholarCross RefCross Ref
  9. Alvin G. Goldstein and June E. Chance. 1971. Visual recognition memory for complex configurations. Atten. Percept. Psychophys. 9, 2 (1971), 237--241.Google ScholarGoogle ScholarCross RefCross Ref
  10. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. MIT Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Helmut Grabner, Fabian Nater, Michel Druey, and Luc Van Gool. 2013. Visual interestingness in image sequences. In Proceedings of the ACM Multimedia Conference. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Michael Gygli, Helmut Grabner, Hayko Riemenschneider, Fabian Nater, and Luc Van Gool. 2013. The interestingness of images. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’13). 1633--1640. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Raisa Halonen, Stina Westman, and Pirkko Oittinen. 2011. Naturalness and interestingness of test images for visual quality evaluation. In Proceedings of the IS&T/SPIE Electronic Imaging Conference. International Society for Optics and Photonics, 78670Z--78670Z.Google ScholarGoogle ScholarCross RefCross Ref
  14. Li He, Hairong Qi, and Russell Zaretzki. 2015. Image color transfer to evoke different emotions based on color combinations. Signal, Image Video Process. 9, 8 (2015), 1965--1973.Google ScholarGoogle ScholarCross RefCross Ref
  15. Xun Huang and Serge Belongie. 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17).Google ScholarGoogle ScholarCross RefCross Ref
  16. Juan Huo. 2016. An image complexity measurement algorithm with visual memory capacity and an EEG study. In Proceedings of the SAI Computing Conference (SAI’16). IEEE, 264--268.Google ScholarGoogle ScholarCross RefCross Ref
  17. Phillip Isola, Devi Parikh, Antonio Torralba, and Aude Oliva. 2011. Understanding the intrinsic memorability of images. In Advances in Neural Information Processing Systems. MIT Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Phillip Isola, Jianxiong Xiao, Devi Parikh, Antonio Torralba, and Aude Oliva. 2014. What makes a photograph memorable?IEEE Trans. Pattern Anal. Mach. Intell. 36, 7 (2014), 1469--1482. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Aditya Khosla. 2017. Predicting human behavior using visual media. http://hdl.handle.net/1721.1/109001.Google ScholarGoogle Scholar
  20. Aditya Khosla, Wilma Bainbridge, Antonio Torralba, and Aude Oliva. 2013. Modifying the memorability of face photographs. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’13). Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Aditya Khosla, Akhil S. Raju, Antonio Torralba, and Aude Oliva. 2015. Understanding and predicting image memorability at a large scale. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’15). Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Aditya Khosla, Jianxiong Xiao, Phillip Isola, Antonio Torralba, and Aude Oliva. 2012. Image memorability and visual inception. In Proceedings of the Conference and Exhibition on Computer Graphics 8 Interactive Techniques in Asia (SIGGRAPHAsia’12). ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Hye-Rin Kim, Henry Kang, and In-Kwon Lee. 2016. Image recoloring with valence-arousal emotion model. In Proceedings of the Computer Graphics Forum, vol. 35. 209--216. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. MIT Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. James L. McGaugh and Larry Cahill. 1995. A novel demonstration of enhanced memory associated with emotional arousal. Consciousness and Cognition 4, 4 (1995), 410--421.Google ScholarGoogle ScholarCross RefCross Ref
  26. David R. Lide. 2018. Handbook of mathematical functions. In A Century of Excellence in Measurements, Standards, and Technology. CRC Press, 135--139.Google ScholarGoogle Scholar
  27. Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. 2017. Deep photo style transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17).Google ScholarGoogle ScholarCross RefCross Ref
  28. Jana Machajdik and Allan Hanbury. 2010. Affective image classification using features inspired by psychology and art theory. In Proceedings of the ACM International Conference on Multimedia. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Stephen Maren. 1999. Long-term potentiation in the amygdala: A mechanism for emotional learning and memory. Trends Neurosci. 22, 12 (1999), 561--567.Google ScholarGoogle ScholarCross RefCross Ref
  30. Weijie Mao, Mengjuan Fei, and Wei Jiang. 2018. Creating memorable video summaries that satisfy the user’s intention for taking the videos. Neurocomputing 275 (2018), 1911--1920. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Kuan-Chuan Peng, Tsuhan Chen, Amir Sadovnik, and Andrew C. Gallagher. 2015. A mixed bag of emotions: Model, predict, and transfer emotion distributions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15).Google ScholarGoogle Scholar
  32. Elizabeth A. Phelps. 2004. Human emotion and memory: Interactions of the amygdala and hippocampal complex. Curr. Opin. Neurobiol. 14, 2 (2004), 198--202.Google ScholarGoogle ScholarCross RefCross Ref
  33. Manuel Ruder, Alexey Dosovitskiy, and Thomas Brox. 2018. Artistic style transfer for videos and spherical images. Int. J. Comput. Vision 126, 11 (2018), 1199--1219. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Andreza Sartori, Victoria Yanulevskaya, Almila Akdag Salah, Jasper Uijlings, Elia Bruni, and Nicu Sebe. 2015. Affective analysis of professional and amateur abstract paintings using statistical analysis and art theory. ACM Trans. Interact. Intell. Syst. 5, 2 (2015), 8. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Sumit Shekhar, Srinivasa Madhava Phaneendra Angara, Manav Kedia, Dhruv Singal, and Akhil Sathyaprakash Shetty. 2017. Techniques for enhancing content memorability of user generated video content. U.S. Patent 9,805,269.Google ScholarGoogle Scholar
  36. Lu Sheng, Ziyi Lin, Jing Shao, and Xiaogang Wang. 2018. Avatar-net: Multi-scale zero-shot style transfer by feature decoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18).Google ScholarGoogle ScholarCross RefCross Ref
  37. Aliaksandr Siarohin, Gloria Zen, Cveta Majtanovic, Xavier Alameda-Pineda, Elisa Ricci, and Nicu Sebe. 2017. How to make an image more memorable? A deep style transfer approach. In Proceedings of the International Conference on Multimedia Retrieval (ICMR’17). Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In International Conference on Machine Learning(ICLR’15).Google ScholarGoogle Scholar
  39. Mohammad Soleymani. 2015. The quest for visual interest. In Proceedings of the ACM International Conference on Multimedia. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Lionel Standing. 1973. Learning 10,000 pictures. Quart. J. Exp. Psychol. 25, 2 (1973), 207--222.Google ScholarGoogle ScholarCross RefCross Ref
  41. Lionel Standing, Jerry Conezio, and Ralph Norman Haber. 1970. Perception and memory for pictures: Single-trial learning of 2500 visual stimuli. Psychonom. Sci. 19, 2 (1970), 73--74.Google ScholarGoogle ScholarCross RefCross Ref
  42. Noah Sulman and Thomas Sanocki. 2011. Color relations increase the capacity of visual short-term memory. Perception 40, 6 (2011).Google ScholarGoogle Scholar
  43. Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor Lempitsky. 2016. Texture networks: Feed-forward synthesis of textures and stylized images. In Proceedings of the International Conference on Machine Learning (ICML’16). Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2017. Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17).Google ScholarGoogle ScholarCross RefCross Ref
  45. Wenguan Wang and Jianbing Shen. 2017. Deep cropping via attention box prediction and aesthetics assessment. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17).Google ScholarGoogle ScholarCross RefCross Ref
  46. Hang Zhang and Kristin Dana. 2018. Multi-style generative network for real-time transfer. In Proceedings of the European Conference on Computer Vision Workshops (ECCV’18).Google ScholarGoogle Scholar
  47. Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. 2014. Learning deep features for scene recognition using places database. In Advances in Neural Information Processing Systems. MIT Press. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Increasing Image Memorability with Neural Style Transfer

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Multimedia Computing, Communications, and Applications
      ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 15, Issue 2
      May 2019
      375 pages
      ISSN:1551-6857
      EISSN:1551-6865
      DOI:10.1145/3339884
      Issue’s Table of Contents

      Copyright © 2019 ACM

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 5 June 2019
      • Revised: 1 February 2019
      • Accepted: 1 February 2019
      • Received: 1 May 2018
      Published in tomm Volume 15, Issue 2

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Research
      • Refereed

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!