Abstract
Recent works in computer vision and multimedia have shown that image memorability can be automatically inferred exploiting powerful deep-learning models. This article advances the state of the art in this area by addressing a novel and more challenging issue: “Given an arbitrary input image, can we make it more memorable?” To tackle this problem, we introduce an approach based on an editing-by-applying-filters paradigm: given an input image, we propose to automatically retrieve a set of “style seeds,” i.e., a set of style images that, applied to the input image through a neural style transfer algorithm, provide the highest increase in memorability. We show the effectiveness of the proposed approach with experiments on the publicly available LaMem dataset, performing both a quantitative evaluation and a user study. To demonstrate the flexibility of the proposed framework, we also analyze the impact of different implementation choices, such as using different state-of-the-art neural style transfer methods. Finally, we show several qualitative results to provide additional insights on the link between image style and memorability.
- Peter P. Aitken. 1974. Judgments of pleasingness and interestingness as functions of visual complexity.J. Exper. Psychol. 103, 2 (1974), 240.Google Scholar
Cross Ref
- Afsheen Rafaqat Ali and Mohsen Ali. 2017. Automatic image transformation for inducing affect. In Proceedings of the British Machine Vision Conference (BMVC’17).Google Scholar
Cross Ref
- Daniel E. Berlyne. 1960. Conflict, Arousal, and Curiosity. McGraw-Hill Book Company.Google Scholar
- Daniel E. Berlyne. 1963. Complexity and incongruity variables as determinants of exploratory choice and evaluative ratings.Canadian J. Psychol./Revue 17, 3 (1963), 274.Google Scholar
- Zoya Bylinskii, Phillip Isola, Constance Bainbridge, Antonio Torralba, and Aude Oliva. 2015. Intrinsic and extrinsic effects on image memorability. Vision Res. 116 (2015), 165--178.Google Scholar
Cross Ref
- Alex J. Champandard. 2016. Semantic style transfer and turning two-bit doodles into fine artworks. arXiv preprint arXiv:1603.01768 (2016).Google Scholar
- Russell Eisenman. 1966. Pleasing and interesting visual complexity: Support for Berlyne. Percept. Motor Skills 23, 3 suppl. (1966), 1167--1170.Google Scholar
- Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16).Google Scholar
Cross Ref
- Alvin G. Goldstein and June E. Chance. 1971. Visual recognition memory for complex configurations. Atten. Percept. Psychophys. 9, 2 (1971), 237--241.Google Scholar
Cross Ref
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. MIT Press. Google Scholar
Digital Library
- Helmut Grabner, Fabian Nater, Michel Druey, and Luc Van Gool. 2013. Visual interestingness in image sequences. In Proceedings of the ACM Multimedia Conference. Google Scholar
Digital Library
- Michael Gygli, Helmut Grabner, Hayko Riemenschneider, Fabian Nater, and Luc Van Gool. 2013. The interestingness of images. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’13). 1633--1640. Google Scholar
Digital Library
- Raisa Halonen, Stina Westman, and Pirkko Oittinen. 2011. Naturalness and interestingness of test images for visual quality evaluation. In Proceedings of the IS&T/SPIE Electronic Imaging Conference. International Society for Optics and Photonics, 78670Z--78670Z.Google Scholar
Cross Ref
- Li He, Hairong Qi, and Russell Zaretzki. 2015. Image color transfer to evoke different emotions based on color combinations. Signal, Image Video Process. 9, 8 (2015), 1965--1973.Google Scholar
Cross Ref
- Xun Huang and Serge Belongie. 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17).Google Scholar
Cross Ref
- Juan Huo. 2016. An image complexity measurement algorithm with visual memory capacity and an EEG study. In Proceedings of the SAI Computing Conference (SAI’16). IEEE, 264--268.Google Scholar
Cross Ref
- Phillip Isola, Devi Parikh, Antonio Torralba, and Aude Oliva. 2011. Understanding the intrinsic memorability of images. In Advances in Neural Information Processing Systems. MIT Press. Google Scholar
Digital Library
- Phillip Isola, Jianxiong Xiao, Devi Parikh, Antonio Torralba, and Aude Oliva. 2014. What makes a photograph memorable?IEEE Trans. Pattern Anal. Mach. Intell. 36, 7 (2014), 1469--1482. Google Scholar
Digital Library
- Aditya Khosla. 2017. Predicting human behavior using visual media. http://hdl.handle.net/1721.1/109001.Google Scholar
- Aditya Khosla, Wilma Bainbridge, Antonio Torralba, and Aude Oliva. 2013. Modifying the memorability of face photographs. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’13). Google Scholar
Digital Library
- Aditya Khosla, Akhil S. Raju, Antonio Torralba, and Aude Oliva. 2015. Understanding and predicting image memorability at a large scale. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’15). Google Scholar
Digital Library
- Aditya Khosla, Jianxiong Xiao, Phillip Isola, Antonio Torralba, and Aude Oliva. 2012. Image memorability and visual inception. In Proceedings of the Conference and Exhibition on Computer Graphics 8 Interactive Techniques in Asia (SIGGRAPHAsia’12). ACM. Google Scholar
Digital Library
- Hye-Rin Kim, Henry Kang, and In-Kwon Lee. 2016. Image recoloring with valence-arousal emotion model. In Proceedings of the Computer Graphics Forum, vol. 35. 209--216. Google Scholar
Digital Library
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. MIT Press. Google Scholar
Digital Library
- James L. McGaugh and Larry Cahill. 1995. A novel demonstration of enhanced memory associated with emotional arousal. Consciousness and Cognition 4, 4 (1995), 410--421.Google Scholar
Cross Ref
- David R. Lide. 2018. Handbook of mathematical functions. In A Century of Excellence in Measurements, Standards, and Technology. CRC Press, 135--139.Google Scholar
- Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. 2017. Deep photo style transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17).Google Scholar
Cross Ref
- Jana Machajdik and Allan Hanbury. 2010. Affective image classification using features inspired by psychology and art theory. In Proceedings of the ACM International Conference on Multimedia. Google Scholar
Digital Library
- Stephen Maren. 1999. Long-term potentiation in the amygdala: A mechanism for emotional learning and memory. Trends Neurosci. 22, 12 (1999), 561--567.Google Scholar
Cross Ref
- Weijie Mao, Mengjuan Fei, and Wei Jiang. 2018. Creating memorable video summaries that satisfy the user’s intention for taking the videos. Neurocomputing 275 (2018), 1911--1920. Google Scholar
Digital Library
- Kuan-Chuan Peng, Tsuhan Chen, Amir Sadovnik, and Andrew C. Gallagher. 2015. A mixed bag of emotions: Model, predict, and transfer emotion distributions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15).Google Scholar
- Elizabeth A. Phelps. 2004. Human emotion and memory: Interactions of the amygdala and hippocampal complex. Curr. Opin. Neurobiol. 14, 2 (2004), 198--202.Google Scholar
Cross Ref
- Manuel Ruder, Alexey Dosovitskiy, and Thomas Brox. 2018. Artistic style transfer for videos and spherical images. Int. J. Comput. Vision 126, 11 (2018), 1199--1219. Google Scholar
Digital Library
- Andreza Sartori, Victoria Yanulevskaya, Almila Akdag Salah, Jasper Uijlings, Elia Bruni, and Nicu Sebe. 2015. Affective analysis of professional and amateur abstract paintings using statistical analysis and art theory. ACM Trans. Interact. Intell. Syst. 5, 2 (2015), 8. Google Scholar
Digital Library
- Sumit Shekhar, Srinivasa Madhava Phaneendra Angara, Manav Kedia, Dhruv Singal, and Akhil Sathyaprakash Shetty. 2017. Techniques for enhancing content memorability of user generated video content. U.S. Patent 9,805,269.Google Scholar
- Lu Sheng, Ziyi Lin, Jing Shao, and Xiaogang Wang. 2018. Avatar-net: Multi-scale zero-shot style transfer by feature decoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18).Google Scholar
Cross Ref
- Aliaksandr Siarohin, Gloria Zen, Cveta Majtanovic, Xavier Alameda-Pineda, Elisa Ricci, and Nicu Sebe. 2017. How to make an image more memorable? A deep style transfer approach. In Proceedings of the International Conference on Multimedia Retrieval (ICMR’17). Google Scholar
Digital Library
- Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In International Conference on Machine Learning(ICLR’15).Google Scholar
- Mohammad Soleymani. 2015. The quest for visual interest. In Proceedings of the ACM International Conference on Multimedia. Google Scholar
Digital Library
- Lionel Standing. 1973. Learning 10,000 pictures. Quart. J. Exp. Psychol. 25, 2 (1973), 207--222.Google Scholar
Cross Ref
- Lionel Standing, Jerry Conezio, and Ralph Norman Haber. 1970. Perception and memory for pictures: Single-trial learning of 2500 visual stimuli. Psychonom. Sci. 19, 2 (1970), 73--74.Google Scholar
Cross Ref
- Noah Sulman and Thomas Sanocki. 2011. Color relations increase the capacity of visual short-term memory. Perception 40, 6 (2011).Google Scholar
- Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor Lempitsky. 2016. Texture networks: Feed-forward synthesis of textures and stylized images. In Proceedings of the International Conference on Machine Learning (ICML’16). Google Scholar
Digital Library
- Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2017. Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17).Google Scholar
Cross Ref
- Wenguan Wang and Jianbing Shen. 2017. Deep cropping via attention box prediction and aesthetics assessment. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17).Google Scholar
Cross Ref
- Hang Zhang and Kristin Dana. 2018. Multi-style generative network for real-time transfer. In Proceedings of the European Conference on Computer Vision Workshops (ECCV’18).Google Scholar
- Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. 2014. Learning deep features for scene recognition using places database. In Advances in Neural Information Processing Systems. MIT Press. Google Scholar
Digital Library
Index Terms
Increasing Image Memorability with Neural Style Transfer
Recommendations
Aesthetic-Aware Image Style Transfer
MM '20: Proceedings of the 28th ACM International Conference on MultimediaStyle transfer aims to synthesize an image which inherits the content of one image while preserving a similar style of the other one. The "style'' of an image usually refers to its unique feeling conveyed from visual features, which is highly related to ...
Optimal Transport of Deep Feature for Image Style Transfer
ICMSSP '19: Proceedings of the 2019 4th International Conference on Multimedia Systems and Signal ProcessingImage style transfer is a classic image editing task which aims to transfer arbitrary visual styles to content images. In recent years, it has been revealed that a well-trained convolutional neural network with sufficient labeled data is powerful to ...
Edge Enhanced Image Style Transfer via Transformers
ICMR '23: Proceedings of the 2023 ACM International Conference on Multimedia RetrievalIn recent years, arbitrary image style transfer has attracted more and more attention. Given a pair of content and style images, a stylized one is hoped that retains the content from the former while catching style patterns from the latter. However, it ...






Comments