Abstract
In this article, we propose to predict human eye fixation through incorporating both audio and visual cues. Traditional visual attention models generally make the utmost of stimuli’s visual features, yet they bypass all audio information. In the real world, however, we not only direct our gaze according to visual saliency, but also are attracted by salient audio cues. Psychological experiments show that audio has an influence on visual attention, and subjects tend to be attracted by the sound sources. Therefore, we propose fusing both audio and visual information to predict eye fixation. In our proposed framework, we first localize the moving--sound-generating objects through multimodal analysis and generate an audio attention map. Then, we calculate the spatial and temporal attention maps using the visual modality. Finally, the audio, spatial, and temporal attention maps are fused to generate the final audiovisual saliency map. The proposed method is applicable to scenes containing moving--sound-generating objects. We gather a set of video sequences and collect eye-tracking data under an audiovisual test condition. Experiment results show that we can achieve better eye fixation prediction performance when taking both audio and visual cues into consideration, especially in some typical scenes in which object motion and audio are highly correlated.
- Ravi Achanta, Sheila Hemami, Francisco Estrada, and Sabine Susstrunk. 2009. Frequency-tuned salient region detection. In Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition. 1597--1604. Google Scholar
Cross Ref
- Bogdan Alexe, Thomas Deselaers, and Vittorio Ferrari. 2012. Measuring the objectness of image windows. IEEE Transactions on Pattern Analysis and Machine Intelligence 34, 11, 2189--2202. Google Scholar
Digital Library
- Xuan Bao and Romit Roy Choudhury. 2010. Movi: Mobile phone based video highlights via collaborative sensing. In Proceedings of the 8th International Conference on Mobile Systems, Applications, and Services. ACM, 357--370. Google Scholar
Digital Library
- Zohar Barzelay and Yoav Y. Schechner. 2007. Harmony in motion. In Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition. 1--8. Google Scholar
Cross Ref
- Ali Borji, Ming-Ming Cheng, Huaizu Jiang, and Jia Li. 2014. Salient object detection: A survey. ArXiv Preprint.Google Scholar
- Ali Borji and Laurent Itti. 2013. State-of-the-art in visual attention modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 1, 185--207. Google Scholar
Digital Library
- Ali Borji, Dicky N. Sihite, and Laurent Itti. 2013. Quantitative analysis of human-model agreement in visual saliency modeling: A comparative study. IEEE Transactions on Image Processing 22, 1, 55--69. Google Scholar
Digital Library
- Zoya Bylinskii, Tilke Judd, Ali Borji, Laurent Itti, Frédo Durand, Aude Oliva, and Antonio Torralba. 2012. MIT Saliency Benchmark. Retrieved September 25, 2016 from http://saliency.mit.edu/.Google Scholar
- Moran Cerf, E. Paxon Frady, and Christof Koch. 2009. Faces and text attract gaze independent of the task: Experimental data and computer model. Journal of Vision 9, 12, 10.Google Scholar
Cross Ref
- Moran Cerf, Jonathan Harel, Wolfgang Einhäuser, and Christof Koch. 2008. Predicting human gaze using low-level saliency combined with face detection. In Advances in Neural Information Processing Systems. 241--248. Google Scholar
Digital Library
- Christel Chamaret, Jean-Claude Chevet, and Olivier Le Meur. 2010. Spatio-temporal combination of saliency maps and eye-tracking assessment of different strategies. In Proceedings of IEEE International Conference on Image Processing. 1077--1080. Google Scholar
Cross Ref
- Yanxiang Chen, Tam V. Nguyen, Mohan Kankanhalli, Jun Yuan, Shuicheng Yan, and Meng Wang. 2014. Audio matters in visual attention. IEEE Transactions on Circuits and Systems for Video Technology 24, 11, 1992--2003. Google Scholar
Cross Ref
- Ming-Ming Cheng, Guo-Xin Zhang, Niloy J. Mitra, Xiaolei Huang, and Shi-Min Hu. 2011. Global contrast based salient region detection. In Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition. 409--416. Google Scholar
Digital Library
- Ming-Ming Cheng, Ziming Zhang, Wen-Yan Lin, and Philip Torr. 2014. BING: Binarized normed gradients for objectness estimation at 300fps. In Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition. 3286--3293. Google Scholar
Digital Library
- Antoine Coutrot and Nathalie Guyader. 2013. Toward the introduction of auditory information in dynamic visual attention models. In Proceedings of IEEE International Workshop on Image Analysis for Multimedia Interactive Services. 1--4. Google Scholar
Cross Ref
- Antoine Coutrot and Nathalie Guyader. 2014. How saliency, faces, and sound influence gaze in dynamic social scenes. Journal of Vision 14, 8, 5.Google Scholar
Cross Ref
- Erkut Erdem and Aykut Erdem. 2013. Visual saliency estimation by nonlinearly integrating features using region covariances. Journal of Vision 13, 4, 11.Google Scholar
Cross Ref
- Georgios Evangelopoulos, Athanasia Zlatintsi, Alexandros Potamianos, Petros Maragos, Konstantinos Rapantzikos, Georgios Skoumas, and Yannis Avrithis. 2013. Multimodal saliency and fusion for movie summarization based on aural, visual, and textual attention. IEEE Transactions on Multimedia 15, 7, 1553--1568. Google Scholar
Digital Library
- Ke Gu, Guangtao Zhai, Weisi Lin, Xiaokang Yang, and Wenjun Zhang. 2015a. Visual saliency detection with free energy theory. IEEE Signal Processing Letters 22, 10, 1552--1555. Google Scholar
Cross Ref
- Ke Gu, Guofu Zhai, Xu Yang, Wensheng Zhang, and Chang Wen Chen. 2015b. Automatic contrast enhancement technology with saliency preservation. IEEE Transactions on Circuits and Systems for Video Technology 25, 9, 1480--1494. Google Scholar
Cross Ref
- Chenlei Guo, Qi Ma, and Liming Zhang. 2008. Spatio-temporal saliency detection using phase spectrum of quaternion Fourier transform. In Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition. 1--8.Google Scholar
- David Hardoon, Sandor Szedmak, and John Shawe-Taylor. 2004. Canonical correlation analysis: An overview with application to learning methods. Neural Computation 16, 12, 2639--2664. Google Scholar
Digital Library
- Jonathan Harel, Christof Koch, and Pietro Perona. 2006. Graph-based visual saliency. In Advances in Neural Information Processing Systems. 545--552. Google Scholar
Digital Library
- Xiaodi Hou and Liqing Zhang. 2007. Saliency detection: A spectral residual approach. In Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition. 1--8. Google Scholar
Cross Ref
- Xiaodi Hou and Liqing Zhang. 2009. Dynamic visual attention: Searching for coding length increments. In Advances in Neural Information Processing Systems. 681--688. Google Scholar
Digital Library
- Laurent Itti. 2004. Automatic foveation for video compression using a neurobiological model of visual attention. IEEE Transactions on Image Processing 13, 10, 1304--1318. Google Scholar
Digital Library
- Laurent Itti, Christof Koch, and Ernst Niebur. 1998. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 20, 11, 1254--1259. Google Scholar
Digital Library
- Hamid Izadinia, Imran Saleemi, and Mubarak Shah. 2013. Multimodal analysis for identification and segmentation of moving-sounding objects. IEEE Transactions on Multimedia 15, 2, 378--390. Google Scholar
Digital Library
- Lloyd A. Jeffress. 1948. A place theory of sound localization. Journal of Comparative and Physiological Psychology 41, 1, 35.Google Scholar
Cross Ref
- Tilke Judd, Krista Ehinger, Fredo Durand, and Antonio Torralba. 2009. Learning to predict where humans look. In Proceedings of IEEE International Conference on Computer Vision. 2106--2113. Google Scholar
Cross Ref
- Einat Kidron, Yoav Y. Schechner, and Michael Elad. 2005. Pixels that sound. In Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition, Vol. 1. 88--95. Google Scholar
Digital Library
- Einat Kidron, Yoav Y. Schechner, and Michael Elad. 2007. Cross-modal localization via sparsity. IEEE Transactions on Signal Processing 55, 4, 1390--1404. Google Scholar
Digital Library
- Hansang Kim, Youngbae Kim, Jae-Young Sim, and Chang-Su Kim. 2015. Spatiotemporal saliency detection for video sequences based on random walk with restart. IEEE Transactions on Image Processing. 24, 8 (Aug. 2015), 2552--2564. Google Scholar
Cross Ref
- Jong-Seok Lee, Francesca De Simone, and Touradj Ebrahimi. 2011. Subjective quality evaluation of foveated video coding using audio-visual focus of attention. IEEE Journal on Selected Topics in Signal Processing 5, 7, 1322--1331. Google Scholar
Cross Ref
- Jian Li, Martin D. Levine, Xiangjing An, Xin Xu, and Hangen He. 2013. Visual saliency based on scale-space analysis in the frequency domain. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 4, 996--1010. Google Scholar
Digital Library
- Kai Li, Jun Ye, and Kien A. Hua. 2014. What’s making that sound? In Proceedings of ACM International Conference on Multimedia. 147--156. Google Scholar
Digital Library
- Ce Liu. 2009. Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. Ph.D. Dissertation. Citeseer. Google Scholar
Digital Library
- Yu-Fei Ma, Xian-Sheng Hua, Lie Lu, and Hong-Jiang Zhang. 2005. A generic framework of user attention model and its application in video summarization. IEEE Transactions on Multimedia 7, 5, 907--919. Google Scholar
Digital Library
- Xiongkuo Min, Guangtao Zhai, Zhongpai Gao, Chunjia Hu, and Xiaokang Yang. 2014. Sound influences visual attention discriminately in videos. In Proceedings of IEEE International Workshop on Quality of Multimedia Experience. 153--158.Google Scholar
- Vicente P. Minotto, Claudio R. Jung, and Bowon Lee. 2014. Simultaneous-speaker voice activity detection and localization using mid-fusion of SVM and HMMs. IEEE Transactions on Multimedia 16, 4, 1032--1044. Google Scholar
Digital Library
- Meinard Müller. 2007. Information Retrieval for Music and Motion. Vol. 2. Springer. Google Scholar
Digital Library
- Alexandre Ninassi, Olivier Le Meur, Patrick Le Callet, and D. Barbba. 2007. Does where you gaze on an image affect your perception of quality? Applying visual attention to image quality metric. In Proceedings of IEEE International Conference on Image Processing, Vol. 2. II--169--II--172. Google Scholar
Cross Ref
- David R. Perrott, Kourosh Saberi, Kathleen Brown, and Thomas Z. Strybel. 1990. Auditory psychomotor coordination and visual search performance. Perception 8 Psychophysics 48, 3, 214--226.Google Scholar
- Robert J. Peters, Asha Iyer, Laurent Itti, and Christof Koch. 2005. Components of bottom-up gaze allocation in natural images. Vision Research 45, 18, 2397--2416. Google Scholar
Cross Ref
- Hae Jong Seo and Peyman Milanfar. 2009. Static and space-time visual saliency detection by self-resemblance. Journal of Vision 9, 12, 15.Google Scholar
Cross Ref
- G. W. Snecdecor and W. G. Cochran. 1989. Statistical Methods (8th ed.). Iowa State University Press, Iowa City, IA.Google Scholar
- Guanghan Song, Denis Pellerin, and Lionel Granjon. 2013. Different types of sounds influence gaze differently in videos. Journal of Eye Movement Research 6, 4, 1--13.Google Scholar
Cross Ref
- Jean Vroomen and Beatrice de Gelder. 2000. Sound enhances visual perception: Cross-modal effects of auditory organization on vision. Journal of Experimental Psychology: Human Perception and Performance 26, 5, 1583.Google Scholar
Cross Ref
- Chenliang Xu, Caiming Xiong, and Jason J. Corso. 2012. Streaming hierarchical video segmentation. In European Conference on Computer Vision, Springer, 626--639. Google Scholar
Digital Library
- Steven Yantis and John Jonides. 1990. Abrupt visual onsets and selective attention: Voluntary versus automatic allocation. Journal of Experimental Psychology: Human Perception and Performance 16, 1, 121.Google Scholar
Cross Ref
- Yun Zhai and Mubarak Shah. 2006. Visual attention detection in video sequences using spatiotemporal cues. In Proceedings of ACM International Conference on Multimedia. 815--824. Google Scholar
Digital Library
- Jianming Zhang and Stan Sclaroff. 2013. Saliency detection: A Boolean map approach. In Proceedings of IEEE International Conference on Computer Vision. 153--160. Google Scholar
Digital Library
- Lingyun Zhang, Matthew H. Tong, Tim K. Marks, Honghao Shan, and Garrison W. Cottrell. 2008. SUN: A Bayesian framework for saliency using natural statistics. Journal of Vision 8, 7, 32.Google Scholar
Cross Ref
Index Terms
Fixation Prediction through Multimodal Analysis
Recommendations
Eye Fixation Assisted Detection of Video Salient Objects
Advances in Brain Inspired Cognitive SystemsAbstractWith the increasing maturity of image saliency detection, more and more people are focusing their research on video saliency detection. Currently, video saliency detection can be divided into two forms, eye fixation detection and salient objects ...
Visual attention modeling based on short-term environmental adaption
Visual attention modeling is crucial for interpreting the structure and functionality of human vision system. A typical computational model of visual attention includes two basic elements: visual representation and saliency measurement. Most existing ...
A psychophysical study of fixation behavior in a computer game
APGV '08: Proceedings of the 5th symposium on Applied perception in graphics and visualizationPrediction of gaze behavior in gaming environments can be a tremendously useful asset to game designers, enabling them to improve gameplay, selectively increase visual fidelity, and optimize the distribution of computing resources. The use of saliency ...






Comments