skip to main content
research-article

Feedforward and Feedback Modulations Based Foveated JND Estimation for Images

Published:16 March 2023Publication History
Skip Abstract Section

Abstract

The just noticeable difference (JND) reveals the key characteristic of visual perception, which has been widely used in many perception-based image and video applications. Nevertheless, the modulatory mechanism of the human visual system (HVS) has not been fully exploited in JND threshold estimation, which results in the existing JND models not being accurate enough. In this article, by analyzing the feedforward and feedback modulatory behaviors in the HVS, an enhanced foveated JND (FJND) estimation model is proposed considering modulatory effects and masking effects in visual perception. The contributions of this article are mainly twofold. On the one hand, by analyzing the modulatory behaviors in the HVS, the modulatory mechanism is incorporated into JND estimation and a hierarchical modulation-based JND estimation framework is proposed for the first time. On the other hand, according to the response characteristics of visual neurons, modulatory effects on visual sensitivity are formulated as several modulatory factors to modulate the estimated JND threshold properly. Compared with existing models, the proposed model is developed in view of not only the masking effects but also the modulatory effects, which makes our model more consistent with the HVS. For different complex input images, experimental results show that the proposed FJND model tolerates more distortion at the same perceptual quality in comparison with other existing models.

REFERENCES

  1. [1] USC-SIPI. 2011. The USC-SIPI Image Database. Retrieved January 5, 2023 from https://sipi.usc.edu/database/database.php/.Google ScholarGoogle Scholar
  2. [2] IRCCyN/IVC. 2013. Subjective Quality Assessment—IVC Database. Retrieved January 5, 2023 from IRCCyN/IVC Image Quality Database - QUALINET Databases.Google ScholarGoogle Scholar
  3. [3] Ahumada A. J. and Peterson H. A.. 1992. Luminance-model-based DCT quantization for color image compression. Proceedings of SPIE: Human Vision, Visual Processing, and Digital Display III 1666 (1992), 365374.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Bae S. and Munchurl K.. 2017. A DCT-based total JND profile for spatiotemporal and foveated masking effects. IEEE Transactions on Circuits and Systems for Video Technology 27, 6 (2017), 11961207.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. [5] Barlow H. B.. 1961. Possible principle underlying the transformation of sensory messages. In Sensory Communication, W. A. Rosenblith (Ed.). MIT Press, Cambridge, MA, 217234.Google ScholarGoogle Scholar
  6. [6] Chen Z. and Guillemot C.. 2010. Perceptually-friendly H.264/AVC video coding based on foveated just-noticeable-distortion model. IEEE Transactions on Circuits and Systems for Video Technology 20, 6 (2010), 806819.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. [7] Daniel Huber, Leopoldo Petreanu, Nima Ghitani, Sachin Ranade, Tomas Hromadka, Zach Mainen, and Karel Svoboda. 2008. Sparse optical microstimulation in barrel cortex drives learned behaviour in freely moving mice. Nature 451, 7174 (2008), 61.Google ScholarGoogle ScholarCross RefCross Ref
  8. [8] Feldman H. and Friston K. L.. 2010. Attention, uncertainty, and free-energy. Frontiers in Human Neuroscience 4, 1 (2010), 125.Google ScholarGoogle ScholarCross RefCross Ref
  9. [9] Feng H., Marcellin M. W., and Bilgin. A.2015. A methodology for visually lossless JPEG2000 compression of monochrome stereo images. IEEE Transactions on Image Processing 24, 2 (2015), 560–572.Google ScholarGoogle Scholar
  10. [10] Karl Friston. 2010. The free-energy principle: A unified brain theory? Nature Reviews Neuroscience 11, 2 (2010), 127.Google ScholarGoogle ScholarCross RefCross Ref
  11. [11] Geisler Wilson S. and Perry Jeffrey S.. 1998. Real-time foveated multiresolution system for low-bandwidth video communication. Proceedings of SPIE: Human Vision and Electronic Imaging III 3299 (1998), 294305.Google ScholarGoogle Scholar
  12. [12] Gu K., Zhai G., Yang X., and Zhang W.. 2014. Using free energy principle for blind image quality assessment. IEEE Transactions on Multimedia 17, 1 (2014), 5063.Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Hochstein S. and Ahissar M.. 2002. View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron 36, 5 (2002), 791804.Google ScholarGoogle ScholarCross RefCross Ref
  14. [14] Itti Laurent and Baldi Pierre. 2009. Bayesian surprise attracts human attention. Vision Research 49, 10 (2009), 12951306.Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] ITU. 2002. Methodology for the Subjective Assessment of the Quality of Television Pictures. Technical Report ITU-R BT.500-11. ITU, Geneva, Switzerland.Google ScholarGoogle Scholar
  16. [16] Judd T., Ehinger K., Durand F., and Torralba A.. 2010. Learning to predict where humans look. In Proceedings of the International Conference on Computer Vision.Google ScholarGoogle Scholar
  17. [17] Friston Karl. 2009. The free-energy principle: A rough guide to the brain? Trends in Cognitive Sciences 13, 7 (2009), 293301.Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Ki Sehwan, Bae Sung Ho, Kim Munchurl, and Ko Hyunsuk. 2018. Learning-based just-noticeable-quantization–distortion modeling for perceptual video coding. IEEE Transactions on Image Processing 27, 7 (2018), 31783193.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Knill David C. and Pouget Alexandre. 2004. The Bayesian brain: The role of uncertainty in neural coding and computation. Trends in Neurosciences 27, 12 (2004), 712719.Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Li Junlin, Yu Li, and Wang Hongkui. 2022. Perceptual redundancy model for compression of screen content videos. IET Image Processing 16, 6 (2022), 17241741.Google ScholarGoogle ScholarCross RefCross Ref
  21. [21] Liu Anmin, Lin Weisi, Paul Manoranjan, Deng Chenwei, and Zhang Fan. 2010. Just noticeable difference for images with decomposition model separating edge and textured regions. IEEE Transactions on Circuits and Systems for Video Technology 20, 11 (2010), 16481652.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. [22] Liu Huanhua, Zhang Yun, Zhang Huan, Fan Chunling, and Fan Xiaoping. 2019. Deep learning-based picture-wise just noticeable distortion prediction model for image compression. IEEE Transactions on Image Processing 29 (2019), 641656.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. [23] Macknik S. L. and Livingstone M. S.. 1998. Neuronal correlates of visibility and invisibility in the primate visual system. Nature Neuroscience 1, 2 (1998), 144149.Google ScholarGoogle ScholarCross RefCross Ref
  24. [24] Mannion D. J., Kersten D. J., Olman C. A., and Foxe J.. 2015. Scene coherence can affect the local response to natural images in human V1. European Journal of Neuroscience 42, 11 (2015), 28952903.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Oosuga T., Tanaka M., Inoue H., and Niiyama Y.. 2012. A study on eye fixation time distribution with and without subjective evaluation of food and related pictures. In Proceedings of the 2012 SICE Annual Conference (SICE’12).Google ScholarGoogle Scholar
  26. [26] Ponomarenko N., Ieremeiev O., Lukin V., Egiazarian K., Jin L., Astola J., Vozel B., Chehdi K., Carli M., and Battisti F.. 2013. Color image database TID2013: Peculiarities and preliminary results. In Proceedings of the European Workshop on Visual Information Processing.Google ScholarGoogle Scholar
  27. [27] Rao R. P. and Ballard D. H.. 1999. Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience 2, 11 (1999), 7987.Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] Rouis Kais, Larabi Mohamed-Chaker, and Tahar Jamel Belhadj. 2018. Perceptually adaptive Lagrangian multiplier for HEVC guided rate-distortion optimization. IEEE Access 6 (2018), 3358933603.Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Russell Bryan C., Torralba Antonio, Kevin P. Murphy, and William T. Freeman. 2008. LabelMe: A database and web-based tool for image annotation. International Journal of Computer Vision 77, 1-3 (2008), 157173.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. [30] Sano Masaki. 2015. Experimental demonstration of information-to-energy conversion in small fluctuating systems. In Proceedings of the APS Meeting.Google ScholarGoogle Scholar
  31. [31] Tian Tao, Wang Hanli, Zuo Lingxuan, Kuo C.-C. Jay, and Kwong Sam. 2020. Just noticeable difference level prediction for perceptual image compression. IEEE Transactions on Broadcasting 66, 3 (2020), 690700.Google ScholarGoogle ScholarCross RefCross Ref
  32. [32] Wallisch P. and Movshon J. A.. 2008. Structure and function come unglued in the visual cortex. Neuron 8, 2 (2008), 195197.Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Wang Hongkui, Wang Shengwei, Li Tiansong, Yin Haibing, and Yu Li. 2019. Surprise based JND estimation for images. IEEE Signal Processing Letters 27 (2019), 181184.Google ScholarGoogle ScholarCross RefCross Ref
  34. [34] Wang Hongkui, Yu Li, Liang Junhui, Yin Haibing, Li Tiansong, and Wang Shengwei. 2021. Hierarchical predictive coding-based JND estimation for image compression. IEEE Transactions on Image Processing 30 (2021), 487500.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. [35] Wang Hongkui, Yu Li, Wang Shengwei, Li Tiansong, and Yin Haibing. 2018. A novel foveated-JND profile based on an adaptive foveated weighting model. In Proceedings of the 2018 IEEE Conference on Visual Communications and Image Processing (VCIP’18).Google ScholarGoogle ScholarCross RefCross Ref
  36. [36] Wang Hongkui, Yu Li, Yin Haibing, Li Tiansong, and Wang Shengwei. 2020. An improved DCT-based JND estimation model considering multiple masking effects. Journal of Visual Communication and Image Representation 71 (2020), 102850. Google ScholarGoogle ScholarCross RefCross Ref
  37. [37] Wang Rubin and Zhang Zhikang. 2007. Energy coding in biological neural networks. Cognitive Neurodynamics 1, 3 (2007), 203212.Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Wang Rubin, Zhang Zhikang, and Chen Guanrong. 2009. Energy coding and energy functions for local activities of the brain. Neurocomputing 73, 1-3 (2009), 139150.Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. [39] Wang S., Ma L., Fang Y. M., Lin W., Ma S., and Gao W.. 2016. Just noticeable difference estimation for screen content images. IEEE Transactions on Image Processing 25, 8 (2016), 38383851.Google ScholarGoogle Scholar
  40. [40] Wang Zhou, Lu Ligang, and Bovik A. C.. 2003. Foveation scalable video coding with automatic fixation selection. IEEE Transactions on Image Processing 12, 2 (2003), 243254.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. [41] Wei Zhenyu and Ngan King N.. 2009. Spatio-temporal just noticeable distortion profile for grey scale image/video in DCT domain. IEEE Transactions on Circuits and Systems for Video Technology 19, 3 (2009), 337346.Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. [42] Wu J., Li L., Dong W., Shi G., Lin W., and Kuo C.-C. Jay. 2017. Enhanced just noticeable difference model for images with pattern complexity. IEEE Transactions on Image Processing 26, 6 (2017), 26822693.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. [43] Wu J., Lin W., Shi G., Wang X., and Qi F.. 2013. Just difference estimation for images with free energy principle. IEEE Transactions on Image Processing 15, 7 (2013), 17051710.Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. [44] Wu J., Shi G., Lin W., Liu A., and Li F.. 2013. Pattern masking estimation in image with structural uncertainty. IEEE Transactions on Image Processing 22, 12 (2013), 48924904.Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. [45] Yang Kaifu, Li Chaoyi, and Li Yongjie. 2014. Multi feature-based surround inhibition improves contour detection in natural images. IEEE Transactions on Image Processing 23, 12 (2014), 50205032.Google ScholarGoogle ScholarCross RefCross Ref
  46. [46] Yang X., Lin W., Lu Z., Ong E., and Yao S.. 2005. Just noticeable distortion model and its applications in video coding. Signal Processing: Image Communication 20 (2005), 662680.Google ScholarGoogle ScholarCross RefCross Ref
  47. [47] Zeng Zhipeng, Zeng Huanqiang, Chen Jing, Zhu Jianqing, Zhang Yun, and Ma Kai Kuang. 2019. Visual attention guided pixel-wise just noticeable difference model. IEEE Access 7 (2019), 132111132119.Google ScholarGoogle ScholarCross RefCross Ref
  48. [48] Zhai G., Wu X., Yang X., Lin W., and Zhang W.. 2012. A psychovisual quality metric in free-energy principle. IEEE Transactions on Image Processing 21, 1 (2012), 4152.Google ScholarGoogle Scholar
  49. [49] Zhang Xinfeng, Wang Shiqi, Gu Ke, Lin Weisi, Ma Siwei, and Gao Wen. 2017. Just-noticeable difference-based perceptual optimization for JPEG compression. IEEE Signal Processing Letters 24, 1 (2017), 96100.Google ScholarGoogle ScholarCross RefCross Ref
  50. [50] Zhang Yun, Liu Huanhua, Yang You, Fan Xiaoping, Kwong Sam, and Kuo C. C. Jay. 2022. Deep learning based just noticeable difference and perceptual quality prediction models for compressed video. IEEE Transactions on Circuits and Systems for Video Technology 32, 3 (2022), 1197–1212.Google ScholarGoogle Scholar
  51. [51] Zhou Mingliang, Wei Xuekai, Wang Shiqi, Kwong Sam, Fong Chi Keung, Wong Peter, Yuen Wilson, and Gao Wei. 2019. SSIM-based global optimization for CTU-level rate control in HEVC. IEEE Transactions on Multimedia 21, 8 (2019), 19211933.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Feedforward and Feedback Modulations Based Foveated JND Estimation for Images

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Multimedia Computing, Communications, and Applications
      ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 5
      September 2023
      262 pages
      ISSN:1551-6857
      EISSN:1551-6865
      DOI:10.1145/3585398
      • Editor:
      • Abdulmotaleb El Saddik
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 16 March 2023
      • Online AM: 4 January 2023
      • Accepted: 18 December 2022
      • Revised: 25 October 2022
      • Received: 8 June 2022
      Published in tomm Volume 19, Issue 5

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
    • Article Metrics

      • Downloads (Last 12 months)155
      • Downloads (Last 6 weeks)17

      Other Metrics

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!