Abstract
The just noticeable difference (JND) reveals the key characteristic of visual perception, which has been widely used in many perception-based image and video applications. Nevertheless, the modulatory mechanism of the human visual system (HVS) has not been fully exploited in JND threshold estimation, which results in the existing JND models not being accurate enough. In this article, by analyzing the feedforward and feedback modulatory behaviors in the HVS, an enhanced foveated JND (FJND) estimation model is proposed considering modulatory effects and masking effects in visual perception. The contributions of this article are mainly twofold. On the one hand, by analyzing the modulatory behaviors in the HVS, the modulatory mechanism is incorporated into JND estimation and a hierarchical modulation-based JND estimation framework is proposed for the first time. On the other hand, according to the response characteristics of visual neurons, modulatory effects on visual sensitivity are formulated as several modulatory factors to modulate the estimated JND threshold properly. Compared with existing models, the proposed model is developed in view of not only the masking effects but also the modulatory effects, which makes our model more consistent with the HVS. For different complex input images, experimental results show that the proposed FJND model tolerates more distortion at the same perceptual quality in comparison with other existing models.
- [1] USC-SIPI. 2011. The USC-SIPI Image Database. Retrieved January 5, 2023 from https://sipi.usc.edu/database/database.php/.Google Scholar
- [2] IRCCyN/IVC. 2013. Subjective Quality Assessment—IVC Database. Retrieved January 5, 2023 from IRCCyN/IVC Image Quality Database - QUALINET Databases.Google Scholar
- [3] . 1992. Luminance-model-based DCT quantization for color image compression. Proceedings of SPIE: Human Vision, Visual Processing, and Digital Display III 1666 (1992), 365–374.Google Scholar
Cross Ref
- [4] . 2017. A DCT-based total JND profile for spatiotemporal and foveated masking effects. IEEE Transactions on Circuits and Systems for Video Technology 27, 6 (2017), 1196–1207.Google Scholar
Digital Library
- [5] . 1961. Possible principle underlying the transformation of sensory messages. In Sensory Communication, W. A. Rosenblith (Ed.). MIT Press, Cambridge, MA, 217–234.Google Scholar
- [6] . 2010. Perceptually-friendly H.264/AVC video coding based on foveated just-noticeable-distortion model. IEEE Transactions on Circuits and Systems for Video Technology 20, 6 (2010), 806–819.Google Scholar
Digital Library
- [7] Daniel Huber, Leopoldo Petreanu, Nima Ghitani, Sachin Ranade, Tomas Hromadka, Zach Mainen, and Karel Svoboda. 2008. Sparse optical microstimulation in barrel cortex drives learned behaviour in freely moving mice. Nature 451, 7174 (2008), 61.Google Scholar
Cross Ref
- [8] . 2010. Attention, uncertainty, and free-energy. Frontiers in Human Neuroscience 4, 1 (2010), 125.Google Scholar
Cross Ref
- [9] 2015. A methodology for visually lossless JPEG2000 compression of monochrome stereo images. IEEE Transactions on Image Processing 24, 2 (2015), 560–572.Google Scholar
- [10] Karl Friston. 2010. The free-energy principle: A unified brain theory? Nature Reviews Neuroscience 11, 2 (2010), 127.Google Scholar
Cross Ref
- [11] . 1998. Real-time foveated multiresolution system for low-bandwidth video communication. Proceedings of SPIE: Human Vision and Electronic Imaging III 3299 (1998), 294–305.Google Scholar
- [12] . 2014. Using free energy principle for blind image quality assessment. IEEE Transactions on Multimedia 17, 1 (2014), 50–63.Google Scholar
Cross Ref
- [13] . 2002. View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron 36, 5 (2002), 791–804.Google Scholar
Cross Ref
- [14] . 2009. Bayesian surprise attracts human attention. Vision Research 49, 10 (2009), 1295–1306.Google Scholar
Cross Ref
- [15] . 2002. Methodology for the Subjective Assessment of the Quality of Television Pictures. Technical Report ITU-R BT.500-11. ITU, Geneva, Switzerland.Google Scholar
- [16] . 2010. Learning to predict where humans look. In Proceedings of the International Conference on Computer Vision.Google Scholar
- [17] . 2009. The free-energy principle: A rough guide to the brain? Trends in Cognitive Sciences 13, 7 (2009), 293–301.Google Scholar
Cross Ref
- [18] . 2018. Learning-based just-noticeable-quantization–distortion modeling for perceptual video coding. IEEE Transactions on Image Processing 27, 7 (2018), 3178–3193.Google Scholar
Cross Ref
- [19] . 2004. The Bayesian brain: The role of uncertainty in neural coding and computation. Trends in Neurosciences 27, 12 (2004), 712–719.Google Scholar
Cross Ref
- [20] . 2022. Perceptual redundancy model for compression of screen content videos. IET Image Processing 16, 6 (2022), 1724–1741.Google Scholar
Cross Ref
- [21] . 2010. Just noticeable difference for images with decomposition model separating edge and textured regions. IEEE Transactions on Circuits and Systems for Video Technology 20, 11 (2010), 1648–1652.Google Scholar
Digital Library
- [22] . 2019. Deep learning-based picture-wise just noticeable distortion prediction model for image compression. IEEE Transactions on Image Processing 29 (2019), 641–656.Google Scholar
Digital Library
- [23] . 1998. Neuronal correlates of visibility and invisibility in the primate visual system. Nature Neuroscience 1, 2 (1998), 144–149.Google Scholar
Cross Ref
- [24] . 2015. Scene coherence can affect the local response to natural images in human V1. European Journal of Neuroscience 42, 11 (2015), 2895–2903.Google Scholar
Cross Ref
- [25] . 2012. A study on eye fixation time distribution with and without subjective evaluation of food and related pictures. In Proceedings of the 2012 SICE Annual Conference (SICE’12).Google Scholar
- [26] . 2013. Color image database TID2013: Peculiarities and preliminary results. In Proceedings of the European Workshop on Visual Information Processing.Google Scholar
- [27] . 1999. Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience 2, 11 (1999), 79–87.Google Scholar
Cross Ref
- [28] . 2018. Perceptually adaptive Lagrangian multiplier for HEVC guided rate-distortion optimization. IEEE Access 6 (2018), 33589–33603.Google Scholar
Cross Ref
- [29] . 2008. LabelMe: A database and web-based tool for image annotation. International Journal of Computer Vision 77, 1-3 (2008), 157–173.Google Scholar
Digital Library
- [30] . 2015. Experimental demonstration of information-to-energy conversion in small fluctuating systems. In Proceedings of the APS Meeting.Google Scholar
- [31] . 2020. Just noticeable difference level prediction for perceptual image compression. IEEE Transactions on Broadcasting 66, 3 (2020), 690–700.Google Scholar
Cross Ref
- [32] . 2008. Structure and function come unglued in the visual cortex. Neuron 8, 2 (2008), 195–197.Google Scholar
Cross Ref
- [33] . 2019. Surprise based JND estimation for images. IEEE Signal Processing Letters 27 (2019), 181–184.Google Scholar
Cross Ref
- [34] . 2021. Hierarchical predictive coding-based JND estimation for image compression. IEEE Transactions on Image Processing 30 (2021), 487–500.Google Scholar
Digital Library
- [35] . 2018. A novel foveated-JND profile based on an adaptive foveated weighting model. In Proceedings of the 2018 IEEE Conference on Visual Communications and Image Processing (VCIP’18).Google Scholar
Cross Ref
- [36] . 2020. An improved DCT-based JND estimation model considering multiple masking effects. Journal of Visual Communication and Image Representation 71 (2020), 102850. Google Scholar
Cross Ref
- [37] . 2007. Energy coding in biological neural networks. Cognitive Neurodynamics 1, 3 (2007), 203–212.Google Scholar
Cross Ref
- [38] . 2009. Energy coding and energy functions for local activities of the brain. Neurocomputing 73, 1-3 (2009), 139–150.Google Scholar
Digital Library
- [39] . 2016. Just noticeable difference estimation for screen content images. IEEE Transactions on Image Processing 25, 8 (2016), 3838–3851.Google Scholar
- [40] . 2003. Foveation scalable video coding with automatic fixation selection. IEEE Transactions on Image Processing 12, 2 (2003), 243–254.Google Scholar
Digital Library
- [41] . 2009. Spatio-temporal just noticeable distortion profile for grey scale image/video in DCT domain. IEEE Transactions on Circuits and Systems for Video Technology 19, 3 (2009), 337–346.Google Scholar
Digital Library
- [42] . 2017. Enhanced just noticeable difference model for images with pattern complexity. IEEE Transactions on Image Processing 26, 6 (2017), 2682–2693.Google Scholar
Digital Library
- [43] . 2013. Just difference estimation for images with free energy principle. IEEE Transactions on Image Processing 15, 7 (2013), 1705–1710.Google Scholar
Digital Library
- [44] . 2013. Pattern masking estimation in image with structural uncertainty. IEEE Transactions on Image Processing 22, 12 (2013), 4892–4904.Google Scholar
Digital Library
- [45] . 2014. Multi feature-based surround inhibition improves contour detection in natural images. IEEE Transactions on Image Processing 23, 12 (2014), 5020–5032.Google Scholar
Cross Ref
- [46] . 2005. Just noticeable distortion model and its applications in video coding. Signal Processing: Image Communication 20 (2005), 662–680.Google Scholar
Cross Ref
- [47] . 2019. Visual attention guided pixel-wise just noticeable difference model. IEEE Access 7 (2019), 132111–132119.Google Scholar
Cross Ref
- [48] . 2012. A psychovisual quality metric in free-energy principle. IEEE Transactions on Image Processing 21, 1 (2012), 41–52.Google Scholar
- [49] . 2017. Just-noticeable difference-based perceptual optimization for JPEG compression. IEEE Signal Processing Letters 24, 1 (2017), 96–100.Google Scholar
Cross Ref
- [50] . 2022. Deep learning based just noticeable difference and perceptual quality prediction models for compressed video. IEEE Transactions on Circuits and Systems for Video Technology 32, 3 (2022), 1197–1212.Google Scholar
- [51] . 2019. SSIM-based global optimization for CTU-level rate control in HEVC. IEEE Transactions on Multimedia 21, 8 (2019), 1921–1933.Google Scholar
Digital Library
Index Terms
Feedforward and Feedback Modulations Based Foveated JND Estimation for Images
Recommendations
Perception-oriented video coding based on foveated JND model
PCS'09: Proceedings of the 27th conference on Picture Coding SymposiumThe ultimate goal of video compression is to provide best visual quality of the video source by using minimum bit budget. To achieve this target, spatial, temporal and perceptual redundancies should be exploited and removed such that the video source ...
A novel attention-guided JND Model for improving robust image watermarking
AbstractJust noticeable distortion (JND) and visual attention (VA), which are two widely used mathematical models of human visual system (HVS) that aim to simulate the human brain mechanism, are sufficiently explored and applied to many researches ...
Towards Attention–aware Foveated Rendering
Foveated graphics is a promising approach to solving the bandwidth challenges of immersive virtual and augmented reality displays by exploiting the falloff in spatial acuity in the periphery of the visual field. However, the perceptual models used in ...






Comments