Beyond blur: real-time ventral metamers for foveated rendering

Abstract

To peripheral vision, a pair of physically different images can look the same. Such pairs are metamers relative to each other, just as physically-different spectra of light are perceived as the same color. We propose a real-time method to compute such ventral metamers for foveated rendering where, in particular for near-eye displays, the largest part of the framebuffer maps to the periphery. This improves in quality over state-of-the-art foveation methods which blur the periphery. Work in Vision Science has established how peripheral stimuli are ventral metamers if their statistics are similar. Existing methods, however, require a costly optimization process to find such metamers. To this end, we propose a novel type of statistics particularly well-suited for practical real-time rendering: smooth moments of steerable filter responses. These can be extracted from images in time constant in the number of pixels and in parallel over all pixels using a GPU. Further, we show that they can be compressed effectively and transmitted at low bandwidth. Finally, computing realizations of those statistics can again be performed in constant time and in parallel. This enables a new level of quality for foveated applications such as such as remote rendering, level-of-detail and Monte-Carlo denoising. In a user study, we finally show how human task performance increases and foveation artifacts are less suspicious, when using our method compared to common blurring.

References

1. Kaan Akşit, Praneeth Chakravarthula, Kishore Rathinavel, Youngmo Jeong, Rachel Albert, Henry Fuchs, and David Luebke. 2019. Manufacturing application-driven foveated near-eye displays. IEEE Trans Vis and Comp Graph 25, 5 (2019), 1928--1939.
2. Rachel Albert, Anjul Patney, David Luebke, and Joohwan Kim. 2017. Latency requirements for foveated rendering in virtual reality. ACM Trans App Perc 14, 4 (2017).
3. Stuart M Anstis. 1974. A chart demonstrating variations in acuity with retinal position. Vis Res 14, 7 (1974), 589--592.
4. H Aubert and R Förster. 1857. Beiträge zur Kenntniss des indirecten Sehens.(I). Untersuchungen über den Raumsinn der Retina. Archiv für Ophthalmologie 3, 2 (1857), 1--37.
5. Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Trans Graph 36, 4 (2017), 97--1.
6. Herman Bouma. 1970. Interaction effects in parafoveal letter recognition. Nature 226, 5241 (1970), 177--178.
7. Matteo Carandini, Jonathan B. Demb, Valerio Mante, David J. Tolhurst, Yang Dan, Bruno A. Olshausen, Jack L. Gallant, and Nicole C. Rust. 2005. Do we know what the early visual system does? J Neuroscience 25, 46 (2005), 10577--10597.
8. Chakravarty R Alla Chaitanya, Anton S Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Trans Graph 36, 4 (2017), 1--12.
9. FJJ Clarke. 1960. A study of Troxler's effect. Optica Acta: Int J Optics 7, 3 (1960), 219--236.
10. Arturo Deza, Aditya Jonnalagadda, and Miguel Eckstein. 2017. Towards metamerism via foveated style transfer. arXiv:1705.10041 (2017).
11. Annette J. Dobson and Adrian G. Barnett. 2018. An Introduction to Generalized Linear Models. Chapman and Hall/CRC.
12. William Donnelly and Andrew Lauritzen. 2006. Variance shadow maps. In Proc. i3D. 161--165.
13. Andrew T Duchowski, Nathan Cournia, and Hunter Murphy. 2004. Gaze-Contingent Displays: A Review. CyberPsychology & Behavior 7, 6 (2004), 621--634.
14. Alexei A Efros and Thomas K Leung. 1999. Texture synthesis by non-parametric sampling. In ICCV, Vol. 2. 1033--1038.
15. Mark D Fairchild. 2013. Color appearance models. John Wiley & Sons.
16. Jenelle Feather, Alex Durango, Ray Gonzalez, and Josh McDermott. 2019. Metamers of neural networks reveal divergence from human perceptual systems. NeurIPS 32 (2019), 1--25.
17. Jeremy Freeman and Eero P Simoncelli. 2011. Metamers of the ventral stream. Nature Neuroscience 14, 9 (2011), 1195--1201.
18. William T Freeman, Edward H Adelson, et al. 1991. The design and use of steerable filters. IEEE PAMI 13, 9 (1991), 891--906.
19. Sebastian Friston, Tobias Ritschel, and Anthony Steed. 2019. Perceptual rasterization for head-mounted display image synthesis. ACM Trans Graph 38, 4 (2019), 1--14.
20. Masahiro Fujita and Takahiro Harada. 2014. Foveated real-time ray tracing for virtual reality headset. SIGGRAPH Asia Posters 14 (2014).
21. Bruno Galerne, Ares Lagae, Sylvain Lefebvre, and George Drettakis. 2012. Gabor noise by example. ACM Trans Graph 31, 4 (2012), 1--9.
22. Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2016. Image style transfer using convolutional neural networks. In CVPR. 2414--2423.
23. Wilson S Geisler and Jeffrey S Perry. 1998. Real-time foveated multiresolution system for low-bandwidth video communication. In HVIE III, Vol. 3299. 294--305.
24. John A. Greenwood, Peter J. Bex, and Steven C. Dakin. 2009. Positional averaging explains crowding with letter-like stimuli. Proc NAS US 106, 31 (2009), 13130--13135.
25. Umut Güçlü and Marcel A.J. van Gerven. 2015. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J Neuroscience 35, 27 (2015), 10005--10014.
26. Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. 2012. Foveated 3D graphics. ACM Trans Graph 31, 6 (2012), 1--10.
27. Yong He, Yan Gu, and Kayvon Fatahalian. 2014. Extending the graphics pipeline with adaptive, multi-rate shading. ACM Trans Graph 33, 4 (2014).
28. David J Heeger and James R Bergen. 1995. Pyramid-based texture analysis/synthesis. In Proc. SIGGRAPH. 229--238.
29. David Hoffman, Zoe Meraz, and Eric Turner. 2018. Limits of peripheral acuity and implications for VR system design. J SID 26, 8 (2018), 483--495.
30. Xun Huang and Serge Belongie. 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV. 1501--1510.
31. David H. Hubel. 1982. Exploration of the primary visual cortex, 1955--78. Nature 299, 5883 (oct 1982), 515--524.
32. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-image translation with conditional adversarial networks. In CVPR. 1125--1134.
33. Nima Khademi Kalantari, Steve Bako, and Pradeep Sen. 2015. A machine learning approach for filtering Monte Carlo noise. ACM Trans Graph 34, 4 (2015), 122--1.
34. Anton S Kaplanyan, Anton Sochenov, Thomas Leimkühler, Mikhail Okunev, Todd Goodall, and Gizem Rufo. 2019. DeepFovea: Neural reconstruction for foveated rendering and video compression using learned statistics of natural videos. ACM Trans Graph 38, 6 (2019), 1--13.
35. Michael Kass and Davide Pesare. 2011. Coherent noise for non-photorealistic rendering. ACM Trans. Graph. (TOG) 30, 4 (2011), 1--6.
36. Jonghyun Kim, Youngmo Jeong, Michael Stengel, Kaan Akşit, Rachel Albert, Ben Boudaoud, Trey Greer, Joohwan Kim, Ward Lopes, Zander Majercik, et al. 2019. Foveated AR: dynamically-foveated augmented reality display. ACM Trans Graph 38, 4 (2019), 1--15.
37. Min H Kim, Tobias Ritschel, and Jan Kautz. 2011. Edge-aware color appearance. ACM Trans Graph 30, 2 (2011), 1--9.
38. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980 (2014).
39. Ares Lagae, Peter Vangorp, Toon Lenaerts, and Philip Dutré. 2010. Procedural isotropic stochastic textures by example. Computers & Graphics 34, 4 (2010), 312--321.
40. Gordon E Legge and Daniel Kersten. 1987. Contrast discrimination in peripheral vision. J OSA A 4, 8 (1987), 1594--1598.
41. Marc Levoy and Ross Whitaker. 1990. Gaze-directed volume rendering. In Proc. i3D. 217--223.
42. Lin Liang, Ce Liu, Ying-Qing Xu, Baining Guo, and Heung-Yeung Shum. 2001. Real-time texture synthesis by patch-based sampling. ACM Trans Graph 20, 3 (2001), 127--150.
43. Rafał Mantiuk, Kil Joong Kim, Allan G. Rempel, and Wolfgang Heidrich. 2011. HDR-VDP-2. ACM Trans Graph 30, 4(2011), 1--14.
44. Xiaoxu Meng, Ruofei Du, Matthias Zwicker, and Amitabh Varshney. 2018. Kernel foveated rendering. Proc. i3D 1, 1 (2018), 1--20.
45. Hunter Murphy and Andrew T Duchowski. 2001. Gaze-contingent level of detail rendering. Proc. Eurographics (2001).
46. Toshikazu Ohshima, Hiroyuki Yamamoto, and Hideyuki Tamura. 1996. Gaze-directed adaptive rendering for interacting with virtual space. Proceedings - Virtual Reality Annual International Symposium (1996), 103--110.
47. Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. 2016. Towards foveated rendering for gaze-tracked virtual reality. ACM Trans Graph 35, 6 (2016), 179.
48. Ken Perlin. 1985. An image synthesizer. ACM Siggraph Computer Graphics 19, 3 (1985), 287--296.
49. Javier Portilla and Eero P Simoncelli. 2000. A parametric texture model based on joint statistics of complex wavelet coefficients. Int J Comp Vis 40, 1 (2000), 49--70.
50. Charles Poynton. 2012. Digital Video and HD: Algorithms and Interfaces. Morgan Kaufmann. 752 pages.
51. Eyal M. Reingold, Lester C. Loschky, George W. McConkie, and David M. Stampe. 2003. Gaze-contingent multiresolutional displays: An integrative review. Human Factors 45, 2 (2003), 307--328.
52. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In MICCAI. Cham, 234--241.
53. R Rosenholtz. 2016. Capabilities and Limitations of Peripheral Vision. Annual review of vision science 2 (2016), 437.
54. Ruth Rosenholtz, Jie Huang, Alvin Raj, Benjamin J. Balas, and Livia Ilie. 2012. A summary statistic representation in peripheral vision explains visual search. J Vision 12, 4 (2012), 14--14.
55. Daniel L. Ruderman, Thomas W. Cronin, and Chuan-Chin Chiao. 1998. Statistics of cone responses to natural images: implications for visual coding. J OSA A 15, 8 (1998), 2036--2045.
56. Anita M. Schmid, Keith P Purpura, Ifije E Ohiorhenuan, Ferenc Mechler, and Jonathan D Victor. 2009. Subpopulations of neurons in visual area V2 perform differentiation and integration operations in space and time. 3 (2009), 1--16.
57. Albulena Shaqiri, Maya Roinishvili, Lukasz Grzeczkowski, Eka Chkonia, Karin Pilz, Christine Mohr, Andreas Brand, Marina Kunchulia, and Michael H. Herzog. 2018. Sex-related differences in vision are heterogeneous. Scientific Rep 8, 1 (2018), 7521.
58. Josef Spjut, Ben Boudaoud, Jonghyun Kim, Trey Greer, Rachel Albert, Michael Stengel, Kaan Aksit, and David Luebke. 2019. Toward standardized classification of foveated displays. arXiv:1905.06229 (2019).
59. Michael Stengel, Steve Grogorick, Martin Eisemann, and Marcus Magnor. 2016. Adaptive image-space sampling for gaze-contingent real-time rendering. 35, 4 (2016), 129--39.
60. Hans Strasburger. 2020. Seven Myths on Crowding and Peripheral Vision. i-Perception 11, 3 (2020).
61. Hans Strasburger, Ingo Rentschler, and Martin Jüttner. 2011. Peripheral vision and pattern recognition: A review. J Vision 11, 5 (2011), 13--13.
62. Nicholas T Swafford, José A Iglesias-Guitian, Charalampos Koniaris, Bochang Moon, Darren Cosker, and Kenny Mitchell. 2016. User, metric, and computational evaluation of foveated rendering methods. In Proc. SAP. 7--14.
63. Keiji Tanaka. 1996. Inferotemporal Cortex and Object Vision. Ann Rev Neuro 19, 1 (1996), 109--39.
64. V Javier Traver and Alexandre Bernardino. 2010. A review of log-polar imaging for visual perception in robotics. Robotics and Autonomous Systems 58, 4 (2010), 378--398.
65. Okan Tarhan Tursun, Elena Arabadzhiyska-Koleva, Marek Wernikowski, Radosław Mantiuk, Hans-Peter Seidel, Karol Myszkowski, and Piotr Didyk. 2019. Luminance-contrast-aware foveated rendering. ACM Trans Graph 38, 4 (2019), 1--14.
66. Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2016. Instance normalization: The missing ingredient for fast stylization. arXiv:1607.08022 (2016).
67. Leslie G Ungerleider and James V Haxby. 1994. 'What' and 'where' in the human brain. Current Opinion in Neurobiology 4, 2 (1994), 157--65.
68. Karthik Vaidyanathan, Marco Salvi, Robert Toth, Tim Foley, Jim Akenine-Möller, Tomas Nilsson, Jacob Munkberg, Jon Hasselgren, Masamichi Sugihara, Petrik Clarberg, Tomasz Janczak, and Aaron Lefohn. 2014. Coarse Pixel Shading. In Proc. HPG.
69. Thomas SA Wallis, Matthias Bethge, and Felix A Wichmann. 2016. Testing models of peripheral encoding using metamerism in an oddity paradigm. J Vision 16, 2 (2016), 4--4.
70. Li-Yi Wei, Sylvain Lefebvre, Vivek Kwatra, and Greg Turk. 2009. State of the Art in Example-based Texture Synthesis. In Eurographics STAR. 93--117.
71. Martin Weier, Thorsten Roth, Ernst Kruijff, André Hinkenjann, Arsène Pérard-Gayot, Philipp Slusallek, and Yongmin Li. 2016. Foveated real-time ray tracing for head-mounted displays. 35, 7 (2016), 289--298.
72. G. N. Wilkinson and C. E. Rogers. 1973. Symbolic Description of Factorial Models for Analysis of Variance. J Royal Stat Soc C 22, 3 (1973), 392--399.
73. Lance Williams. 1983. Pyramidal parametrics. In Proc. SIGGRAPH. 1--11.
74. Xin Zhang, Wei Chen, Zhonglei Yang, Chuan Zhu, and Qunsheng Peng. 2011. A new foveation ray casting approach for real-time rendering of 3D scenes. Proc CAD/Graphics (2011), 99--102.

Index Terms

• Computing methodologies

  • Computer graphics

    • Graphics systems and interfaces

      • Perception

    • Image compression

    • Rendering

      • Ray tracing