Abstract
Foveated rendering coupled with eye-tracking has the potential to dramatically accelerate interactive 3D graphics with minimal loss of perceptual detail. In this paper, we parameterize foveated rendering by embedding polynomial kernel functions in the classic log-polar mapping. Our GPU-driven technique uses closed-form, parameterized foveation that mimics the distribution of photoreceptors in the human retina. We present a simple two-pass kernel foveated rendering (KFR) pipeline that maps well onto modern GPUs. In the first pass, we compute the kernel log-polar transformation and render to a reduced-resolution buffer. In the second pass, we carry out the inverse-log-polar transformation with anti-aliasing to map the reduced-resolution rendering to the full-resolution screen. We have carried out pilot and formal user studies to empirically identify the KFR parameters. We observe a 2.8X -- 3.2X speedup in rendering on 4K UHD (2160p) displays with minimal perceptual loss of detail. The relevance of eye-tracking-guided kernel foveated rendering can only increase as the anticipated rise of display resolution makes it ever more difficult to resolve the mutually conflicting goals of interactive rendering and perceptual realism.
- Marco Antonelli, Francisco D. Igual, Francisco Ramos, and V. Javier Traver. 2015. Speeding up the log-polar transform with inexpensive parallel hardware: graphics units and multi-core architectures. J. Real-Time Image Process. 10, 3 (Sept. 2015), 533--550. Google Scholar
Digital Library
- H. Araujo and J. M. Dias. 1996. An introduction to the log-polar mapping {image sampling}. In Proceedings II Workshop on Cybernetic Vision. 139--144.Google Scholar
- Nir Benty, Kai-Hwa Yao, Tim Foley, Anton S. Kaplanyan, Conor Lavelle, Chris Wyman, and Ashwin Vijay. 2017. The Falcor rendering framework. https://github.com/NVIDIAGameWorks/FalcorGoogle Scholar
- Peter J Burt. 1988. Smart sensing within a pyramid vision machine. Proc. IEEE 76, 8 (1988), 1006--1015.Google Scholar
Cross Ref
- Matthäus G. Chajdas, Morgan McGuire, and David Luebke. 2011. Subpixel reconstruction antialiasing for deferred shading. In Symposium on Interactive 3D Graphics and Games (I3D '11). ACM, New York, NY, USA, 15-22 [email protected]. Google Scholar
Digital Library
- Ee-Chien Chang, Stéphane Mallat, and Chee Yap. 2000. Wavelet foveation. Applied and Computational Harmonic Analysis 9, 3 (2000), 312--335.Google Scholar
Cross Ref
- Petrik Clarberg, Robert Toth, Jon Hasselgren, Jim Nilsson, and Tomas Akenine-Möller. 2014. AMFS: adaptive multi-frequency shading for future graphics processors. ACM Trans. Graph. 33, 4, Article 141 (July 2014), 12 pages. Google Scholar
Digital Library
- Cyril Crassin, Morgan McGuire, Kayvon Fatahalian, and Aaron Lefohn. 2015. Aggregate G-buffer anti-aliasing. In Proceedings of the 19th Symposium on Interactive 3D Graphics and Games (I3D '15). ACM, New York, NY, USA, 109--119. Google Scholar
Digital Library
- Jerome F Duluk Jr, Richard E Hessel, Vaughn T Arnold, Jack Benkual, Joseph P Bratt, George Cuan, Stephen L Dodgen, Emerson S Fang, Zhaoyu Gong, Thomas Y Ho, et al. 2004. Deferred shading graphics pipeline processor having advanced features. US Patent 6,717,576.Google Scholar
- Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. 1996. The lumigraph. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '96). ACM, New York, NY, USA, 43--54. Google Scholar
Digital Library
- Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. 2012. Foveated 3D graphics. ACM Trans. Graph. 31, 6, Article 164 (Nov. 2012), 10 pages. Google Scholar
Digital Library
- Shawn Hargreaves and Mark Harris. 2004. Deferred shading. In Game Developers Conference, Vol. 2. 31.Google Scholar
- Yong He, Yan Gu, and Kayvon Fatahalian. 2014. Extending the graphics pipeline with adaptive, multi-rate shading. ACM Trans. Graph. 33, 4, Article 142 (July 2014), 12 pages. Google Scholar
Digital Library
- Hugues Hoppe. 1998. Smooth view-dependent level-of-detail control and its application to terrain rendering. In Proceedings of the Conference on Visualization '98 (VIS '98). IEEE Computer Society Press, Los Alamitos, CA, USA, 35--42. http://dl.acm.org/citation.cfm?id=288216.288221 Google Scholar
Digital Library
- Chih-Fan Hsu, Anthony Chen, Cheng-Hsin Hsu, Chun-Ying Huang, Chin-Laung Lei, and Kuan-Ta Chen. 2017. Is foveated rendering perceivable in virtual reality?: exploring the efficiency and consistency of quality assessment methods. In Proceedings of the 2017 ACM on Multimedia Conference (MM '17). ACM, New York, NY, USA, 55--63. Google Scholar
Digital Library
- L. Hu, P. V. Sander, and H. Hoppe. 2010. Parallel view-dependent level-of-detail control. IEEE Transactions on Visualization and Computer Graphics 16, 5 (Sept 2010), 718--728. Google Scholar
Digital Library
- Cheuk Yiu Ip, M. Adil Yalçin, David Luebke, and Amitabh Varshney. 2013. PixelPie: maximal Poisson-disk sampling with rasterization. In Proceedings of the 5th High-Performance Graphics Conference (HPG '13). ACM, New York, NY, USA, 17--26. Google Scholar
Digital Library
- B Karis. 2014. High-quality temporal supersampling. Advances in Real-Time Rendering in Games, SIGGRAPH Courses 1 (2014), 1--55.Google Scholar
- Youngmin Kim, Amitabh Varshney, David Jacobs, and Francois Guimbretere. 2010. Mesh Saliency and Human Eye Fixations. ACM Transactions on Applied Perception 7, 2 (2010), 1--13. Google Scholar
Digital Library
- Philip Kortum and Wilson S Geisler. 1996. Implementation of a foveated image coding system for image bandwidth reduction. In Electronic Imaging: Science 8 Technology. International Society for Optics and Photonics, 350--360.Google Scholar
- Lee and Sanghoon. 2000. Foveated video compression and visual communications over wireless and wireline networks. Ph.D. Dissertation. Dept. of ECE, University of Texas at Austin.Google Scholar
- Chang Ha Lee, Amitabh Varshney, and David Jacobs. 2005. Mesh Saliency. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2005) 24, 3 (August 2005), 659--666.Google Scholar
- Sanghoon Lee, M. S. Pattichis, and A. C. Bovik. 2001. Foveated video compression with optimal rate control. IEEE Transactions on Image Processing 10, 7 (Jul 2001), 977--992. Google Scholar
Digital Library
- Sanghoon Lee, M. S. Pattichis, and A. C. Bovik. 2002. Foveated video quality assessment. IEEE Transactions on Multimedia 4, 1 (Mar 2002), 129--132. Google Scholar
Digital Library
- Marc Levoy and Ross Whitaker. 1990. Gaze-directed volume rendering. In Proceedings of the 1990 Symposium on Interactive 3D Graphics (I3D '90). ACM, New York, NY, USA, 217--223. Google Scholar
Digital Library
- Amazon Lumberyard. 2017. Amazon Lumberyard Bistro, Open Research Content Archive (ORCA). http://developer.nvidia.com/orca/amazon-lumberyard-bistroGoogle Scholar
- Bochang Moon, Jose A. Iglesias-Guitian, Steven McDonagh, and Kenny Mitchell. 2017. Noise reduction on G-buffers for Monte Carlo filtering. Computer Graphics Forum 36, 8 (2017), 600--612.Google Scholar
Cross Ref
- Toshikazu Ohshima, Hiroyuki Yamamoto, and Hideyuki Tamura. 1996. Gaze-directed adaptive rendering for interacting with virtual space. In Proceedings of the 1996 Virtual Reality Annual International Symposium (VRAIS 96) (VRAIS '96). IEEE Computer Society, Washington, DC, USA, 103-110, 267. Google Scholar
Digital Library
- C. Papadopoulos and A. E. Kaufman. 2013. Acuity-driven gigapixel visualization. IEEE Transactions on Visualization and Computer Graphics 19, 12 (Dec 2013), 2886--2895. Google Scholar
Digital Library
- Kashinath D Patil. 1975. Cochran's Q test: Exact distribution. J. Amer. Statist. Assoc. 70, 349 (1975), 186--189.Google Scholar
Cross Ref
- Anjul Patney, Joohwan Kim, Marco Salvi, Anton Kaplanyan, Chris Wyman, Nir Benty, Aaron Lefohn, and David Luebke. 2016a. Perceptually-based foveated virtual reality. In ACM SIGGRAPH 2016 Emerging Technologies (SIGGRAPH '16). ACM, New York, NY, USA, Article 17, 2 pages. Google Scholar
Digital Library
- Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. 2016b. Towards foveated rendering for gaze-tracked virtual reality. ACM Trans. Graph. 35, 6, Article 179 (Nov. 2016), 12 pages. Google Scholar
Digital Library
- T Pengo, A Muñoz-Barrutia, and C Ortiz-de solórzano. 2009. Halton sampling for autofocus. Journal of Microscopy 235, 1 (2009), 50--58.Google Scholar
Cross Ref
- Matt Pharr and Greg Humphreys. 2010. Physically Based rendering, second edition: from theory to implementation (2nd ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. Google Scholar
Digital Library
- D. Pohl, X. Zhang, and A. Bulling. 2016. Combining eye tracking with optimizations for lens astigmatism in modern wide-angle HMDs. In 2016 IEEE Virtual Reality (VR). 269--270.Google Scholar
- Jonathan Ragan-Kelley, Jaakko Lehtinen, Jiawen Chen, Michael Doggett, and Frédo Durand. 2011. Decoupled sampling for graphics pipelines. ACM Trans. Graph. 30, 3, Article 17 (May 2011), 17 pages. Google Scholar
Digital Library
- T. H. Reeves and J. A. Robinson. 1996. Adaptive foveation of MPEG video. In Proceedings of the Fourth ACM International Conference on Multimedia (MULTIMEDIA '96). ACM, New York, NY, USA, 231--241. Google Scholar
Digital Library
- A. Said and W. A. Pearlman. 1996. A new, fast, and efficient image codec based on set partitioning in hierarchical trees. IEEE Transactions on Circuits and Systems for Video Technology 6, 3 (Jun 1996), 243--250. Google Scholar
Digital Library
- Kai Selgrad, Christian Reintges, Dominik Penk, Pascal Wagner, and Marc Stamminger. 2015. Real-time depth of field using multi-layer filtering. In Proceedings of the 19th Symposium on Interactive 3D Graphics and Games (i3D '15). ACM, New York, NY, USA, 121--127. Google Scholar
Digital Library
- Jerome M Shapiro. 1993. Embedded image coding using zerotrees of wavelet coefficients. IEEE Transactions on Signal Processing 41, 12 (1993), 3445--3462. Google Scholar
Digital Library
- Hamid R. Sheikh, Brian L. Evans, and Alan C. Bovik. 2003. Real-time foveation techniques for low bit rate video coding. Real-Time Imaging 9, 1 (Feb. 2003), 27--40. Google Scholar
Digital Library
- H. R. Sheikh, S. Liu, Z. Wang, and A. C. Bovik. 2002. Foveated multipoint videoconferencing at low bit rates. In 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing, Vol. 2. II-2069-II-2072.Google Scholar
- Michael Stengel, Steve Grogorick, Martin Eisemann, and Marcus Magnor. 2016. Adaptive image-space sampling for gaze-contingent real-time rendering. Computer Graphics Forum 35, 4 (2016), 129--139. Google Scholar
Digital Library
- Qi Sun, Fu-Chung Huang, Joohwan Kim, Li-Yi Wei, David Luebke, and Arie Kaufman. 2017. Perceptually-guided foveation for light field displays. ACM Trans. Graph. 36, 6, Article 192 (Nov. 2017), 13 pages. Google Scholar
Digital Library
- Nicholas T. Swafford, José A. Iglesias-Guitian, Charalampos Koniaris, Bochang Moon, Darren Cosker, and Kenny Mitchell. 2016. User, metric, and computational evaluation of foveated rendering methods. In Proceedings of the ACM Symposium on Applied Perception (SAP '16). ACM, New York, NY, USA, 7--14. Google Scholar
Digital Library
- K. Vaidyanathan, M. Salvi, R. Toth, T. Foley, T. Akenine-Möller, J. Nilsson, J. Munkberg, J. Hasselgren, M. Sugihara, P. Clarberg, T. Janczak, and A. Lefohn. 2014. Coarse pixel shading. In Proceedings of High Performance Graphics (HPG '14). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 9--18. http://dl.acm.org/citation.cfm?id=2980009.2980011 Google Scholar
Digital Library
- Margarita Vinnikov and Robert S Allison. 2014. Gaze-contingent depth of field in realistic scenes: The user experience. In Proceedings of the Symposium on Eye Tracking Research and Applications. ACM, 119--126. Google Scholar
Digital Library
- Zhou Wang and Alan C Bovik. 2001. Embedded foveation image coding. IEEE Transactions on Image Processing 10, 10 (2001), 1397--1410. Google Scholar
Digital Library
- Zhou Wang and Alan C Bovik. 2005. Foveated image and video coding. In Digitial Video Image Quality and Perceptual Coding. 1--28.Google Scholar
- M. Weier, M. Stengel, T. Roth, P. Didyk, E. Eisemann, M. Eisemann, S. Grogorick, A. Hinkenjann, E. Kruijff, M. Magnor, K. Myszkowski, and P. Slusallek. 2017. Perception-driven accelerated rendering. Comput. Graph. Forum 36, 2 (May 2017), 611--643. Google Scholar
Digital Library
- Frank W Weymouth. 1958. Visual sensory units and the minimal angle of resolution. American Journal of Ophthalmology 46, 1 (1958), 102--113.Google Scholar
Cross Ref
Index Terms
Kernel Foveated Rendering
Recommendations
Luminance-contrast-aware foveated rendering
Current rendering techniques struggle to fulfill quality and power efficiency requirements imposed by new display devices such as virtual reality headsets. A promising solution to overcome these problems is foveated rendering, which exploits gaze ...
User, metric, and computational evaluation of foveated rendering methods
SAP '16: Proceedings of the ACM Symposium on Applied PerceptionPerceptually lossless foveated rendering methods exploit human perception by selectively rendering at different quality levels based on eye gaze (at a lower computational cost) while still maintaining the user's perception of a full quality render. We ...
Perceptually-based foveated virtual reality
SIGGRAPH '16: ACM SIGGRAPH 2016 Emerging TechnologiesHumans have two distinct vision systems: foveal and peripheral vision. Foveal vision is sharp and detailed, while peripheral vision lacks fidelity. The difference in characteristics of the two systems enable recently popular foveated rendering systems, ...






Comments