Perceptual rasterization for head-mounted display image synthesis

Abstract

We suggest a rasterization pipeline tailored towards the needs of HMDs, where latency and field-of-view requirements pose new challenges beyond those of traditional desktop displays. Instead of image warping for low latency, or using multiple passes for foveation, we show how both can be produced directly in a single perceptual rasterization pass. We do this with per-fragment ray-casting. This is enabled by derivations of tight space-time-fovea pixel bounds, introducing just enough flexibility for the requisite geometric tests, but retaining most of the simplicity and efficiency of the traditional rasterizaton pipeline. To produce foveated images, we rasterize to an image with spatially varying pixel density. To compensate for latency, we extend the image formation model to directly produce "rolling" images where the time at each pixel depends on its display location. Our approach overcomes limitations of warping with respect to disocclusions, object motion and view-dependent shading, as well as geometric aliasing artifacts in other foveated rendering techniques. A set of perceptual user studies demonstrates the efficacy of our approach.

References

1. Tomas Akenine-Möller, Jacob Munkberg, and Jon Hasselgren. 2007. Stochastic rasterization using time-continuous triangles. In Proc. Graphics Hardware. 9.
2. Tomas Akenine-Möller, Robert Toth, Jacob Munkberg, and Jon Hasselgren. 2012. Efficient Depth of Field Rasterization Using a Tile Test Based on Half-Space Culling. Comp. Graph. Forum 31, 1 (2012).
3. Michael Antonov. 2015. https://developer3.oculus.com/blog/asynchronous-timewarp-examined/.
4. Stefan Brabec, Thomas Annen, and Hans-Peter Seidel. 2002. Shadow mapping for hemispherical and omnidirectional light sources. Advances in Modelling, Animation and Rendering (2002), 397--408.
5. John Brosz, Faramarz F Samavati, M Sheelagh T Carpendale, and Mario Costa Sousa. 2007. Single camera flexible projection. In Proc. NPAR. 33--42.
6. J. S. Brunhaver, K. Fatahalian, and P. Hanrahan. 2010. Hardware Implementation of Micropolygon Rasterization with Motion and Defocus Blur. In Proc. HPG. 1--9.
7. Timothy J. Buker, Dennis A. Vincenzi, and John E. Deaton. 2012. The Effect of Apparent Latency on Simulator Sickness While Using a See-Through Helmet-Mounted Display: Reducing Apparent Latency With Predictive Compensation. Human Factors 54, 2 (2012), 235--249.
8. Shenchang Eric Chen and Lance Williams. 1993. View interpolation for image synthesis. In Proc. SIGGRAPH. 279--88.
9. PM Daniel and D Whitteridge. 1961. The representation of the visual field on the cerebral cortex in monkeys. J Physiology 159, 2 (1961), 203--21.
10. Piotr Didyk, Elmar Eisemann, Tobias Ritschel, Karol Myszkowski, and Hans-Peter Seidel. 2010. Perceptually-motivated Real-time Temporal Upsampling of 3D Content for High-refresh-rate Displays. Comp. Graph. Forum 29, 2 (2010), 713--22.
11. S.R. Ellis, B.D. Adelstein, S. Baumeler, G.J. Jense, and R.H. Jacoby. 1999. Sensor spatial distortion, visual latency, and update rate effects on 3D tracking in virtual environments. In Proc. VR. 218--21.
12. Kayvon Fatahalian, Edward Luong, Solomon Boulos, Kurt Akeley, William R Mark, and Pat Hanrahan. 2009. Data-parallel rasterization of micropolygons with defocus and motion blur. In Proc. HPG. 59--68.
13. FOVE. 2018. https://www.getfove.com/.
14. Sebastian Friston, Anthony Steed, Simon Tilbury, and Georgi Gaydadjiev. 2016. Construction and Evaluation of an Ultra Low Latency Frameless Renderer for VR. IEEE TVCG 22, 4 (2016), 1377--86.
15. George W Furnas. 1986. Generalized fisheye views. In Proc. CHI.
16. Jean-Dominique Gascuel, Nicolas Holzschuch, Gabriel Fournier, and Bernard Peroche. 2008. Fast non-linear projections using graphics hardware. In Proc. i3D. 107--14.
17. Ragnar Granit and Phyllis Harper. 1930. Comparative studies on the peripheral and central retina. J Physiology 95, 1 (1930), 211--28.
18. Gerard E Grossman, R John Leigh, LA Abel, Douglas J Lanska, and SE Thurston. 1988. Frequency and velocity of rotational head perturbations during locomotion. Exp. Brain Res. 70, 3 (1988), 470--6.
19. Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. 2012. Foveated 3D graphics. ACM Trans. Graph. (Proc. SIGGRAPH) 31, 6 (2012), 164.
20. Yong He, Yan Gu, and Kayvon Fatahalian. 2014. Extending the graphics pipeline with adaptive, multi-rate shading. ACM Trans. Graph (Proc. SIGGRAPH) 33, 4 (2014), 142.
21. Samuli Laine, Timo Aila, Tero Karras, and Jaakko Lehtinen. 2011. Clipless dual-space bounds for faster stochastic rasterization. ACM Trans. Graph (Proc. SIGGRAPH) 30, 4 (2011), 106.
22. Baoquan Liu, Li-Yi Wei, Xu Yang, Chongyang Ma, Ying-Qing Xu, Baining Guo, and Enhua Wu. 2011. Non-Linear Beam Tracing on a GPU. Comp. Graph. Forum 30, 8 (2011), 2156--69.
23. William R Mark, Leonard McMillan, and Gary Bishop. 1997. Post-rendering 3D warping. In Proc. i3D. 7--ff.
24. Morgan McGuire, Eric Enderton, Peter Shirley, and David Luebke. 2010. Real-time stochastic rasterization on conventional GPU architectures. In Proc. HPG. 173--82.
25. Xiaoxu Meng, Ruofei Du, Matthias Zwicker, and Amitabh Varshney. 2018. Kernel Foveated Rendering. Proc. ACM Comput. Graph. Interact. Tech. 1, 1 (2018), 5:1--5:20.
26. Nvidia. 2017. NV_clip_space_w_scaling OpenGL extension.
27. Oculus. 2018. https://developer.oculus.com/documentation/mobilesdk/0.4/concepts/mobile-timewarp-overview/.
28. Oculus VR. 2017. Asynchronous TimeWarp.
29. Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. 2016. Towards foveated rendering for gaze-tracked virtual reality. ACM Trans. Graph. (Proc. SIGGRAPH) 35, 6 (2016), 179.
30. Juan Pineda. 1988. A parallel algorithm for polygon rasterization. ACM SIGGRAPH Computer Graphics 22, 4 (1988), 17--20.
31. Voicu Popescu, Paul Rosen, and Nicoletta Adamo-Villani. 2009. The graph camera. ACM Trans. Graph. 28, 5 (2009), 158.
32. Matthew Regan and Ronald Pose. 1994. Priority rendering with a virtual reality address recalculation pipeline. In Proc. SIGGRAPH. 155--162.
33. Bernhard Reinert, Johannes Kopf, Tobias Ritschel, Eduardo Cuervo, David Chu, and Hans-Peter Seidel. 2016. Proxy-guided Image-based Rendering for Mobile Devices. Comp. Graph. Forum (Proc. Pacific Graphics) 35, 7 (2016).
34. Andre Schollmeyer, Simon Schneegans, Stephan Beck, Anthony Steed, and Bernd Froehlich. 2017. Efficient Hybrid Image Warping for High Frame-Rate Stereoscopic Rendering. IEEE Trans. Vis. and Comp. Graph. 23, 4 (2017), 1332--41.
35. Mel Slater. 2002. Presence and The Sixth Sense. Presence 11, 4 (2002), 435--439.
36. AAS Sluyterman. 2006. What is needed in LCD panels to achieve CRT-like motion portrayal? J SID 14, 8 (2006), 681--686.
37. Michael Stengel, Steve Grogorick, Martin Eisemann, Elmar Eisemann, and Marcus A Magnor. 2015. An affordable solution for binocular eye tracking and calibration in head-mounted displays. In Proc. ACM Multimedia. 15--24.
38. Michael Stengel, Steve Grogorick, Martin Eisemann, and Marcus Magnor. 2016. Adaptive Image-Space Sampling for Gaze-Contingent Real-time Rendering. Comp. Graph. Forum 35, 4 (2016), 129--39.
39. Qi Sun, Fu-Chung Huang, Joohwan Kim, Li-Yi Wei, David Luebke, and Arie Kaufman. 2017. Perceptually-guided foveation for light field displays. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 36, 6 (2017), 192.
40. Robert Toth, Jim Nilsson, and Tomas Akenine-Möller. 2016. Comparison of projection methods for rendering virtual reality. In Proc. HPG. 163--71.
41. Ingo Wald, Sven Woop, Carsten Benthin, Gregory S Johnson, and Manfred Ernst. 2014. Embree: a kernel framework for efficient CPU ray tracing. ACM Trans. Graph (Proc. SIGGRAPH) 33, 4 (2014), 143.
42. Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Proc. 13, 4 (2004), 600--12.
43. Martin Weier, Thorsten Roth, Ernst Kruijff, André Hinkenjann, Arsène Pérard-Gayot, Philipp Slusallek, and Yongmin Li. 2016. Foveated Real-Time Ray Tracing for Head-Mounted Displays. Comp. Graph. Forum 35, 7 (2016), 89--298.
44. Martin Weier, Michael Stengel, Thorsten Roth, Piotr Didyk, Elmar Eisemann, Martin Eisemann, Steve Grogorick, André Hinkenjann, Ernst Kruijff, Marcus Magnor, et al. 2017. Perception-driven Accelerated Rendering. Comp. Graph. Forum 36, 2 (2017), 611--43.
45. P.H. Wicksteed and F.M. Cornford. 1929. Aristotle. Physics. W. Heinemann.
46. Lei Yang, Yu-Chiu Tse, Pedro V Sander, Jason Lawrence, Diego Nehab, Hugues Hoppe, and Clara L Wilkins. 2011. Image-based bidirectional scene reprojection. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 30, 6 (2011), 150.

Index Terms

• Computing methodologies

  • Computer graphics

    • Graphics systems and interfaces

      • Perception

      • Virtual reality

    • Rendering

      • Rasterization