Towards retina-quality VR video streaming: 15ms could save you 80% of your bandwidth

Abstract

Virtual reality systems today cannot yet stream immersive, retina-quality virtual reality video over a network. One of the greatest challenges to this goal is the sheer data rates required to transmit retina-quality video frames at high resolutions and frame rates. Recent work has leveraged the decay of visual acuity in human perception in novel gaze-contingent video compression techniques. In this paper, we show that reducing the motion-to-photon latency of a system itself is a key method for improving the compression ratio of gaze-contingent compression. Our key finding is that a client and streaming server system with sub-15ms latency can achieve 5x better compression than traditional techniques while also using simpler software algorithms than previous work.

References

1. Rachel Albert, Anjul Patney, David Luebke, and Joohwan Kim. 2017. Latency Requirements for Foveated Rendering in Virtual Reality. ACM Transactions on Applied Perception 14, 4 (Sept. 2017), 25:1–25:13. 1544-3558
2. Stephen J. Anderson, Kathy T. Mullen, and Robert F. Hess. 1991. Human Peripheral Spatial Resolution for Achromatic and Chromatic Stimuli: Limits Imposed By Optical and Retinal Factors. The Journal of Physiology 442, 1 (1991), 47–64.
3. Anastasios N. Angelopoulos, Julien N.P. Martel, Amit P. Kohli, Jörg Conradt, and Gordon Wetzstein. 2021. Event-Based Near-Eye Gaze Tracking Beyond 10,000 Hz. IEEE Transactions on Visualization and Computer Graphics 27, 5 (2021), 2577–2586.
4. Jean-Baptiste Bernard, Scherlen Anne-Catherine, and Castet Eric. 2007. Page Mode Reading With Simulated Scotomas: A Modest Effect of Interline Spacing on Reading Speed. Vision research 47, 28 (2007), 3447–3459.
5. Christopher J. Bockisch and Joel M. Miller. 1999. Different Motor Systems Use Similar Damped Extraretinal Eye Position Information. Vision research 39, 5 (1999), 1025–1038.
6. Roger H.S. Carpenter. 1988. Movements of the Eyes, 2nd Rev. Pion Limited.
7. Jiawen Chen, Miao Hu, Zhenxiao Luo, Zelong Wang, and Di Wu. 2020. SR360: Boosting 360-Degree Video Streaming with Super-Resolution. In Proceedings of the 30th ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (Istanbul, Turkey) (NOSSDAV '20). Association for Computing Machinery, New York, NY, USA, 1–6.
8. Eduardo Cuervo, Krishna Chintalapudi, and Manikanta Kotaru. 2018. Creating the Perfect Illusion: What Will It Take to Create Life-Like Virtual Reality Headsets?. In Proceedings of the 19th International Workshop on Mobile Computing Systems & Applications (Tempe, Arizona, USA) (HotMobile '18). Association for Computing Machinery, New York, NY, USA, 7–12.
9. Michael F. Deering. 1998. The Limits of Human Vision. In 2nd International Immersive Projection Technology Workshop, Vol. 2. 1.
10. Sebastian Friston, Tobias Ritschel, and Anthony Steed. 2019. Perceptual Rasterization for Head-Mounted Display Image Synthesis. ACM Transactions on Graphics 38, 4, Article 97 (July 2019), 14 pages. 0730-0301
11. Wilson S. Geisler and Jeffrey S. Perry. 1998. Real-Time Foveated Multiresolution System for Low-Bandwidth Video Communication. In Human Vision and Electronic Imaging III, Bernice E. Rogowitz and Thrasyvoulos N. Pappas (Eds.), Vol. 3299. International Society for Optics and Photonics, SPIE, 294–305.
12. Yu Guan, Chengyuan Zheng, Xinggong Zhang, Zongming Guo, and Junchen Jiang. 2019. Pano: Optimizing 360 Video Streaming with a Better Understanding of Quality Perception. In Proceedings of the ACM Special Interest Group on Data Communication (Beijing, China) (SIGCOMM '19). Association for Computing Machinery, New York, NY, USA, 394–407.
13. Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. 2012. Foveated 3D graphics. ACM Transactions on Graphics 31, 6 (Nov. 2012), 164:1–164:10. 0730-0301
14. horsten Hansen, Lars Pracejus, and Karl R. Gegenfurtner. 2009. Color Perception in the Intermediate Periphery of the Visual Field. Journal of Vision 9, 4 (04 2009), 26–26. 1534-7362
15. . Hartmann, B. Lachenmayr, and H. Brettel. 1979. The Peripheral Critical Flicker Frequency. Vision Research 19, 9 (1979), 1019–1023. 0042-6989
16. azi Karam Illahi, Thomas Van Gemert, Matti Siekkinen, Enrico Masala, Antti Oulasvirta, and Antti Ylä-Jääski. 2020. Cloud Gaming with Foveated Video Encoding. ACM Transactions on Multimedia Compututing, Communications, and Applications 16, 1, Article 7 (Feb. 2020), 24 pages. 1551-6857
17. attis Jeppsson, Håvard Espeland, Tomas Kupka, Ragnar Langseth, Andreas Petlund, Peng Qiaoqiao, Chuansong Xue, Konstantin Pogorelov, Micheal Riegler, Dag Johansen, Carsten Griwodz, and Pål Halvorsen. 2018. Efficient Live and On-Demand Tiled HEVC 360 VR Video Streaming. In 2018 IEEE International Symposium on Multimedia (ISM). 81–88.
18. nton S. Kaplanyan, Anton Sochenov, Thomas Leimkühler, Mikhail Okunev, Todd Goodall, and Gizem Rufo. 2019. DeepFovea: Neural Reconstruction for Foveated Rendering and Video Compression Using Learned Statistics of Natural Videos. ACM Transactions on Graphics 38, 6, Article 212 (Nov. 2019), 13 pages. 0730-0301
19. Jonghyun Kim, Youngmo Jeong, Michael Stengel, Kaan Akşit, Rachel Albert, Ben Boudaoud, Trey Greer, Joohwan Kim, Ward Lopes, Zander Majercik, Peter Shirley, Josef Spjut, Morgan McGuire, and David Luebke. 2019. Foveated AR: Dynamically-Foveated Augmented Reality Display. ACM Transactions on Graphics 38, 4, Article 99 (July 2019), 15 pages. 0730-0301
20. Arnold Knapp. 1938. An Introduction to Clinical Perimetry. Archives of Ophthalmology 20, 6 (1938), 1116–1117.
21. G. A. Koulieris, K. Akşit, M. Stengel, R. K. Mantiuk, K. Mania, and C. Richardt. 2019. Near-Eye Display and Tracking Technologies for Virtual and Augmented Reality. Computer Graphics Forum 38, 2 (2019), 493–519.
22. Eileen Kowler. 2011. Eye Movements: The Past 25 Years. Vision Research 51, 13 (2011), 1457–1483. 0042-6989 Vision Research 50th Anniversary Issue: Part 2.
23. Brooke Krajancich, Petr Kellnhofer, and Gordon Wetzstein. 2021. A Perceptual Model for Eccentricity-dependent Spatio-temporal Flicker Fusion and its Applications to Foveated Graphics. arXiv preprint arXiv:2104.13514 (2021).
24. Sanghoon Lee, M.S. Pattichis, and A.C. Bovik. 2001. Foveated Video Compression With Optimal Rate Control. IEEE Transactions on Image Processing 10, 7 (July 2001), 977–992. 1941-0042
25. Lester C. Loschky and Gary S. Wolverton. 2007. How Late Can You Update Gaze-Contingent Multiresolutional Displays without Detection? ACM Transactions on Multimedia Computing, Communications, and Applications 3, 4, Article 7 (Dec. 2007), 10 pages. 1551-6857
26. David Luebke and Benjamin Hallen. 2001. Perceptually Driven Simplification for Interactive Rendering. In Eurographics Workshop on Rendering Techniques. Springer, 223–234.
27. John D. McCarthy, M. Angela Sasse, and Dimitrios Miras. 2004. Sharp or Smooth? Comparing the Effects of Quantization vs. Frame Rate for Streamed Video. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria) (CHI '04). Association for Computing Machinery, New York, NY, USA, 535–542.
28. Hunter Murphy and Andrew T Duchowski. 2001. Gaze-Contingent Level of Detail Rendering. EuroGraphics (2001).
29. T. Ohshima, H. Yamamoto, and H. Tamura. 1996. Gaze-Directed Adaptive Rendering for Interacting With Virtual Space. In Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium. 103–110.
30. Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. 2016. Towards Foveated Rendering for Gaze-Tracked Virtual Reality. ACM Transactions on Graphics 35, 6, Article 179 (Nov. 2016), 12 pages. 0730-0301
31. Eyal M. Reingold. 2014. Eye Tracking Research and Technology: Towards Objective Measurement of Data Quality. Visual Cognition 22, 3–4 (2014), 635–652.
32. J.G. Robson and Norma Graham. 1981. Probability Summation and Regional Variation in Contrast Sensitivity Across the Visual Field. Vision Research 21, 3 (1981), 409–418. 0042-6989
33. Miguel Fabian Romero-Rondón, Lucile Sassatelli, Frédéric Precioso, and Ramon Aparicio-Pardo. 2018. Foveated Streaming of Virtual Reality Videos. In Proceedings of the 9th ACM Multimedia Systems Conference (Amsterdam, Netherlands) (MMSys '18). Association for Computing Machinery, New York, NY, USA, 494–497.
34. Ruth Rosenholtz. 2016. Capabilities and Limitations of Peripheral Vision. Annual Review of Vision Science 2, 1 (2016), 437–457.
35. John Ross, M. Concetta Morrone, Michael E. Goldberg, and David C. Burr. 2001. Changes in Visual Perception at the Time of Saccades. Trends in Neurosciences 24, 2 (2001), 113–121. 0166-2236
36. Michele Rucci and Martina Poletti. 2015. Control and Functions of Fixational Eye Movements. Annual Review of Vision Science 1, 1 (2015), 499–518.
37. John Siderov and Ronald S. Harwerth. 1995. Stereopsis, Spatial Frequency and Retinal Eccentricity. Vision Research 35, 16 (1995), 2329–2337. 0042-6989
38. Niklas Stein, Diederick C Niehorster, Tamara Watson, Frank Steinicke, Katharina Rifai, Siegfried Wahl, and Markus Lappe. 2021. A Comparison of Eye Tracking Latencies Among Several Commercial Head-Mounted Displays. i-Perception 12, 1 (2021), 1–16.
39. Michael Stengel, Steve Grogorick, Martin Eisemann, and Marcus Magnor. 2016. Adaptive Image-Space Sampling for Gaze-Contingent Real-time Rendering. Computer Graphics Forum 35, 4 (July 2016), 129–139. 0167-7055
40. Hans Strasburger, Ingo Rentschler, and Martin Jüttner. 2011. Peripheral Vision and Pattern Recognition: A Review. Journal of Vision 11, 5 (12 2011), 1–82. 1534-7362
41. Liyang Sun, Yixiang Mao, Tongyu Zong, Yong Liu, and Yao Wang. 2020. Flocking-Based Live Streaming of 360-Degree Video. In Proceedings of the 11th ACM Multimedia Systems Conference (Istanbul, Turkey) (MMSys '20). Association for Computing Machinery, New York, NY, USA, 26–37.
42. Guanjun Tan, Yun-Han Lee, Tao Zhan, Jilin Yang, Sheng Liu, Dongfeng Zhao, and Shin-Tson Wu. 2018. Foveated Imaging for Near-Eye Displays. Optics Express 26, 19 (Sept. 2018), 25076–25085.
43. Timothy Terriberry. [n. d.]. Derf's Test Media Collection. https://media.xiph.org/video/derf/ Retrieved March, 2021 from
44. L. N. Thibos, F. E. Cheney, and D. J. Walsh. 1987. Retinal Limits to the Detection and Resolution of Gratings. Journal of the Optical Society of America A 4, 8 (Aug. 1987), 1524–1529.
45. Robin Thunström. 2014. Passive Gaze-Contingent Techniques Relation to System Latency. Master'sthesis. Blekinge Institute of Technology.
46. Oliver Wiedemann, Vlad Hosu, Hanhe Lin, and Dietmar Saupe. 2020. Foveated Video Coding for Real-Time Streaming Applications. In 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX). 1–6.

Index Terms

• Computing methodologies

  • Computer graphics

    • Graphics systems and interfaces

      • Virtual reality

    • Image compression

• Hardware

  • Communication hardware, interfaces and storage

    • Displays and imagers