skip to main content
research-article

Post-Render Warp with Late Input Sampling Improves Aiming Under High Latency Conditions

Published:26 August 2020Publication History
Skip Abstract Section

Abstract

End-to-end latency in remote-rendering systems can reduce user task performance. This notably includes aiming tasks on game streaming services, which are presently below the standards of competitive first-person desktop gaming. We evaluate the latency-induced penalty on task completion time in a controlled environment and show that it can be significantly mitigated by adopting and modifying image and simulation-warping techniques from virtual reality, eliminating up to 80% of the penalty from 80 ms of added latency. This has potential to enable remote rendering for esports and increase the effectiveness of remote-rendered content creation and robotic teleoperation. We provide full experimental methodology, analysis, implementation details, and source code.

Skip Supplemental Material Section

Supplemental Material

3406187.mp4

Presentation Video

References

  1. Sudhir Aggarwal, Hemant Banavar, Amit Khandelwal, Sarit Mukherjee, and Sampath Rangarajan. 2004. Accuracy in Dead-Reckoning Based Distributed Multi-Player Games. In Proceedings of 3rd ACM SIGCOMM Workshop on Network and System Support for Games (Portland, Oregon, USA) (NetGames '04). Association for Computing Machinery, New York, NY, USA, 161--165. https://doi.org/10.1145/1016540.1016559Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Dmitry Andreev. 2010. Real-Time Frame Rate up-Conversion for Video Games: Or How to Get from 30 to 60 Fps for "Free". In ACM SIGGRAPH 2010 Talks (Los Angeles, California) (SIGGRAPH '10). Association for Computing Machinery, New York, NY, USA, Article 16, 1 pages. https://doi.org/10.1145/1837026.1837047Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Dean Beeler, Ed Hutchins, and Paul Pedriana. 2016. Asynchronous Spacewarp. https://developer.oculus.com/blog/asynchronous-spacewarp/ Oculus Blog Post.Google ScholarGoogle Scholar
  4. Tom Beigbeder, Rory Coughlan, Corey Lusher, John Plunkett, Emmanuel Agu, and Mark Claypool. 2004. The Effects of Loss and Latency on User Performance in Unreal Tournament 2003®. In Proceedings of 3rd ACM SIGCOMM Workshop on Network and System Support for Games (Portland, Oregon, USA) (NetGames '04). Association for Computing Machinery, New York, NY, USA, 144--151. https://doi.org/10.1145/1016540.1016556Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Yahn W. Bernier. 2003. Latency Compensating Methods in Client/Server In-game Protocol Design and Optimization. (2003).Google ScholarGoogle Scholar
  6. Kuan-Ta Chen, Polly Huang, and Chin-Laung Lei. 2006. How sensitive are online gamers to network quality? Commun. ACM 49 (11 2006), 34--38. https://doi.org/10.1145/1167838.1167859Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Shenchang Eric Chen and Lance Williams. 1993. View Interpolation for Image Synthesis. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques (Anaheim, CA) (SIGGRAPH '93). Association for Computing Machinery, New York, NY, USA, 279--288. https://doi.org/10.1145/166117.166153Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Sharon Choy, Bernard Wong, Gwendal Simon, and Catherine Rosenberg. 2012. The Brewing Storm in Cloud Gaming: A Measurement Study on Cloud to End-User Latency. In Proceedings of the 11th Annual Workshop on Network and Systems Support for Games (Venice, Italy) (NetGames '12). IEEE Press, Article 2, 6 pages.Google ScholarGoogle ScholarCross RefCross Ref
  9. Sharon Choy, Bernard Wong, Gwendal Simon, and Catherine Rosenberg. 2014. A Hybrid Edge-Cloud Architecture for Reducing On-Demand Gaming Latency. Multimedia Systems 20 (10 2014), 503--519. https://doi.org/10.1007/s00530-014-0367-zGoogle ScholarGoogle ScholarDigital LibraryDigital Library
  10. Mark Claypool. 2005. The Effect of Latency on User Performance in Real-Time Strategy Games. Comput. Netw. 49, 1 (Sept. 2005), 52--70.Google ScholarGoogle ScholarCross RefCross Ref
  11. Mark Claypool and Kajal Claypool. 2006. Latency and Player Actions in Online Games. Commun. ACM 49, 11 (Nov. 2006), 40--45. https://doi.org/10.1145/1167838.1167860Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Mark Claypool and Kajal Claypool. 2010. Latency Can Kill: Precision and Deadline in Online Games. In Proceedings of the First Annual ACM SIGMM Conference on Multimedia Systems (Phoenix, Arizona, USA) (MMSys '10). Association for Computing Machinery, New York, NY, USA, 215--222. https://doi.org/10.1145/1730836.1730863Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Piotr Didyk, Elmar Eisemann, Tobias Ritschel, Karol Myszkowski, and Hans-Peter Seidel. 2010. Perceptually-motivated realtime temporal upsampling of 3D content for high-refresh-rate displays. Computer Graphics Forum (Proc. of Eurographics) 29, 2 (2010), 713--722. http://graphics.tudelft.nl/Publications-new/2010/DERMS10cGoogle ScholarGoogle ScholarCross RefCross Ref
  14. Sebastian Friston, Tobias Ritschel, and Anthony Steed. 2019. Perceptual Rasterization for Head-Mounted Display Image Synthesis. ACM Trans. Graph. 38, 4, Article 97 (July 2019), 14 pages. https://doi.org/10.1145/3306346.3323033Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Benjamin Goyette. 2016. Fighting Latency on Call of Duty Black Ops III. (2016). https://www.gdcvault.com/play/1023220/GDC.Google ScholarGoogle Scholar
  16. Akimitsu Hogge. 2019. Controller to Display Latency in 'Call of Duty'. (2019). https://www.gdcvault.com/play/1026327/GDC.Google ScholarGoogle Scholar
  17. Zenja Ivkovic, Ian Stavness, Carl Gutwin, and Steven Sutcliffe. 2015. Quantifying and Mitigating the Negative Effects of Local Latencies on Aiming in 3D Shooter Games. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI '15). Association for Computing Machinery, New York, NY, USA, 135--144. https://doi.org/10.1145/2702123.2702432Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Teemu Kämäräinen, Matti Siekkinen, Jukka Eerikäinen, and Antti Ylä-Jääski. 2018. CloudVR: Cloud Accelerated Interactive Mobile Virtual Reality. In Proceedings of the 26th ACM International Conference on Multimedia (Seoul, Republic of Korea) (MM '18). Association for Computing Machinery, New York, NY, USA, 1181--1189. https://doi.org/10.1145/3240508.3240620Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Babis Koniaris, Maggie Kosek, David Sinclair, and Kenny Mitchell. 2017. Real-Time Rendering with Compressed Animated Light Fields. In Proceedings of the 43rd Graphics Interface Conference (Edmonton, Alberta, Canada) (GI '17). Canadian Human-Computer Communications Society, Waterloo, CAN, 33--40.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Z. Lai, Y. C. Hu, Y. Cui, L. Sun, N. Dai, and H. Lee. 2019. Furion: Engineering High-Quality Immersive Virtual Reality on Today's Mobile Devices. IEEE Transactions on Mobile Computing (2019), 1--1.Google ScholarGoogle Scholar
  21. John R. Lange, Peter A. Dinda, and Samuel Rossoff. 2008. Experiences with Client-Based Speculative Remote Display. In USENIX 2008 Annual Technical Conference (Boston, Massachusetts) (ATC'08). USENIX Association, USA, 419--432.Google ScholarGoogle Scholar
  22. Kyungmin Lee, David Chu, Eduardo Cuervo, Johannes Kopf, Yury Degtyarev, Sergey Grizan, Alec Wolman, and Jason Flinn. 2015. Outatime: Using Speculation to Enable Low-Latency Continuous Interaction for Mobile Cloud Gaming. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services (Florence, Italy) (MobiSys '15). Association for Computing Machinery, New York, NY, USA, 151--165. https://doi.org/10.1145/2742647.2742656Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Andrew Lippman. 1980. Movie-Maps: An Application of the Optical Videodisc to Computer Graphics. SIGGRAPH Comput. Graph. 14, 3 (July 1980), 32--42. https://doi.org/10.1145/965105.807465Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Gerrit Lochmann, Bernhard Reinert, Tobias Ritschel, Stefan Müller, and Hans-Peter Seidel. 2014. Real-time Reflective and Refractive Novel-view Synthesis.. In VMV. 9--16.Google ScholarGoogle Scholar
  25. Michael Long and Carl Gutwin. 2018. Characterizing and Modeling the Effects of Local Latency on Game Performance and Experience. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play (Melbourne, VIC, Australia) (CHI PLAY '18). Association for Computing Machinery, New York, NY, USA, 285--297. https://doi.org/10.1145/3242671.3242678Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. I Scott MacKenzie. 2012. Human-computer interaction: An empirical research perspective. Newnes.Google ScholarGoogle Scholar
  27. William R. Mark, Leonard McMillan, and Gary Bishop. 1997. Post-Rendering 3D Warping. In Proceedings of the 1997 Symposium on Interactive 3D Graphics (Providence, Rhode Island, USA) (I3D '97). Association for Computing Machinery, New York, NY, USA, 7-ff. https://doi.org/10.1145/253284.253292Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Joanna Nelius. 2020. PCGamer GeForce Now beats Statdia in our input latency testing. https:https://www.pcgamer.com/geforce-now-beats-stadia-in-our-input-latency-testing/. Accessed: 2020-04-13.Google ScholarGoogle Scholar
  29. Newzoo. 2020. Global esports market report. Vol. April 2020 Update. Newzoo.Google ScholarGoogle Scholar
  30. Oculus Blog, Facebook 2019. Introducing ASW 2.0: Better Accuracy, Lower Latency. Oculus Blog, Facebook. https://www.oculus.com/blog/introducing-asw-2-point-0-better-accuracy-lower-latency/?fb_comment_id=2229988523715492_2230021217045556.Google ScholarGoogle Scholar
  31. Kjetil Raaen and Andreas Petlund. 2015. How Much Delay is There Really in Current Games?. In Proceedings of the 6th ACM Multimedia Systems Conference (Portland, Oregon) (MMSys '15). Association for Computing Machinery, New York, NY, USA, 89--92. https://doi.org/10.1145/2713168.2713188Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Bernhard Reinert, Johannes Kopf, Tobias Ritschel, Eduardo Cuervo, David Chu, and Hans-Peter Seidel. 2016. Proxy-guided image-based rendering for mobile devices. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 353--362.Google ScholarGoogle Scholar
  33. Riot Games. 2020. Valorant.Google ScholarGoogle Scholar
  34. Cheryl Savery and T. C. Graham. 2013. Timelines: Simplifying the Programming of Lag Compensation for the next Generation of Networked Games. Multimedia Syst. 19, 3 (June 2013), 271--287. https://doi.org/10.1007/s00530-012-0271-3Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Andre Schollmeyer, Simon Schneegans, Stephan Beck, Anthony Steed, and Bernd Froehlich. 2017. Efficient Hybrid Image Warping for High Frame-Rate Stereoscopic. IEEE Transactions on Visualization and Computer Graphics PP (01 2017), 1--1. https://doi.org/10.1109/TVCG.2017.2657078Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski. 1998. Layered Depth Images. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '98). Association for Computing Machinery, New York, NY, USA, 231--242. https://doi.org/10.1145/280814.280882Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Nathan Sheldon, Eric Girard, Seth Borg, Mark Claypool, and Emmanuel Agu. 2003. The Effect of Latency on User Performance in Warcraft III. In Proceedings of the 2nd Workshop on Network and System Support for Games (Redwood City, California) (NetGames '03). Association for Computing Machinery, New York, NY, USA, 3--14. https://doi.org/10.1145/963900.963901Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Josef Spjut, Ben Boudaoud, Kamran Binaee, Jonghyun Kim, Alexander Majercik, Morgan McGuire, David Luebke, and Joohwan Kim. 2019a. Latency of 30 ms Benefits First Person Targeting Tasks More Than Refresh Rate Above 60 Hz. In SIGGRAPH Asia 2019 Technical Briefs (Brisbane, QLD, Australia) (SA '19). Association for Computing Machinery, New York, NY, USA, 110--113. https://doi.org/10.1145/3355088.3365170Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Josef Spjut, Ben Boudaoud, Kamran Binaee, Alexander Majercik, Morgan McGuire, and Joohwan Kim. 2019b. FirstPerson-Science: Quantifying Psychophysics for First Person Shooter Tasks. In UCI Esports Conference.Google ScholarGoogle Scholar
  40. Josef Spjut, Ben Boudaoud, Pyarelal Knowles, Zander Majercik, Morgan McGuire, and Joohwan Kim. 2019c. FirstPerson-Science. https://github.com/NVlabs/abstract-fpsGoogle ScholarGoogle Scholar
  41. Art Tevs, Ivo Ihrke, and Hans-Peter Seidel. 2008. Maximum Mipmaps for Fast, Accurate, and Scalable Dynamic Height Field Rendering. In Proceedings of the 2008 Symposium on Interactive 3D Graphics and Games (Redwood City, California) (I3D '08). Association for Computing Machinery, New York, NY, USA, 183--190. https://doi.org/10.1145/1342250.1342279Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Valve Software. 2012. Counter-Strike: Global Offensive.Google ScholarGoogle Scholar
  43. J. M. P. van Waveren. 2016. The Asynchronous Time Warp for Virtual Reality on Consumer Hardware. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology (Munich, Germany) (VRST '16). Association for Computing Machinery, New York, NY, USA, 37--46. https://doi.org/10.1145/2993369.2993375Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Post-Render Warp with Late Input Sampling Improves Aiming Under High Latency Conditions

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          • Published in

            cover image Proceedings of the ACM on Computer Graphics and Interactive Techniques
            Proceedings of the ACM on Computer Graphics and Interactive Techniques  Volume 3, Issue 2
            August 2020
            218 pages
            EISSN:2577-6193
            DOI:10.1145/3420254
            Issue’s Table of Contents

            Copyright © 2020 ACM

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            • Published: 26 August 2020
            Published in pacmcgit Volume 3, Issue 2

            Permissions

            Request permissions about this article.

            Request Permissions

            Check for updates

            Qualifiers

            • research-article
            • Research
            • Refereed

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!