Abstract
End-to-end latency in remote-rendering systems can reduce user task performance. This notably includes aiming tasks on game streaming services, which are presently below the standards of competitive first-person desktop gaming. We evaluate the latency-induced penalty on task completion time in a controlled environment and show that it can be significantly mitigated by adopting and modifying image and simulation-warping techniques from virtual reality, eliminating up to 80% of the penalty from 80 ms of added latency. This has potential to enable remote rendering for esports and increase the effectiveness of remote-rendered content creation and robotic teleoperation. We provide full experimental methodology, analysis, implementation details, and source code.
Supplemental Material
Available for Download
Supplemental movie, appendix, image and software files for, Post-Render Warp with Late Input Sampling Improves Aiming Under High Latency Conditions
- Sudhir Aggarwal, Hemant Banavar, Amit Khandelwal, Sarit Mukherjee, and Sampath Rangarajan. 2004. Accuracy in Dead-Reckoning Based Distributed Multi-Player Games. In Proceedings of 3rd ACM SIGCOMM Workshop on Network and System Support for Games (Portland, Oregon, USA) (NetGames '04). Association for Computing Machinery, New York, NY, USA, 161--165. https://doi.org/10.1145/1016540.1016559Google Scholar
Digital Library
- Dmitry Andreev. 2010. Real-Time Frame Rate up-Conversion for Video Games: Or How to Get from 30 to 60 Fps for "Free". In ACM SIGGRAPH 2010 Talks (Los Angeles, California) (SIGGRAPH '10). Association for Computing Machinery, New York, NY, USA, Article 16, 1 pages. https://doi.org/10.1145/1837026.1837047Google Scholar
Digital Library
- Dean Beeler, Ed Hutchins, and Paul Pedriana. 2016. Asynchronous Spacewarp. https://developer.oculus.com/blog/asynchronous-spacewarp/ Oculus Blog Post.Google Scholar
- Tom Beigbeder, Rory Coughlan, Corey Lusher, John Plunkett, Emmanuel Agu, and Mark Claypool. 2004. The Effects of Loss and Latency on User Performance in Unreal Tournament 2003®. In Proceedings of 3rd ACM SIGCOMM Workshop on Network and System Support for Games (Portland, Oregon, USA) (NetGames '04). Association for Computing Machinery, New York, NY, USA, 144--151. https://doi.org/10.1145/1016540.1016556Google Scholar
Digital Library
- Yahn W. Bernier. 2003. Latency Compensating Methods in Client/Server In-game Protocol Design and Optimization. (2003).Google Scholar
- Kuan-Ta Chen, Polly Huang, and Chin-Laung Lei. 2006. How sensitive are online gamers to network quality? Commun. ACM 49 (11 2006), 34--38. https://doi.org/10.1145/1167838.1167859Google Scholar
Digital Library
- Shenchang Eric Chen and Lance Williams. 1993. View Interpolation for Image Synthesis. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques (Anaheim, CA) (SIGGRAPH '93). Association for Computing Machinery, New York, NY, USA, 279--288. https://doi.org/10.1145/166117.166153Google Scholar
Digital Library
- Sharon Choy, Bernard Wong, Gwendal Simon, and Catherine Rosenberg. 2012. The Brewing Storm in Cloud Gaming: A Measurement Study on Cloud to End-User Latency. In Proceedings of the 11th Annual Workshop on Network and Systems Support for Games (Venice, Italy) (NetGames '12). IEEE Press, Article 2, 6 pages.Google Scholar
Cross Ref
- Sharon Choy, Bernard Wong, Gwendal Simon, and Catherine Rosenberg. 2014. A Hybrid Edge-Cloud Architecture for Reducing On-Demand Gaming Latency. Multimedia Systems 20 (10 2014), 503--519. https://doi.org/10.1007/s00530-014-0367-zGoogle Scholar
Digital Library
- Mark Claypool. 2005. The Effect of Latency on User Performance in Real-Time Strategy Games. Comput. Netw. 49, 1 (Sept. 2005), 52--70.Google Scholar
Cross Ref
- Mark Claypool and Kajal Claypool. 2006. Latency and Player Actions in Online Games. Commun. ACM 49, 11 (Nov. 2006), 40--45. https://doi.org/10.1145/1167838.1167860Google Scholar
Digital Library
- Mark Claypool and Kajal Claypool. 2010. Latency Can Kill: Precision and Deadline in Online Games. In Proceedings of the First Annual ACM SIGMM Conference on Multimedia Systems (Phoenix, Arizona, USA) (MMSys '10). Association for Computing Machinery, New York, NY, USA, 215--222. https://doi.org/10.1145/1730836.1730863Google Scholar
Digital Library
- Piotr Didyk, Elmar Eisemann, Tobias Ritschel, Karol Myszkowski, and Hans-Peter Seidel. 2010. Perceptually-motivated realtime temporal upsampling of 3D content for high-refresh-rate displays. Computer Graphics Forum (Proc. of Eurographics) 29, 2 (2010), 713--722. http://graphics.tudelft.nl/Publications-new/2010/DERMS10cGoogle Scholar
Cross Ref
- Sebastian Friston, Tobias Ritschel, and Anthony Steed. 2019. Perceptual Rasterization for Head-Mounted Display Image Synthesis. ACM Trans. Graph. 38, 4, Article 97 (July 2019), 14 pages. https://doi.org/10.1145/3306346.3323033Google Scholar
Digital Library
- Benjamin Goyette. 2016. Fighting Latency on Call of Duty Black Ops III. (2016). https://www.gdcvault.com/play/1023220/GDC.Google Scholar
- Akimitsu Hogge. 2019. Controller to Display Latency in 'Call of Duty'. (2019). https://www.gdcvault.com/play/1026327/GDC.Google Scholar
- Zenja Ivkovic, Ian Stavness, Carl Gutwin, and Steven Sutcliffe. 2015. Quantifying and Mitigating the Negative Effects of Local Latencies on Aiming in 3D Shooter Games. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI '15). Association for Computing Machinery, New York, NY, USA, 135--144. https://doi.org/10.1145/2702123.2702432Google Scholar
Digital Library
- Teemu Kämäräinen, Matti Siekkinen, Jukka Eerikäinen, and Antti Ylä-Jääski. 2018. CloudVR: Cloud Accelerated Interactive Mobile Virtual Reality. In Proceedings of the 26th ACM International Conference on Multimedia (Seoul, Republic of Korea) (MM '18). Association for Computing Machinery, New York, NY, USA, 1181--1189. https://doi.org/10.1145/3240508.3240620Google Scholar
Digital Library
- Babis Koniaris, Maggie Kosek, David Sinclair, and Kenny Mitchell. 2017. Real-Time Rendering with Compressed Animated Light Fields. In Proceedings of the 43rd Graphics Interface Conference (Edmonton, Alberta, Canada) (GI '17). Canadian Human-Computer Communications Society, Waterloo, CAN, 33--40.Google Scholar
Digital Library
- Z. Lai, Y. C. Hu, Y. Cui, L. Sun, N. Dai, and H. Lee. 2019. Furion: Engineering High-Quality Immersive Virtual Reality on Today's Mobile Devices. IEEE Transactions on Mobile Computing (2019), 1--1.Google Scholar
- John R. Lange, Peter A. Dinda, and Samuel Rossoff. 2008. Experiences with Client-Based Speculative Remote Display. In USENIX 2008 Annual Technical Conference (Boston, Massachusetts) (ATC'08). USENIX Association, USA, 419--432.Google Scholar
- Kyungmin Lee, David Chu, Eduardo Cuervo, Johannes Kopf, Yury Degtyarev, Sergey Grizan, Alec Wolman, and Jason Flinn. 2015. Outatime: Using Speculation to Enable Low-Latency Continuous Interaction for Mobile Cloud Gaming. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services (Florence, Italy) (MobiSys '15). Association for Computing Machinery, New York, NY, USA, 151--165. https://doi.org/10.1145/2742647.2742656Google Scholar
Digital Library
- Andrew Lippman. 1980. Movie-Maps: An Application of the Optical Videodisc to Computer Graphics. SIGGRAPH Comput. Graph. 14, 3 (July 1980), 32--42. https://doi.org/10.1145/965105.807465Google Scholar
Digital Library
- Gerrit Lochmann, Bernhard Reinert, Tobias Ritschel, Stefan Müller, and Hans-Peter Seidel. 2014. Real-time Reflective and Refractive Novel-view Synthesis.. In VMV. 9--16.Google Scholar
- Michael Long and Carl Gutwin. 2018. Characterizing and Modeling the Effects of Local Latency on Game Performance and Experience. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play (Melbourne, VIC, Australia) (CHI PLAY '18). Association for Computing Machinery, New York, NY, USA, 285--297. https://doi.org/10.1145/3242671.3242678Google Scholar
Digital Library
- I Scott MacKenzie. 2012. Human-computer interaction: An empirical research perspective. Newnes.Google Scholar
- William R. Mark, Leonard McMillan, and Gary Bishop. 1997. Post-Rendering 3D Warping. In Proceedings of the 1997 Symposium on Interactive 3D Graphics (Providence, Rhode Island, USA) (I3D '97). Association for Computing Machinery, New York, NY, USA, 7-ff. https://doi.org/10.1145/253284.253292Google Scholar
Digital Library
- Joanna Nelius. 2020. PCGamer GeForce Now beats Statdia in our input latency testing. https:https://www.pcgamer.com/geforce-now-beats-stadia-in-our-input-latency-testing/. Accessed: 2020-04-13.Google Scholar
- Newzoo. 2020. Global esports market report. Vol. April 2020 Update. Newzoo.Google Scholar
- Oculus Blog, Facebook 2019. Introducing ASW 2.0: Better Accuracy, Lower Latency. Oculus Blog, Facebook. https://www.oculus.com/blog/introducing-asw-2-point-0-better-accuracy-lower-latency/?fb_comment_id=2229988523715492_2230021217045556.Google Scholar
- Kjetil Raaen and Andreas Petlund. 2015. How Much Delay is There Really in Current Games?. In Proceedings of the 6th ACM Multimedia Systems Conference (Portland, Oregon) (MMSys '15). Association for Computing Machinery, New York, NY, USA, 89--92. https://doi.org/10.1145/2713168.2713188Google Scholar
Digital Library
- Bernhard Reinert, Johannes Kopf, Tobias Ritschel, Eduardo Cuervo, David Chu, and Hans-Peter Seidel. 2016. Proxy-guided image-based rendering for mobile devices. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 353--362.Google Scholar
- Riot Games. 2020. Valorant.Google Scholar
- Cheryl Savery and T. C. Graham. 2013. Timelines: Simplifying the Programming of Lag Compensation for the next Generation of Networked Games. Multimedia Syst. 19, 3 (June 2013), 271--287. https://doi.org/10.1007/s00530-012-0271-3Google Scholar
Digital Library
- Andre Schollmeyer, Simon Schneegans, Stephan Beck, Anthony Steed, and Bernd Froehlich. 2017. Efficient Hybrid Image Warping for High Frame-Rate Stereoscopic. IEEE Transactions on Visualization and Computer Graphics PP (01 2017), 1--1. https://doi.org/10.1109/TVCG.2017.2657078Google Scholar
Digital Library
- Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski. 1998. Layered Depth Images. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '98). Association for Computing Machinery, New York, NY, USA, 231--242. https://doi.org/10.1145/280814.280882Google Scholar
Digital Library
- Nathan Sheldon, Eric Girard, Seth Borg, Mark Claypool, and Emmanuel Agu. 2003. The Effect of Latency on User Performance in Warcraft III. In Proceedings of the 2nd Workshop on Network and System Support for Games (Redwood City, California) (NetGames '03). Association for Computing Machinery, New York, NY, USA, 3--14. https://doi.org/10.1145/963900.963901Google Scholar
Digital Library
- Josef Spjut, Ben Boudaoud, Kamran Binaee, Jonghyun Kim, Alexander Majercik, Morgan McGuire, David Luebke, and Joohwan Kim. 2019a. Latency of 30 ms Benefits First Person Targeting Tasks More Than Refresh Rate Above 60 Hz. In SIGGRAPH Asia 2019 Technical Briefs (Brisbane, QLD, Australia) (SA '19). Association for Computing Machinery, New York, NY, USA, 110--113. https://doi.org/10.1145/3355088.3365170Google Scholar
Digital Library
- Josef Spjut, Ben Boudaoud, Kamran Binaee, Alexander Majercik, Morgan McGuire, and Joohwan Kim. 2019b. FirstPerson-Science: Quantifying Psychophysics for First Person Shooter Tasks. In UCI Esports Conference.Google Scholar
- Josef Spjut, Ben Boudaoud, Pyarelal Knowles, Zander Majercik, Morgan McGuire, and Joohwan Kim. 2019c. FirstPerson-Science. https://github.com/NVlabs/abstract-fpsGoogle Scholar
- Art Tevs, Ivo Ihrke, and Hans-Peter Seidel. 2008. Maximum Mipmaps for Fast, Accurate, and Scalable Dynamic Height Field Rendering. In Proceedings of the 2008 Symposium on Interactive 3D Graphics and Games (Redwood City, California) (I3D '08). Association for Computing Machinery, New York, NY, USA, 183--190. https://doi.org/10.1145/1342250.1342279Google Scholar
Digital Library
- Valve Software. 2012. Counter-Strike: Global Offensive.Google Scholar
- J. M. P. van Waveren. 2016. The Asynchronous Time Warp for Virtual Reality on Consumer Hardware. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology (Munich, Germany) (VRST '16). Association for Computing Machinery, New York, NY, USA, 37--46. https://doi.org/10.1145/2993369.2993375Google Scholar
Digital Library
Index Terms
Post-Render Warp with Late Input Sampling Improves Aiming Under High Latency Conditions
Recommendations
Gaming at Warp Speed: Improving Aiming with Late Warp
SIGGRAPH '21: ACM SIGGRAPH 2021 Emerging TechnologiesLatency can make all the difference in competitive online games. Late warp is a class of techniques used in VR that can reduce latency in FPS games as well. Prior work has demonstrated these techniques can recover most of the player performance lost to ...
Latency can kill: precision and deadline in online games
MMSys '10: Proceedings of the first annual ACM SIGMM conference on Multimedia systemsThe growth in network capacities and availability has been accompanied by a proliferation of online games. While online games perform well under many network conditions, Internet delays can often degrade online game performance. The precise impact that ...
Fine-grained latency and loss measurements in the presence of reordering
SIGMETRICS '11: Proceedings of the ACM SIGMETRICS joint international conference on Measurement and modeling of computer systemsModern trading and cluster applications require microsecond latencies and almost no losses in data centers. This paper introduces an algorithm called FineComb that can estimate fine-grain end-to-end loss and latency measurements between edge routers in ...






Comments