skip to main content
research-article

Dissecting Cloud Gaming Performance with DECAF

Published:15 December 2021Publication History
Skip Abstract Section

Abstract

Cloud gaming platforms have witnessed tremendous growth over the past two years with a number of large Internet companies including Amazon, Facebook, Google, Microsoft, and Nvidia publicly launching their own platforms. While cloud gaming platforms continue to grow, the visibility in their performance and relative comparison is lacking. This is largely due to absence of systematic measurement methodologies which can generally be applied. As such, in this paper, we implement DECAF, a methodology to systematically analyze and dissect the performance of cloud gaming platforms across different game genres and game platforms. DECAF is highly automated and requires minimum manual intervention. By applying DECAF, we measure the performance of three commercial cloud gaming platforms including Google Stadia, Amazon Luna, and Nvidia GeForceNow, and uncover a number of important findings. First, we find that processing delays in the cloud comprise majority of the total round trip delay experienced by users, accounting for as much as 73.54% of total user-perceived delay. Second, we find that video streams delivered by cloud gaming platforms are characterized by high variability of bitrate, frame rate, and resolution. Platforms struggle to consistently serve 1080p/60 frames per second streams across different game genres even when the available bandwidth is 8-20× that of platform's recommended settings. Finally, we show that game platforms exhibit performance cliffs by reacting poorly to packet losses, in some cases dramatically reducing the delivered bitrate by up to 6.6× when loss rates increase from 0.1% to 1%. Our work has important implications for cloud gaming platforms and opens the door for further research on comprehensive measurement methodologies for cloud gaming.

References

  1. Amazon Luna. https://www.amazon.com/luna/landing-page.Google ScholarGoogle Scholar
  2. Cedexis-Citrix. https://www.cedexis.com/.Google ScholarGoogle Scholar
  3. FFmpeg. https://www.ffmpeg.org/.Google ScholarGoogle Scholar
  4. G.1072 : Opinion model predicting gaming quality of experience for cloud gaming services. https://www.itu.int/rec/TREC-G.1072/en.Google ScholarGoogle Scholar
  5. GaiKai. https://www.crunchbase.com/organization/gaikai.Google ScholarGoogle Scholar
  6. Gaming Monitor Alienware - AW2521HFL. https://www.bestbuy.com/site/alienware-aw2521hfl-24--5-ips-led-fhdfreesync-and-g-sync-compatible-monitor-displayport-hdmi-usb-lunar-light/6406940.p'skuId=6406940.Google ScholarGoogle Scholar
  7. GeForce Now. https://www.nvidia.com/en-us/geforce-now/.Google ScholarGoogle Scholar
  8. Google Stadia. https://stadia.google.com/.Google ScholarGoogle Scholar
  9. Hulu. https://www.hulu.com/.Google ScholarGoogle Scholar
  10. International Center For History Of Electronic Games. https://www.museumofplay.org/about/icheg.Google ScholarGoogle Scholar
  11. Netflix. https://www.netflix.com/.Google ScholarGoogle Scholar
  12. OnLive. http://onlive.com/.Google ScholarGoogle Scholar
  13. Ookla. https://www.ookla.com/.Google ScholarGoogle Scholar
  14. Open vSwitch. https://www.openvswitch.org/.Google ScholarGoogle Scholar
  15. Prime video. https://www.primevideo.com/.Google ScholarGoogle Scholar
  16. The Technology Behind A Low Latency Cloud Gaming Service. https://parsec.app/blog/description-of-parsectechnology-b2738dcc3842.Google ScholarGoogle Scholar
  17. The Cloud Gaming Market Will Pass the Billion-Dollar Mark in 2021 with Revenues of 1.4B. https: //newzoo.com/insights/articles/cloud-gaming-market-first-billion-dollar-year-23--7-million-paying-users-willgenerate-revenues-of-1--4-billion-in-2021/.Google ScholarGoogle Scholar
  18. ThousandEyes. https://www.thousandeyes.com/.Google ScholarGoogle Scholar
  19. VMAF - Video Multi-Method Assessment Fusion. https://github.com/Netflix/vmaf.Google ScholarGoogle Scholar
  20. X-Cloud cloud gaming. https://www.xbox.com/en-US/xbox-game-pass/cloud-gaming.Google ScholarGoogle Scholar
  21. X-Cloud game pass. https://www.xbox.com/en-US/xbox-game-pass/cloud-gaming?xr=shellnav.Google ScholarGoogle Scholar
  22. Yahoo. https://money.yahoo.com/cloud-gaming-market-growth-trends-141300368.html.Google ScholarGoogle Scholar
  23. Google Stadia bandwidth requirements - what kind of connection will you need to stream games? https://www. gamesradar.com/google-stadia-bandwidth-requirement/.Google ScholarGoogle Scholar
  24. Assassins Creed Valhalla. https://www.ubisoft.com/en-us/game/assassins-creed/valhalla.Google ScholarGoogle Scholar
  25. Cloud Gaming, Meet Facebook Gaming. https://www.facebook.com/fbgaminghome/blog/cloud-gaming-meetfacebook-gaming.Google ScholarGoogle Scholar
  26. Crew. https://www.ubisoft.com/en-us/game/the-crew/the-crew.Google ScholarGoogle Scholar
  27. Crew 2. https://www.ubisoft.com/en-us/game/the-crew/the-crew-2.Google ScholarGoogle Scholar
  28. Far Cry 5. https://www.ubisoft.com/en-us/game/far-cry/far-cry-5.Google ScholarGoogle Scholar
  29. I used cloud gaming exclusively for a month. Here's what happened. https://www.digitaltrends.com/gaming/i-usedcloud-gaming-for-a-month-shadow-stadia-geforce-now/.Google ScholarGoogle Scholar
  30. To succeed, cloud gaming needs to diappear. https://www.theverge.com/2021/6/23/22547334/cloud-gaming-xboxxcloud-microsoft-streaming-google-stadia-amazon.Google ScholarGoogle Scholar
  31. 5 reasons cloud gaming isn't doing it for me -- yet. https://www.cnet.com/tech/computing/5-reasons-cloud-gamingisnt-doing-it-for-me-yet/.Google ScholarGoogle Scholar
  32. Bandwidth, data usage, and stream quality. https://support.google.com/stadia/answer/9607891?hl=en.Google ScholarGoogle Scholar
  33. The Chromium Project. https://www.chromium.org/.Google ScholarGoogle Scholar
  34. DECAF: Dissecting Cloud Gaming Performace. https://github.com/decafCG/decaf.Google ScholarGoogle Scholar
  35. Geforce Now System Requirements. https://www.nvidia.com/en-us/geforce-now/system-reqs/.Google ScholarGoogle Scholar
  36. Iperf. https://iperf.fr/.Google ScholarGoogle Scholar
  37. Maxmind. https://www.maxmind.com/en/home.Google ScholarGoogle Scholar
  38. Python Imaging Library. https://pypi.org/project/Pillow/.Google ScholarGoogle Scholar
  39. Python-tesseract: an optical character recognition tool for python. https://pypi.org/project/pytesseract/.Google ScholarGoogle Scholar
  40. Python win32con Module. https://www.programcreek.com/python/index/475/win32con.Google ScholarGoogle Scholar
  41. pywin32. https://pypi.org/project/pywin32/.Google ScholarGoogle Scholar
  42. What Is Amazon Luna? https://www.amazon.com/gp/help/customer/display.html?nodeId=G8WBF7CLZX7W345R. Proc. ACM Meas. Anal. Comput. Syst., Vol. 5, No. 3, Article 31. Publication date: December 2021. 31:24 Hassan Iqbal, Ayesha Khalid, & Muhammad ShahzadGoogle ScholarGoogle Scholar
  43. Adnan Ahmed, Zubair Shafiq, and Amir Khakpour. QoE Analysis of a Large-Scale Live Video Streaming Event (ACM SIGMETRICS '16).Google ScholarGoogle Scholar
  44. Zahaib Akhtar, Yun Seong Nam, Ramesh Govindan, Sanjay Rao, Jessica Chen, Ethan Katz-Bassett, Bruno Ribeiro, Jibin Zhan, and Hui Zhang. Oboe: Auto-tuning Video ABR Algorithms to Network Conditions (ACM SIGCOMM '18).Google ScholarGoogle Scholar
  45. Todd Arnold, Jia He, Weifan Jiang, Matt Calder, Italo Cunha, Vasileios Giotsas, and Ethan Katz-Bassett. Cloud Provider Connectivity in the Flat Internet (ACM IMC '20).Google ScholarGoogle Scholar
  46. Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, and Hui Zhang. Developing a Predictive Model of Quality of Experience for Internet Video (ACM SIGCOMM '13).Google ScholarGoogle Scholar
  47. Hafiz Mohsin Bashir, Abdullah Bin Faisal, M Asim Jamshed, Peter Vondras, Ali Musa Iftikhar, Ihsan Ayyub Qazi, and Fahad R. Dogar. Reducing Tail Latency Using Duplication: A Multi-Layered Approach (ACM CoNEXT '19).Google ScholarGoogle Scholar
  48. Jan A Bergstra and CA Middelburg. ITU-T Recommendation G. 107: The E-Model, a computational model for use in transmission planning. (2003).Google ScholarGoogle Scholar
  49. Wei Cai, Ryan Shea, Chun-Ying Huang, Kuan-Ta Chen, Jiangchuan Liu, and Cheng-Hsin Hsu. A Survey on Cloud Gaming: Future of Computer Games. IEEE Access (2016).Google ScholarGoogle Scholar
  50. Eduardo F Camacho and Carlos Bordons Alba. Model predictive control. Springer science & business media.Google ScholarGoogle Scholar
  51. Marc Carrascosa and Boris Bellalta. Cloud-gaming:Analysis of Google Stadia traffic. arXiv:2009.09786 [cs.NI]Google ScholarGoogle Scholar
  52. Kuan-Ta Chen, Yu-Chun Chang, Po-Han Tseng, Chun-Ying Huang, and Chin-Laung Lei. Measuring the Latency of Cloud Gaming Systems (ACM MM '11).Google ScholarGoogle Scholar
  53. Kuan-Ta Chen, Polly Huang, and Chin-Laung Lei. How Sensitive Are Online Gamers to Network Quality? Commun. ACM (2006).Google ScholarGoogle Scholar
  54. Yi-Ching Chiu, Brandon Schlinker, Abhishek Balaji Radhakrishnan, Ethan Katz-Bassett, and Ramesh Govindan. Are We One Hop Away from a Better Internet? (ACM IMC '15).Google ScholarGoogle Scholar
  55. Jonathan Deber, Ricardo Jota, Clifton Forlines, and Daniel Wigdor. How Much Faster is Fast Enough? User Perception of Latency & Latency Improvements in Direct and Indirect Touch (ACM SIGCHI '15).Google ScholarGoogle Scholar
  56. Florin Dobrian, Vyas Sekar, Asad Awan, Ion Stoica, Dilip Joseph, Aditya Ganjam, Jibin Zhan, and Hui Zhang. Understanding the Impact of Video Quality on User Engagement (ACM SIGCOMM '11).Google ScholarGoogle Scholar
  57. Andrea Di Domenico, Gianluca Perna, Martino Trevisan, Luca Vassio, and Danilo Giordano. A network analysis on cloud gaming: Stadia, GeForce Now and PSNow. arXiv:2012.06774 [cs.NI]Google ScholarGoogle Scholar
  58. Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 aaai fall symposium series.Google ScholarGoogle Scholar
  59. Mohamed Hegazy, Khaled Diab, Mehdi Saeedi, Boris Ivanovic, Ihab Amer, Yang Liu, Gabor Sines, and Mohamed Hefeeda. Content-Aware Video Encoding for Cloud Gaming.Google ScholarGoogle Scholar
  60. Te-Yuan Huang, Nikhil Handigol, Brandon Heller, Nick McKeown, and Ramesh Johari. Confused, Timid, and Unstable: Picking a Video Streaming Rate is Hard (ACM IMC '12).Google ScholarGoogle Scholar
  61. Niels Justesen, Philip Bontrager, Julian Togelius, and Sebastian Risi. Deep learning for video game playing. IEEE Transactions on Games 12 (2019).Google ScholarGoogle Scholar
  62. Teemu Kämäräinen, Matti Siekkinen, Antti Ylä-Jääski, Wenxiao Zhang, and Pan Hui. A Measurement Study on Achieving Imperceptible Latency in Mobile Cloud Gaming. Association for Computing Machinery.Google ScholarGoogle Scholar
  63. Guillaume Lample and Devendra Singh Chaplot. Playing FPS games with deep reinforcement learning. In Thirty-First AAAI Conference on Artificial Intelligence.Google ScholarGoogle Scholar
  64. Xing Liu, Bo Han, Feng Qian, and Matteo Varvello. LIME: Understanding Commercial 360° Live Video Streaming Services (MMSys '19).Google ScholarGoogle Scholar
  65. Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh. Neural Adaptive Video Streaming with Pensieve (ACM SIGCOMM '17).Google ScholarGoogle Scholar
  66. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature 518, 7540 (2015), 529--533.Google ScholarGoogle Scholar
  67. Donald A. Norman. The design of everyday things. Basic Books.Google ScholarGoogle ScholarDigital LibraryDigital Library
  68. Jesse Schell. The art of game design: a book of lenses. CRC Press, Taylor & Francis Group.Google ScholarGoogle ScholarDigital LibraryDigital Library
  69. Steven Schmidt, Saman Zadtootaghaj, Saeed Shafiee Sabet, and Sebastian Möller. Modeling and Understanding the Quality of Experience of Online Mobile Gaming Services (IEEE QoMEX '21).Google ScholarGoogle Scholar
  70. C.E. Shannon. Communication in the Presence of Noise. Proceedings of the IRE (1949).Google ScholarGoogle ScholarCross RefCross Ref
  71. Iraj Sodagar. The MPEG-DASH Standard for Multimedia Streaming Over the Internet. IEEE MultiMedia (2011).Google ScholarGoogle ScholarDigital LibraryDigital Library
  72. German Sviridov, Cedric Beliard, Andrea Bianco, Paolo Giaccone, and Dario Rossi. Removing human players from the loop: AI-assisted assessment of Gaming QoE (IEEE INFOCOM WKSHPS '20).Google ScholarGoogle Scholar
  73. Bolun Wang, Xinyi Zhang, Gang Wang, Haitao Zheng, and Ben Y. Zhao. Anatomy of a Personalized Livestreaming System (ACM IMC '16). Proc. ACM Meas. Anal. Comput. Syst., Vol. 5, No. 3, Article 31. Publication date: December 2021. Dissecting Cloud Gaming Performance with DECAF 31:25Google ScholarGoogle Scholar
  74. S. Wang and S. Dey. Modeling and Characterizing User Experience in a Cloud Server Based Mobile Gaming Approach (IEEE GLOBECOM '09).Google ScholarGoogle Scholar
  75. Marek Wydmuch, Micha Kempka, and Wojciech Jakowski. Vizdoom competitions: Playing doom from pixels. IEEE Transactions on Games 11, 3 (2018), 248--259.Google ScholarGoogle Scholar
  76. Zheng Xue, Di Wu, Jian He, Xiaojun Hei, and Yong Liu. Playing High-End Video Games in the Cloud: A Measurement Study. IEEE Transactions on Circuits and Systems for Video Technology (2015).Google ScholarGoogle ScholarDigital LibraryDigital Library
  77. Jun Yi, Shiqing Luo, and Zhisheng Yan. A Measurement Study of YouTube 360° Live Video Streaming (ACM NOSSDAV '19).Google ScholarGoogle Scholar
  78. Saman Zadtootaghaj, Steven Schmidt, Saeed Shafiee Sabet, Sebastian Möller, and Carsten Griwodz. Quality estimation models for gaming video streaming services using perceptual video quality dimensions (ACM MM '20).Google ScholarGoogle Scholar
  79. Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Relational deep reinforcement learning. arXiv preprint arXiv:1806.01830 (2018).Google ScholarGoogle Scholar
  80. Timothy Zhu, Alexey Tumanov, Michael A. Kozuch, Mor Harchol-Balter, and Gregory R. Ganger. PriorityMeister: Tail Latency QoS for Shared Networked Storage (ACM SOCC '14).Google ScholarGoogle Scholar

Index Terms

  1. Dissecting Cloud Gaming Performance with DECAF

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!