Abstract
Cloud gaming platforms have witnessed tremendous growth over the past two years with a number of large Internet companies including Amazon, Facebook, Google, Microsoft, and Nvidia publicly launching their own platforms. While cloud gaming platforms continue to grow, the visibility in their performance and relative comparison is lacking. This is largely due to absence of systematic measurement methodologies which can generally be applied. As such, in this paper, we implement DECAF, a methodology to systematically analyze and dissect the performance of cloud gaming platforms across different game genres and game platforms. DECAF is highly automated and requires minimum manual intervention. By applying DECAF, we measure the performance of three commercial cloud gaming platforms including Google Stadia, Amazon Luna, and Nvidia GeForceNow, and uncover a number of important findings. First, we find that processing delays in the cloud comprise majority of the total round trip delay experienced by users, accounting for as much as 73.54% of total user-perceived delay. Second, we find that video streams delivered by cloud gaming platforms are characterized by high variability of bitrate, frame rate, and resolution. Platforms struggle to consistently serve 1080p/60 frames per second streams across different game genres even when the available bandwidth is 8-20× that of platform's recommended settings. Finally, we show that game platforms exhibit performance cliffs by reacting poorly to packet losses, in some cases dramatically reducing the delivered bitrate by up to 6.6× when loss rates increase from 0.1% to 1%. Our work has important implications for cloud gaming platforms and opens the door for further research on comprehensive measurement methodologies for cloud gaming.
- Amazon Luna. https://www.amazon.com/luna/landing-page.Google Scholar
- Cedexis-Citrix. https://www.cedexis.com/.Google Scholar
- FFmpeg. https://www.ffmpeg.org/.Google Scholar
- G.1072 : Opinion model predicting gaming quality of experience for cloud gaming services. https://www.itu.int/rec/TREC-G.1072/en.Google Scholar
- GaiKai. https://www.crunchbase.com/organization/gaikai.Google Scholar
- Gaming Monitor Alienware - AW2521HFL. https://www.bestbuy.com/site/alienware-aw2521hfl-24--5-ips-led-fhdfreesync-and-g-sync-compatible-monitor-displayport-hdmi-usb-lunar-light/6406940.p'skuId=6406940.Google Scholar
- GeForce Now. https://www.nvidia.com/en-us/geforce-now/.Google Scholar
- Google Stadia. https://stadia.google.com/.Google Scholar
- Hulu. https://www.hulu.com/.Google Scholar
- International Center For History Of Electronic Games. https://www.museumofplay.org/about/icheg.Google Scholar
- Netflix. https://www.netflix.com/.Google Scholar
- OnLive. http://onlive.com/.Google Scholar
- Ookla. https://www.ookla.com/.Google Scholar
- Open vSwitch. https://www.openvswitch.org/.Google Scholar
- Prime video. https://www.primevideo.com/.Google Scholar
- The Technology Behind A Low Latency Cloud Gaming Service. https://parsec.app/blog/description-of-parsectechnology-b2738dcc3842.Google Scholar
- The Cloud Gaming Market Will Pass the Billion-Dollar Mark in 2021 with Revenues of 1.4B. https: //newzoo.com/insights/articles/cloud-gaming-market-first-billion-dollar-year-23--7-million-paying-users-willgenerate-revenues-of-1--4-billion-in-2021/.Google Scholar
- ThousandEyes. https://www.thousandeyes.com/.Google Scholar
- VMAF - Video Multi-Method Assessment Fusion. https://github.com/Netflix/vmaf.Google Scholar
- X-Cloud cloud gaming. https://www.xbox.com/en-US/xbox-game-pass/cloud-gaming.Google Scholar
- X-Cloud game pass. https://www.xbox.com/en-US/xbox-game-pass/cloud-gaming?xr=shellnav.Google Scholar
- Yahoo. https://money.yahoo.com/cloud-gaming-market-growth-trends-141300368.html.Google Scholar
- Google Stadia bandwidth requirements - what kind of connection will you need to stream games? https://www. gamesradar.com/google-stadia-bandwidth-requirement/.Google Scholar
- Assassins Creed Valhalla. https://www.ubisoft.com/en-us/game/assassins-creed/valhalla.Google Scholar
- Cloud Gaming, Meet Facebook Gaming. https://www.facebook.com/fbgaminghome/blog/cloud-gaming-meetfacebook-gaming.Google Scholar
- Crew. https://www.ubisoft.com/en-us/game/the-crew/the-crew.Google Scholar
- Crew 2. https://www.ubisoft.com/en-us/game/the-crew/the-crew-2.Google Scholar
- Far Cry 5. https://www.ubisoft.com/en-us/game/far-cry/far-cry-5.Google Scholar
- I used cloud gaming exclusively for a month. Here's what happened. https://www.digitaltrends.com/gaming/i-usedcloud-gaming-for-a-month-shadow-stadia-geforce-now/.Google Scholar
- To succeed, cloud gaming needs to diappear. https://www.theverge.com/2021/6/23/22547334/cloud-gaming-xboxxcloud-microsoft-streaming-google-stadia-amazon.Google Scholar
- 5 reasons cloud gaming isn't doing it for me -- yet. https://www.cnet.com/tech/computing/5-reasons-cloud-gamingisnt-doing-it-for-me-yet/.Google Scholar
- Bandwidth, data usage, and stream quality. https://support.google.com/stadia/answer/9607891?hl=en.Google Scholar
- The Chromium Project. https://www.chromium.org/.Google Scholar
- DECAF: Dissecting Cloud Gaming Performace. https://github.com/decafCG/decaf.Google Scholar
- Geforce Now System Requirements. https://www.nvidia.com/en-us/geforce-now/system-reqs/.Google Scholar
- Iperf. https://iperf.fr/.Google Scholar
- Maxmind. https://www.maxmind.com/en/home.Google Scholar
- Python Imaging Library. https://pypi.org/project/Pillow/.Google Scholar
- Python-tesseract: an optical character recognition tool for python. https://pypi.org/project/pytesseract/.Google Scholar
- Python win32con Module. https://www.programcreek.com/python/index/475/win32con.Google Scholar
- pywin32. https://pypi.org/project/pywin32/.Google Scholar
- What Is Amazon Luna? https://www.amazon.com/gp/help/customer/display.html?nodeId=G8WBF7CLZX7W345R. Proc. ACM Meas. Anal. Comput. Syst., Vol. 5, No. 3, Article 31. Publication date: December 2021. 31:24 Hassan Iqbal, Ayesha Khalid, & Muhammad ShahzadGoogle Scholar
- Adnan Ahmed, Zubair Shafiq, and Amir Khakpour. QoE Analysis of a Large-Scale Live Video Streaming Event (ACM SIGMETRICS '16).Google Scholar
- Zahaib Akhtar, Yun Seong Nam, Ramesh Govindan, Sanjay Rao, Jessica Chen, Ethan Katz-Bassett, Bruno Ribeiro, Jibin Zhan, and Hui Zhang. Oboe: Auto-tuning Video ABR Algorithms to Network Conditions (ACM SIGCOMM '18).Google Scholar
- Todd Arnold, Jia He, Weifan Jiang, Matt Calder, Italo Cunha, Vasileios Giotsas, and Ethan Katz-Bassett. Cloud Provider Connectivity in the Flat Internet (ACM IMC '20).Google Scholar
- Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, and Hui Zhang. Developing a Predictive Model of Quality of Experience for Internet Video (ACM SIGCOMM '13).Google Scholar
- Hafiz Mohsin Bashir, Abdullah Bin Faisal, M Asim Jamshed, Peter Vondras, Ali Musa Iftikhar, Ihsan Ayyub Qazi, and Fahad R. Dogar. Reducing Tail Latency Using Duplication: A Multi-Layered Approach (ACM CoNEXT '19).Google Scholar
- Jan A Bergstra and CA Middelburg. ITU-T Recommendation G. 107: The E-Model, a computational model for use in transmission planning. (2003).Google Scholar
- Wei Cai, Ryan Shea, Chun-Ying Huang, Kuan-Ta Chen, Jiangchuan Liu, and Cheng-Hsin Hsu. A Survey on Cloud Gaming: Future of Computer Games. IEEE Access (2016).Google Scholar
- Eduardo F Camacho and Carlos Bordons Alba. Model predictive control. Springer science & business media.Google Scholar
- Marc Carrascosa and Boris Bellalta. Cloud-gaming:Analysis of Google Stadia traffic. arXiv:2009.09786 [cs.NI]Google Scholar
- Kuan-Ta Chen, Yu-Chun Chang, Po-Han Tseng, Chun-Ying Huang, and Chin-Laung Lei. Measuring the Latency of Cloud Gaming Systems (ACM MM '11).Google Scholar
- Kuan-Ta Chen, Polly Huang, and Chin-Laung Lei. How Sensitive Are Online Gamers to Network Quality? Commun. ACM (2006).Google Scholar
- Yi-Ching Chiu, Brandon Schlinker, Abhishek Balaji Radhakrishnan, Ethan Katz-Bassett, and Ramesh Govindan. Are We One Hop Away from a Better Internet? (ACM IMC '15).Google Scholar
- Jonathan Deber, Ricardo Jota, Clifton Forlines, and Daniel Wigdor. How Much Faster is Fast Enough? User Perception of Latency & Latency Improvements in Direct and Indirect Touch (ACM SIGCHI '15).Google Scholar
- Florin Dobrian, Vyas Sekar, Asad Awan, Ion Stoica, Dilip Joseph, Aditya Ganjam, Jibin Zhan, and Hui Zhang. Understanding the Impact of Video Quality on User Engagement (ACM SIGCOMM '11).Google Scholar
- Andrea Di Domenico, Gianluca Perna, Martino Trevisan, Luca Vassio, and Danilo Giordano. A network analysis on cloud gaming: Stadia, GeForce Now and PSNow. arXiv:2012.06774 [cs.NI]Google Scholar
- Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 aaai fall symposium series.Google Scholar
- Mohamed Hegazy, Khaled Diab, Mehdi Saeedi, Boris Ivanovic, Ihab Amer, Yang Liu, Gabor Sines, and Mohamed Hefeeda. Content-Aware Video Encoding for Cloud Gaming.Google Scholar
- Te-Yuan Huang, Nikhil Handigol, Brandon Heller, Nick McKeown, and Ramesh Johari. Confused, Timid, and Unstable: Picking a Video Streaming Rate is Hard (ACM IMC '12).Google Scholar
- Niels Justesen, Philip Bontrager, Julian Togelius, and Sebastian Risi. Deep learning for video game playing. IEEE Transactions on Games 12 (2019).Google Scholar
- Teemu Kämäräinen, Matti Siekkinen, Antti Ylä-Jääski, Wenxiao Zhang, and Pan Hui. A Measurement Study on Achieving Imperceptible Latency in Mobile Cloud Gaming. Association for Computing Machinery.Google Scholar
- Guillaume Lample and Devendra Singh Chaplot. Playing FPS games with deep reinforcement learning. In Thirty-First AAAI Conference on Artificial Intelligence.Google Scholar
- Xing Liu, Bo Han, Feng Qian, and Matteo Varvello. LIME: Understanding Commercial 360° Live Video Streaming Services (MMSys '19).Google Scholar
- Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh. Neural Adaptive Video Streaming with Pensieve (ACM SIGCOMM '17).Google Scholar
- Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature 518, 7540 (2015), 529--533.Google Scholar
- Donald A. Norman. The design of everyday things. Basic Books.Google Scholar
Digital Library
- Jesse Schell. The art of game design: a book of lenses. CRC Press, Taylor & Francis Group.Google Scholar
Digital Library
- Steven Schmidt, Saman Zadtootaghaj, Saeed Shafiee Sabet, and Sebastian Möller. Modeling and Understanding the Quality of Experience of Online Mobile Gaming Services (IEEE QoMEX '21).Google Scholar
- C.E. Shannon. Communication in the Presence of Noise. Proceedings of the IRE (1949).Google Scholar
Cross Ref
- Iraj Sodagar. The MPEG-DASH Standard for Multimedia Streaming Over the Internet. IEEE MultiMedia (2011).Google Scholar
Digital Library
- German Sviridov, Cedric Beliard, Andrea Bianco, Paolo Giaccone, and Dario Rossi. Removing human players from the loop: AI-assisted assessment of Gaming QoE (IEEE INFOCOM WKSHPS '20).Google Scholar
- Bolun Wang, Xinyi Zhang, Gang Wang, Haitao Zheng, and Ben Y. Zhao. Anatomy of a Personalized Livestreaming System (ACM IMC '16). Proc. ACM Meas. Anal. Comput. Syst., Vol. 5, No. 3, Article 31. Publication date: December 2021. Dissecting Cloud Gaming Performance with DECAF 31:25Google Scholar
- S. Wang and S. Dey. Modeling and Characterizing User Experience in a Cloud Server Based Mobile Gaming Approach (IEEE GLOBECOM '09).Google Scholar
- Marek Wydmuch, Micha Kempka, and Wojciech Jakowski. Vizdoom competitions: Playing doom from pixels. IEEE Transactions on Games 11, 3 (2018), 248--259.Google Scholar
- Zheng Xue, Di Wu, Jian He, Xiaojun Hei, and Yong Liu. Playing High-End Video Games in the Cloud: A Measurement Study. IEEE Transactions on Circuits and Systems for Video Technology (2015).Google Scholar
Digital Library
- Jun Yi, Shiqing Luo, and Zhisheng Yan. A Measurement Study of YouTube 360° Live Video Streaming (ACM NOSSDAV '19).Google Scholar
- Saman Zadtootaghaj, Steven Schmidt, Saeed Shafiee Sabet, Sebastian Möller, and Carsten Griwodz. Quality estimation models for gaming video streaming services using perceptual video quality dimensions (ACM MM '20).Google Scholar
- Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Relational deep reinforcement learning. arXiv preprint arXiv:1806.01830 (2018).Google Scholar
- Timothy Zhu, Alexey Tumanov, Michael A. Kozuch, Mor Harchol-Balter, and Gregory R. Ganger. PriorityMeister: Tail Latency QoS for Shared Networked Storage (ACM SOCC '14).Google Scholar
Index Terms
Dissecting Cloud Gaming Performance with DECAF
Recommendations
Dissecting Cloud Gaming Performance with DECAF
SIGMETRICS/PERFORMANCE '22: Abstract Proceedings of the 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer SystemsCloud gaming platforms have witnessed tremendous growth over the past two years, with a number of large Internet companies including Amazon, Facebook, Google, Microsoft, and Nvidia publicly launching their own platforms. However, there is an absence of ...
Dissecting Cloud Gaming Performance with DECAF
SIGMETRICS '22Cloud gaming platforms have witnessed tremendous growth over the past two years, with a number of large Internet companies including Amazon, Facebook, Google, Microsoft, and Nvidia publicly launching their own platforms. However, there is an absence of ...
Dissecting the protocol and network traffic of the OnLive cloud gaming platform
Cloud gaming is a new paradigm that is envisaged to play a pivotal role in the video game industry in forthcoming years. Cloud gaming, or gaming on demand, is a type of online gaming that allows on-demand streaming of game content onto non-specialised ...






Comments