skip to main content
research-article

Dissecting the Performance of VR Video Streaming through the VR-EXP Experimentation Platform

Published:16 December 2019Publication History
Skip Abstract Section

Abstract

To cope with the massive bandwidth demands of Virtual Reality (VR) video streaming, both the scientific community and the industry have been proposing optimization techniques such as viewport-aware streaming and tile-based adaptive bitrate heuristics. As most of the VR video traffic is expected to be delivered through mobile networks, a major problem arises: both the network performance and VR video optimization techniques have the potential to influence the video playout performance and the Quality of Experience (QoE). However, the interplay between them is neither trivial nor has it been properly investigated. To bridge this gap, in this article, we introduce VR-EXP, an open-source platform for carrying out VR video streaming performance evaluation. Furthermore, we consolidate a set of relevant VR video streaming techniques and evaluate them under variable network conditions, contributing to an in-depth understanding of what to expect when different combinations are employed. To the best of our knowledge, this is the first work to propose a systematic approach, accompanied by a software toolkit, which allows one to compare different optimization techniques under the same circumstances. Extensive evaluations carried out using realistic datasets demonstrate that VR-EXP is instrumental in providing valuable insights regarding the interplay between network performance and VR video streaming optimization techniques.

References

  1. Zahaib Akhtar, Yun Seong Nam, Ramesh Govindan, Sanjay Rao, Jessica Chen, Ethan Katz-Bassett, Bruno Ribeiro, Jibin Zhan, and Hui Zhang. 2018. Oboe: Auto-tuning video ABR algorithms to network conditions. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (SIGCOMM’18). ACM, New York, NY, 44--58. DOI:https://doi.org/10.1145/3230543.3230558Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Mathias Almquist, Viktor Almquist, Vengatanathan Krishnamoorthi, Niklas Carlsson, and Derek Eager. 2018. The prefetch aggressiveness tradeoff in 360 video streaming. In Proceedings of the 9th ACM Multimedia Systems Conference (MMSys’18). ACM, New York, NY, 258--269. DOI:https://doi.org/10.1145/3204949.3204970Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Jill Boyce, Elena Alshina, Adeel Abbas, and Yan Ye. 2017. JVET common test conditions and evaluation procedures for 360 video. Joint Video Exploration Team of ITU-T SG 16 (2017). ITU - International Telecommunication Union. Retrieved on 21 June, 2010 from http://phenix.it sudparis.eu/jvet/doc_end_user/documents/16_Geneva/wg11/JVET-P0006-v1.zip.Google ScholarGoogle Scholar
  4. Zhenzhong Chen, Yiming Li, and Yingxue Zhang. 2018. Recent advances in omnidirectional video coding for virtual reality: Projection and evaluation. Sig. Proc. 146 (2018), 66--78. DOI:https://doi.org/10.1016/j.sigpro.2018.01.004Google ScholarGoogle ScholarCross RefCross Ref
  5. Cisco. 2019. Cisco Visual Networking Index: Forecast and Trends, 2017--2022. Technical Report. Cisco Systems.Google ScholarGoogle Scholar
  6. X. Corbillon, G. Simon, A. Devlic, and J. Chakareski. 2017. Viewport-adaptive navigable 360-degree video delivery. In Proceedings of the IEEE International Conference on Communications (ICC’17). 1--7. DOI:https://doi.org/10.1109/ICC.2017.7996611Google ScholarGoogle Scholar
  7. R. I. T. da Costa Filho, W. Lautenschlager, N. Kagami, V. Roesler, and L. P. Gaspary. 2016. Network fortune cookie: Using network measurements to predict video streaming performance and QoE. In Proceedings of the IEEE Global Communications Conference (GLOBECOM’16). 1--6. DOI:https://doi.org/10.1109/GLOCOM.2016.7842022Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Roberto Irajá Tavares da Costa Filho, Marcelo Caggiani Luizelli, Maria Torres Vega, Jeroen van der Hooft, Stefano Petrangeli, Tim Wauters, Filip De Turck, and Luciano Paschoal Gaspary. 2018. Predicting the performance of virtual reality video streaming in mobile networks. In Proceedings of the 9th ACM Multimedia Systems Conference (MMSys’18). ACM, New York, NY, 270--283. DOI:https://doi.org/10.1145/3204949.3204966Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Giorgos Dimopoulos, Ilias Leontiadis, Pere Barlet-Ros, and Konstantina Papagiannaki. 2016. Measuring video QoE from encrypted traffic. In Proceedings of the Internet Measurement Conference (IMC’16). ACM, New York, NY, 513--526. DOI:https://doi.org/10.1145/2987443.2987459Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Glederson Lessa dos Santos, Vinicius Tavares Guimaraes, Jorge Guedes Silveira, Alexandre T. Vieira, Jose Augusto de Oliveira Neto, R. I. T. da Costa, and Ricardo Balbinot. 2007. UAMA: A unified architecture for active measurements in IP networks; end-to-end objetive quality indicators. In Proceedings of the 10th IFIP/IEEE International Symposium on Integrated Network Management (IM’07). 246--253.Google ScholarGoogle ScholarCross RefCross Ref
  11. Ching-Ling Fan, Jean Lee, Wen-Chih Lo, Chun-Ying Huang, Kuan-Ta Chen, and Cheng-Hsin Hsu. 2017. Fixation prediction for 360 video streaming in head-mounted virtual reality. In Proceedings of the 27th Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV’17). ACM, New York, NY, 67--72. DOI:https://doi.org/10.1145/3083165.3083180Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Mario Graf, Christian Timmerer, and Christopher Mueller. 2017. Towards bandwidth-efficient adaptive streaming of omnidirectional video over HTTP: Design, implementation, and evaluation. In Proceedings of the 8th ACM Conference on Multimedia Systems Conference (MMSys’17). ACM, New York, NY, 261--271. DOI:https://doi.org/10.1145/3083187.3084016Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Jian He, Mubashir Adnan Qureshi, Lili Qiu, Jin Li, Feng Li, and Lei Han. 2018. Favor: Fine-grained video rate adaptation. In Proceedings of the 9th ACM Multimedia Systems Conference (MMSys’18). ACM, New York, NY, 64--75. DOI:https://doi.org/10.1145/3204949.3204957Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. M. Hosseini and V. Swaminathan. 2016. Adaptive 360 VR video streaming: Divide and conquer. In Proceedings of the IEEE International Symposium on Multimedia (ISM’16). 107--110. DOI:https://doi.org/10.1109/ISM.2016.0028Google ScholarGoogle Scholar
  15. Xueshi Hou, Sujit Dey, Jianzhong Zhang, and Madhukar Budagavi. 2018. Predictive view generation to enable mobile 360-degree and VR experiences. In Proceedings of the Morning Workshop on Virtual Reality and Augmented Reality Network (VR/AR Network’18). ACM, New York, NY, 20--26. DOI:https://doi.org/10.1145/3229625.3229629Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. H. Hristova, X. Corbillon, G. Simon, V. Swaminathan, and A. Devlic. 2018. Heterogeneous spatial quality for omnidirectional video. In Proceedings of the IEEE 20th International Workshop on Multimedia Signal Processing (MMSP’18). 1--6. DOI:https://doi.org/10.1109/MMSP.2018.8547114Google ScholarGoogle Scholar
  17. E. Jeong, D. You, C. Hyun, B. Seo, N. Kim, D. H. Kim, and Y. H. Lee. 2018. Viewport prediction method of 360 VR video using sound localization information. In Proceedings of the 10th International Conference on Ubiquitous and Future Networks (ICUFN’18). 679--681. DOI:https://doi.org/10.1109/ICUFN.2018.8436981Google ScholarGoogle Scholar
  18. Ron Kohavi et al. 1995. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’95), Vol. 14. 1137--1145.Google ScholarGoogle Scholar
  19. L. Ma, Y. Xu, J. Sun, W. Huang, S. Xie, Y. Li, and N. Liu. 2018. Buffer control in VR video transmission over MMT system. In Proceedings of the IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB’18). 1--5. DOI:https://doi.org/10.1109/BMSB.2018.8436817Google ScholarGoogle Scholar
  20. Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh. 2017. Neural adaptive video streaming with Pensieve. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (SIGCOMM’17). ACM, New York, NY, 197--210. DOI:https://doi.org/10.1145/3098822.3098843Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. A. Morton. 2016. RFC 7799 - Active and Passive Metrics and Methods (with Hybrid Types In-Between). Technical Report. IETF.Google ScholarGoogle Scholar
  22. T. C. Nguyen and J. Yun. 2018. Predictive tile selection for 360-degree VR video streaming in bandwidth-limited networks. IEEE Commun. Lett. 22, 9 (Sept. 2018), 1858--1861. DOI:https://doi.org/10.1109/LCOMM.2018.2848915Google ScholarGoogle ScholarCross RefCross Ref
  23. Stefano Petrangeli, Jeroen Famaey, Maxim Claeys, Steven Latré, and Filip De Turck. 2015. QoE-driven rate adaptation heuristic for fair adaptive video streaming. ACM Trans. Multimedia Comput. Commun. Appl. 12, 2, Article 28 (Oct. 2015), 24 pages. DOI:https://doi.org/10.1145/2818361Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. S. Petrangeli, G. Simon, and V. Swaminathan. 2018. Trajectory-based viewport prediction for 360-degree virtual reality videos. In Proceedings of the IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR’18). 157--160. DOI:https://doi.org/10.1109/AIVR.2018.00033Google ScholarGoogle Scholar
  25. Stefano Petrangeli, Viswanathan Swaminathan, Mohammad Hosseini, and Filip De Turck. 2017. An HTTP/2-Based adaptive streaming framework for 360 virtual reality videos. In Proceedings of the ACM on Multimedia Conference (MM’17). ACM, New York, NY, 306--314. DOI:https://doi.org/10.1145/3123266.3123453Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Feng Qian, Lusheng Ji, Bo Han, and Vijay Gopalakrishnan. 2016. Optimizing 360 video delivery over cellular networks. In Proceedings of the 5th Workshop on All Things Cellular: Operations, Applications and Challenges (ATC’16). ACM, New York, NY, 1--6. DOI:https://doi.org/10.1145/2980055.2980056Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Iraj Sodagar. 2011. The MPEG-DASH standard for multimedia streaming over the internet. IEEE Multimedia 18, 4 (2011), 62--67.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Kevin Spiteri, Ramesh Sitaraman, and Daniel Sparacio. 2018. From theory to practice: Improving bitrate adaptation in the DASH reference player. In Proceedings of the 9th ACM Multimedia Systems Conference (MMSys’18). ACM, New York, NY, 123--137. DOI:https://doi.org/10.1145/3204949.3204953Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. K. Spiteri, R. Urgaonkar, and R. K. Sitaraman. 2016. BOLA: Near-optimal bitrate adaptation for online videos. In Proceedings of the 35th IEEE International Conference on Computer Communications (INFOCOM’16). 1--9. DOI:https://doi.org/10.1109/INFOCOM.2016.7524428Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. K. Stangherlin, R. C. Filho, W. Lautenschläger, V. Guadagnin, L. Balbinot, R. Balbinot, and V. Roesler. 2011. One-way delay measurement in wired and wireless mobile full-mesh networks. In Proceedings of the IEEE Wireless Communications and Networking Conference. 1044--1049. DOI:https://doi.org/10.1109/WCNC.2011.5779279Google ScholarGoogle Scholar
  31. TELCO. 2018. Mobile Operators Market Share in Brazil. Retrieved from: http://www.teleco.com.br/en/en_mshare.asp.Google ScholarGoogle Scholar
  32. J. van der Hooft, S. Petrangeli, T. Wauters, R. Huysegems, P. R. Alface, T. Bostoen, and F. De Turck. 2016. HTTP/2-Based adaptive streaming of HEVC video over 4G/LTE networks. IEEE Commun. Lett. 20, 11 (2016), 2177--2180.Google ScholarGoogle ScholarCross RefCross Ref
  33. M. Viitanen, A. Koivula, A. Lemmetti, J. Vanne, and T. D. Hämäläinen. 2015. Kvazaar HEVC encoder for efficient intra coding. In Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS’15). 1662--1665. DOI:https://doi.org/10.1109/ISCAS.2015.7168970Google ScholarGoogle Scholar
  34. Chenglei Wu, Zhihao Tan, Zhi Wang, and Shiqiang Yang. 2017. A dataset for exploring user behaviors in VR spherical video streaming. In Proceedings of the 8th ACM on Multimedia Systems Conference (MMSys’17). ACM, New York, NY, 193--198. DOI:https://doi.org/10.1145/3083187.3083210Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Xiaoqi Yin, Abhishek Jindal, Vyas Sekar, and Bruno Sinopoli. 2015. A control-theoretic approach for dynamic adaptive video streaming over HTTP. In Proceedings of the ACM Conference on Special Interest Group on Data Communication (SIGCOMM’15). ACM, New York, NY, 325--338. DOI:https://doi.org/10.1145/2785956.2787486Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Xiaoqi Yin, Abhishek Jindal, Vyas Sekar, and Bruno Sinopoli. 2015. A control-theoretic approach for dynamic adaptive video streaming over HTTP. SIGCOMM Comput. Commun. Rev. 45, 4 (Aug. 2015), 325--338. DOI:https://doi.org/10.1145/2829988.2787486Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Chao Zhou, Zhenhua Li, Joe Osgood, and Yao Liu. 2018. On the effectiveness of offset projections for 360-degree video streaming. ACM Trans. Multimedia Comput. Commun. Appl. 14, 3s, Article 62 (June 2018), 24 pages. DOI:https://doi.org/10.1145/3209660Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Dissecting the Performance of VR Video Streaming through the VR-EXP Experimentation Platform

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          • Published in

            cover image ACM Transactions on Multimedia Computing, Communications, and Applications
            ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 15, Issue 4
            November 2019
            322 pages
            ISSN:1551-6857
            EISSN:1551-6865
            DOI:10.1145/3376119
            Issue’s Table of Contents

            Copyright © 2019 ACM

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            • Published: 16 December 2019
            • Accepted: 1 September 2019
            • Revised: 1 July 2019
            • Received: 1 February 2017
            Published in tomm Volume 15, Issue 4

            Permissions

            Request permissions about this article.

            Request Permissions

            Check for updates

            Qualifiers

            • research-article
            • Research
            • Refereed

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format .

          View HTML Format
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!