skip to main content
research-article

Tile-based Adaptive Streaming for Virtual Reality Video

Authors Info & Claims
Published:16 December 2019Publication History
Skip Abstract Section

Abstract

The increasing popularity of head-mounted devices and 360° video cameras allows content providers to provide virtual reality (VR) video streaming over the Internet, using a two-dimensional representation of the immersive content combined with traditional HTTP adaptive streaming (HAS) techniques. However, since only a limited part of the video (i.e., the viewport) is watched by the user, the available bandwidth is not optimally used. Recent studies have shown the benefits of adaptive tile-based video streaming; rather than sending the whole 360° video at once, the video is cut into temporal segments and spatial tiles, each of which can be requested at a different quality level. This allows prioritization of viewable video content and thus results in an increased bandwidth utilization. Given the early stages of research, there are still a number of open challenges to unlock the full potential of adaptive tile-based VR streaming. The aim of this work is to provide an answer to several of these open research questions. Among others, we propose two tile-based rate adaptation heuristics for equirectangular VR video, which use the great-circle distance between the viewport center and the center of each of the tiles to decide upon the most appropriate quality representation. We also introduce a feedback loop in the quality decision process, which allows the client to revise prior decisions based on more recent information on the viewport location. Furthermore, we investigate the benefits of parallel TCP connections and the use of HTTP/2 as an application layer optimization. Through an extensive evaluation, we show that the proposed optimizations result in a significant improvement in terms of video quality (more than twice the time spent on the highest quality layer), compared to non-tiled HAS solutions.

References

  1. M. Seufert, S. Egger, M. Slanina, T. Zinner, T. Hoßfeld, and P. Tran-Gia. 2015. A survey on quality of experience of HTTP adaptive streaming. IEEE Commun. Surveys Tutor. 17, 1 (2015), 469--492.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. R. Mok, E. Chan, and R. Chang. 2011. Measuring the quality of experience of HTTP video streaming. In Proceedings of the IFIP/IEEE International Symposium on Integrated Network Management. 485--492.Google ScholarGoogle Scholar
  3. ISO/IEC 23009-1 2014. Information Technology—Dynamic Adaptive Streaming over HTTP (DASH) - Part 1: Media Presentation Description and Segment Formats. Technical Report. International Organization for Standardization.Google ScholarGoogle Scholar
  4. OpenSignal. 2018. State of Mobile Networks: USA. Retrieved from https://opensignal.com/reports/2018/01/usa/state-of-the-mobile-network/.Google ScholarGoogle Scholar
  5. OpenSignal. 2018. State of Mobile Networks: Belgium. Retrieved from https://opensignal.com/reports/2018/03/belgium/state-of-the-mobile-network/.Google ScholarGoogle Scholar
  6. S. Petrangeli, V. Swaminathan, M. Hosseini, and F. De Turck. 2017. An HTTP/2-based adaptive streaming framework for 360-degree virtual reality videos. In Proceedings of the ACM Multimedia Conference. 306--314.Google ScholarGoogle Scholar
  7. J. van der Hooft, M. Torres Vega, S. Petrangeli, T. Wauters, and F. De Turck. 2019. Optimizing adaptive tile-based virtual reality video streaming. In Proceedings of the IFIP/IEEE International Symposium on Integrated Network Management.Google ScholarGoogle Scholar
  8. M. Yu, H. Lakshman, and B. Girod. 2015. A framework to evaluate omnidirectional video coding schemes. In Proceedings of the IEEE International Symposium on Mixed and Augmented Reality. 31--36.Google ScholarGoogle Scholar
  9. R. Skupin, Y. Sanchez, D. Podborski, C. Hellge, and T. Schierl. 2017. HEVC tile-based streaming to head-mounted displays. In Proceedings of the IEEE Annual Consumer Communications Networking Conference. 613--615.Google ScholarGoogle Scholar
  10. M. Budagavi, J. Furton, G. Jin, A. Saxena, J. Wilkinson, and A. Dickerson. 2015. 360 degrees video coding using region adaptive smoothing. In Proceedings of the IEEE International Conference on Image Processing. 750--754.Google ScholarGoogle Scholar
  11. M. Hosseini and V. Swaminathan. 2016. Adaptive 360 VR video streaming: Divide and conquer! In Proceedings of the IEEE International Symposium on Multimedia. 107--110.Google ScholarGoogle Scholar
  12. Team Pixvana. 2016. An Intro to FOVAS: Field of View Adaptive Streaming for Virtual Reality. Retrieved from https://pixvana.com/intro-to-field-of-view-adaptive-streaming-for-vr/.Google ScholarGoogle Scholar
  13. E. Kuzyakov and D. Pio. 2016. Next-Generation Video Encoding Techniques for 360 Video and VR. Retrieved from https://code.fb.com/virtual-reality/next-generation-video-encoding-techniques-for-360-video-and-vr/.Google ScholarGoogle Scholar
  14. A. Zare, A. Aminlou, and M. M. Hunnuksela. 2017. Virtual reality content streaming: Viewport-dependent projection and tile-based techniques. In Proceedings of the IEEE International Conference on Image Processing. 1432--1436.Google ScholarGoogle Scholar
  15. JCTVC-AC0038 2017. Temporal MCTS Coding Constraints Implementation. Technical Report. Joint Collaborative Team on Video Coding.Google ScholarGoogle Scholar
  16. J. Son, D. Jang, and E. Ryu. 2018. Implementing motion-constrained tile and viewport extraction for VR streaming. In Proceedings of the 28th ACM SIGMM Workshop on Network and Operating Systems Support for Digital Audio and Video. ACM, 61--66.Google ScholarGoogle Scholar
  17. A. Taghavi Nasrabadi, A. Mahzari, J. D. Beshay, and R. Prakash. 2017. Adaptive 360-degree video streaming using layered video coding. In Proceedings of the IEEE Virtual Reality Conference. 347--348.Google ScholarGoogle Scholar
  18. F. Qian, L. Ji, B. Han, and V. Gopalakrishnan. 2016. Optimizing 360 video delivery over cellular networks. In Proceedings of the Workshop on All Things Cellular: Operations, Applications and Challenges. 1--6.Google ScholarGoogle Scholar
  19. Z. Xu, X. Zhang, K. Zhang, and Z. Guo. 2018. Probabilistic viewport adaptive streaming for 360-degree videos. In Proceedings of the IEEE International Symposium on Circuits and Systems. 1--5.Google ScholarGoogle Scholar
  20. C. Fan, J. Lee, W. Lo, C. Huang, K. Chen, and C. Hsu. 2017. Fixation prediction for 360-degree; video streaming in head-mounted virtual reality. In Proceedings of the Workshop on Network and Operating Systems Support for Digital Audio and Video. 67--72.Google ScholarGoogle Scholar
  21. V. Sitzmann, A. Serrano, A. Pavel, M. Agrawala, D. Gutierrez, B. Masia, and G. Wetzstein. 2018. Saliency in VR: How do people explore virtual environments? IEEE Trans. Visual. Comput. Graph. 24, 4 (2018), 1633--1642.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. H. Hu, Y. Lin, M. Liu, H. Cheng, Y. Chang, and M. Sun. 2017. Deep 360 pilot: Learning a deep agent for piloting through 360-degree sports video. Comput. Res. Repos. abs/1705.01759 (2017).Google ScholarGoogle Scholar
  23. A. Ghosh, V. Aggarwal, and F. Qian. 2017. A rate adaptation algorithm for tile-vased 360-degree video streaming. Comput. Res. Repos. abs/1704.08215 (2017).Google ScholarGoogle Scholar
  24. A. Zare, A. Aminlou, M. M. Hannuksela, and M. Gabbouj. 2016. HEVC-compliant tile-based streaming of panoramic video for virtual reality applications. In Proceedings of the ACM International Conference on Multimedia. ACM, 601--605.Google ScholarGoogle Scholar
  25. J. Le Feuvre and C. Concolato. 2016. Tiled-based adaptive streaming using MPEG-DASH. In Proceedings of the International Conference on Multimedia Systems. 41:1--41:3.Google ScholarGoogle Scholar
  26. L. D’Acunto, J. van den Berg, E. Thomas, and O. Niamut. 2016. Using MPEG DASH SRD for zoomable and navigable video. In Proceedings of the International Conference on Multimedia Systems. 34:1--34:4.Google ScholarGoogle Scholar
  27. R. Fielding and J. Reschke. 2014. Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing. RFC 7230. RFC Editor. Retrieved from https://www.rfc-editor.org/rfc/rfc7230.txt.Google ScholarGoogle Scholar
  28. M. Belshe, G. Peon, and M. Thomson. 2015. Hypertext Transfer Protocol Version 2 (HTTP/2). RFC 7540. RFC Editor. Retrieved from https://www.rfc-editor.org/rfc/rfc7540.txt.Google ScholarGoogle Scholar
  29. W3Techs. 2019. Usage of HTTP/2 for Websites. Retrieved from https://w3techs.com/technologies/details/ce-http2/all/all.Google ScholarGoogle Scholar
  30. S. Wei and V. Swaminathan. 2014. Low latency live video streaming over HTTP 2.0. In Proceedings of the Network and Operating System Support on Digital Audio and Video Workshop. 37:37--37:42.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. L. Xie, Z. Xu, Y. Ban, X. Zhang, and Z. Guo. 2017. 360ProbDASH: Improving QoE of 360 video streaming using tile-based HTTP adaptive streaming. In Proceedings of the ACM Multimedia Systems Conference. 315--323.Google ScholarGoogle Scholar
  32. H. Ahmadi, O. Eltobgy, and M. Hefeeda. 2017. Adaptive multicast streaming of virtual reality content to mobile users. In Proceedings of the Thematic Workshops of ACM Multimedia. 170--178.Google ScholarGoogle Scholar
  33. R. I. T. da Costa Filho, M. C. Luizelli, M. Torres Vega, J. van der Hooft, S. Petrangeli, T. Wauters, F. De Turck, and L. P. Gaspary. 2018. Predicting the performance of virtual reality video streaming in mobile networks. In Proceedings of the ACM Multimedia Systems Conference. 270--283.Google ScholarGoogle Scholar
  34. Y. Rai, P. Le Callet, and P. Guillotel. 2017. Which saliency weighting for omni directional image quality assessment? In Proceedings of the International Conference on Quality of Multimedia Experience. 1--6.Google ScholarGoogle Scholar
  35. J. van der Hooft, M. Torres Vega, S. Petrangeli, T. Wauters, and F. De Turck. 2019. Quality assessment for adaptive virtual reality video streaming: A probabilistic approach on the user’s gaze. In Proceedings of the International Workshop on Quality of Experience Management.Google ScholarGoogle Scholar
  36. P. Juluri, V. Tamarapalli, and D. Medhi. 2015. SARA: Segment-aware rate adaptation algorithm for dynamic adaptive streaming over HTTP. In Proceedings of the IEEE International Conference on Communication Workshop. 1765--1770.Google ScholarGoogle Scholar
  37. C. Wu, Z. Tan, Z. Wang, and S. Yang. 2017. A dataset for exploring user behaviors in VR spherical video streaming. In Proceedings of the ACM Multimedia Systems Conference. 193--198.Google ScholarGoogle Scholar
  38. H. Kellerer, U. Pferschy, and D. Pisinger. 2004. The Multiple-Choice Knapsack Problem. Springer, Berlin, 317--347.Google ScholarGoogle Scholar
  39. J. van der Hooft, S. Petrangeli, T. Wauters, R. Huysegems, P. Rondao Alface, T. Bostoen, and F. De Turck. 2016. HTTP/2-based adaptive streaming of HEVC video over 4G/LTE networks. IEEE Commun. Lett. 20, 11 (2016), 2177--2180.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Tile-based Adaptive Streaming for Virtual Reality Video

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Multimedia Computing, Communications, and Applications
          ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 15, Issue 4
          November 2019
          322 pages
          ISSN:1551-6857
          EISSN:1551-6865
          DOI:10.1145/3376119
          Issue’s Table of Contents

          Copyright © 2019 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 16 December 2019
          • Revised: 1 September 2019
          • Accepted: 1 September 2019
          • Received: 1 January 2019
          Published in tomm Volume 15, Issue 4

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Research
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format .

        View HTML Format
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!