skip to main content
research-article

Millimeter Wave and Free-space-optics for Future Dual-connectivity 6DOF Mobile Multi-user VR Streaming

Published:06 February 2023Publication History
Skip Abstract Section

Abstract

Dual-connectivity streaming is a key enabler of next-generation six Degrees Of Freedom (6DOF) Virtual Reality (VR) scene immersion. Indeed, using conventional sub-6 GHz WiFi only allows to reliably stream a low-quality baseline representation of the VR content, while emerging high-frequency communication technologies allow to stream in parallel a high-quality user viewport-specific enhancement representation that synergistically integrates with the baseline representation to deliver high-quality VR immersion. We investigate holistically as part of an entire future VR streaming system two such candidate emerging technologies, Free Space Optics (FSO) and millimeter-Wave (mmWave), that benefit from a large available spectrum to deliver unprecedented data rates. We analytically characterize the key components of the envisioned dual-connectivity 6DOF VR streaming system that integrates in addition edge computing and scalable 360° video tiling, and we formulate an optimization problem to maximize the immersion fidelity delivered by the system, given the WiFi and mmWave/FSO link rates, and the computing capabilities of the edge server and the users’ VR headsets. This optimization problem is mixed integer programming of high complexity and we formulate a geometric programming framework to compute the optimal solution at low complexity. We carry out simulation experiments to assess the performance of the proposed system using actual 6DOF navigation traces from multiple mobile VR users that we collected. Our results demonstrate that our system considerably advances the traditional state of the art and enables streaming of 8K-120 frames-per-second (fps) 6DOF content at high fidelity.

REFERENCES

  1. [1] Chakareski Jacob. 2017. Drone networks for virtual human teleportation. In Proc. ACM Workshop on Micro Aerial Vehicle Networks, Systems, and Applications. 2126.Google ScholarGoogle Scholar
  2. [2] Apostolopoulos J. G., Chou P. A., Culbertson Bruce, Kalker T., Trott M. D., and Wee S.. 2012. The road to immersive communication. Proc. IEEE 100, 4 (April 2012), 974990.Google ScholarGoogle ScholarCross RefCross Ref
  3. [3] Chakareski Jacob. 2019. UAV-IoT for next generation virtual reality. IEEE Transactions on Image Processing 28, 12 (2019), 59775990.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Champel T. Stockhammer, Fautier T., Thomas E., and Koenen R.. 2016. Quality requirements for VR. In Proc. 116th MPEG Meeting of ISO/IEC JTC1/SC29/WG11 MPEG, vol. 116, p. m39532.Google ScholarGoogle Scholar
  5. [5] Dimitrov Svilen and Haas Harald. 2015. Principles of LED Light Communications: Towards Networked Li-Fi. Cambridge University Press.Google ScholarGoogle ScholarCross RefCross Ref
  6. [6] Strinati E. Calvanese, Barbarossa S., Gonzalez-Jimenez J. L., Dimitri Ktenas, Nicolas Cassiau, Luc Maret, and Cedric Dehos. 2019. 6G: The next frontier: From holographic messaging to artificial intelligence using subterahertz and visible light communication. IEEE Vehicular Technology Magazine 14, 3 (2019), 4250. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  7. [7] Rahman Md Shaifur, Zheng Kai, and Gupta Himanshu. 2018. FSO-VR: Steerable free space optics link for virtual reality headsets. In Proc. ACM Workshop on Wearable Systems and Applications. 1115.Google ScholarGoogle Scholar
  8. [8] Beysens J., Wang Q., Galisteo A., Giustiniano D., and Pollin S.. 2020. A cell-free networking system with visible light. IEEE/ACM Transactions on Networking 28, 2 (2020), 461476. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Blandino S., Mangraviti G., Desset C., et al. 2019. Multi-user hybrid MIMO at 60 GHz using 16-antenna transmitters. IEEE Transactions on Circuits and Systems I: Regular Papers 66, 2 (2019), 848858. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  10. [10] Chakareski J. and Frossard P.. 2008. Distributed collaboration for enhanced sender-driven video streaming. IEEE Trans. Multimedia 10, 5 (Aug. 2008), 858870.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. [11] Begole Bo. 2016. Why the Internet pipes will burst when virtual reality takes off. Forbes Magazine.Google ScholarGoogle Scholar
  12. [12] Chakareski J., Aksu R., Corbillon X., Simon G., and Swaminathan V.. 2018. Viewport-driven rate-distortion optimized 360\(^\circ\) video streaming. In Proc. IEEE Int’l Conf. Communications.Google ScholarGoogle Scholar
  13. [13] Corbillon X., Devlic A., Simon G., and Chakareski J.. 2017. Viewport-adaptive navigable 360-degree video delivery. In Proc. Int’l Conf. Communications. IEEE.Google ScholarGoogle Scholar
  14. [14] Hosseini Mohammad and Swaminathan Viswanathan. 2016. Adaptive 360 VR video streaming: Divide and conquer! In Proc. IEEE Int’l Symp. Multimedia.Google ScholarGoogle Scholar
  15. [15] Cuervo Eduardo, Chintalapudi Krishna, and Kotaru Manikanta. 2018. Creating the perfect illusion: What will it take to create life-like virtual reality headsets? In Proc. HotMobile. ACM, 712.Google ScholarGoogle Scholar
  16. [16] Zhong Ruiguang, Wang Manni, Chen Zijian, Liu Luyang, Liu Yunxin, Zhang Jiansong, Zhang Lintao, and Moscibroda Thomas. 2017. On building a programmable wireless high-quality virtual reality system using commodity hardware. In Proceedings of the 8th Asia-Pacific Workshop on Systems. ACM, 7.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. [17] Abari Omid, Bharadia Dinesh, Duffield Austin, and Katabi Dina. 2017. Enabling high-quality untethered virtual reality. In Proc. Symposium on Networked Systems Design and Implementation. 531544.Google ScholarGoogle Scholar
  18. [18] Khan Mahmudur and Chakareski Jacob. 2019. Visible light communication for next generation untethered virtual reality systems. In Proc. IEEE International Conference on Communications Workshops. 16.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Clemm Alexander, Ravuri Hemanth Torres Vega, Maria Kumar, Wauters Tim, and Turck Filip De. 2020. Viewport-adaptive scalable multi-user virtual reality mobile-edge streaming. IEEE Communications Magazine 58, 1 (Jan. 2020), 9399.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. [20] Liu Zhi, Li Qiyue, Chen Xianfu, Wu Celimuge, Ishihara Susumu, Li Jie, and Ji Yusheng. 2021. Point cloud video streaming: Challenges and solutions. IEEE Network Magazine 35, 5 (Sept. 2021), 202209.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. [21] Chakareski J. and Frossard P.. 2006. Rate-distortion optimized distributed packet scheduling of multiple video streams over shared communication resources. IEEE Trans. Multimedia 8, 2 (April 2006), 207218. Special issue on Distributed Media Technologies and Applications.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. [22] Reis A. B., Chakareski J., Kassler A., and Sargento S.. 2010. Distortion optimized multi-service scheduling for next generation wireless mesh networks. In Proc. IEEE INFOCOM Int’l Workshop on Carrier-grade Wireless Mesh Networks. USA.Google ScholarGoogle Scholar
  23. [23] Chakareski J., Apostolopoulos J., Tan W.-T., Wee S., and Girod B.. 2004. Distortion chains for predicting the video distortion for general packet loss patterns. In Proc. Int’l Conf. Acoustics, Speech, and Signal Processing, Vol. 5. IEEE, 10011004.Google ScholarGoogle Scholar
  24. [24] Chakareski J., Apostolopoulos J., Wee S., Tan W.-T., and Girod B.. 2005. Rate-distortion hint tracks for adaptive video streaming. IEEE Trans. Circuits and Systems for Video Technology 15, 10 (Oct. 2005), 12571269. Special issue on Analysis and Understanding for Video Adaptation.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. [25] Chakareski J. and Chou P. A.. 2006. RaDiO edge: Rate-distortion optimized proxy-driven streaming from the network edge. IEEE/ACM Trans. Networking 14, 6 (Dec. 2006), 13021312.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. [26] Chakareski J. and Chou P. A.. 2004. Application layer error correction coding for rate-distortion optimized streaming to wireless clients. IEEE Trans. Communications 52, 10 (Oct. 2004), 16751687.Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Hussein Ahmed Taha and Elmirghani Jaafar M. H.. 2015. 10 Gbps mobile visible light communication system employing angle diversity, imaging receivers, and relay nodes. OSA Journal of Optical Communications and Networking 7, 8 (2015), 718735.Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] NVIDIA GeForce GTX 980. n.d. https://www.techpowerup.com/gpu-specs/geforce-gtx-980.c2621.Google ScholarGoogle Scholar
  29. [29] GPU Systems. n.d. https://www.supermicro.com/en/products/GPU/.Google ScholarGoogle Scholar
  30. [30] NVIDIA Video Encode and Decode GPU Support Matrix. n.d. https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new.Google ScholarGoogle Scholar
  31. [31] Boyce Jill M., Ye Yan, Chen Jianle, and Ramasubramonian Adarsh K.. 2015. Overview of SHVC: Scalable extensions of the high efficiency video coding standard. IEEE Transactions on Circuits and Systems for Video Technology 26, 1 (2015), 2034.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. [32] Chakareski J.. 2020. Viewport-adaptive scalable multi-user virtual reality mobile-edge streaming. IEEE Trans. Image Processing 29 (2020), 6330–6342.Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] QualComm Adreno 650 GPU. n.d. https://www.qualcomm.com/snapdragon/processors/comparison.Google ScholarGoogle Scholar
  34. [34] Xu Gongxian. 2014. Global optimization of signomial geometric programming problems. European Journal of Operational Research 233, 3 (2014), 500510.Google ScholarGoogle ScholarCross RefCross Ref
  35. [35] Boyd Stephen, Boyd Stephen P., and Vandenberghe Lieven. 2004. Convex Optimization. Cambridge University Press.Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. [36] Willebrand Heinz and Ghuman Baksheesh S.. 2002. Free Space Optics: Enabling Optical Connectivity in Today’s Networks. SAMS Publishing.Google ScholarGoogle Scholar
  37. [37] Zeng Zhihong, Soltani Mohammad Dehghani, Safari Majid, and Haas Harald. May 2019. Angle diversity receiver in LiFi cellular networks. In Proc. IEEE International Conference on Communications. 16.Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Brennan D. G.. 2003. Linear diversity combining techniques. Proc. IEEE 91, 2 (2003), 331356.Google ScholarGoogle ScholarCross RefCross Ref
  39. [39] Maltsev Alexander, Pudeyev Andrey, Karls Ingolf, Bolotin Ilya, Morozov Gregory, Weiler Richard, Peter Michael, and Keusgen Wilhelm. 2014. Quasi-deterministic approach to mmWave channel modeling in a non-stationary environment. In 2014 IEEE Globecom Workshops (GC Wkshps’14). IEEE, 966971.Google ScholarGoogle Scholar
  40. [40] Punnen Abraham P. and Nair K. P. K.. 1994. Improved complexity bound for the maximum cardinality bottleneck bipartite matching problem. Discrete Applied Mathematics 55, 1 (1994), 9193.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. [41] Blandino Steve, Senic Jelena, Gentile Camillo, Caudill Derek, Chuang Jack, and Kayani Atiqa. 2022. Markov multi-beamtracking on 60 GHz mobile channel measurements. IEEE Open Journal of Vehicular Technology 3 (2022), 2639. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  42. [42] Virtual Museum. n.d. https://assetstore.unity.com/packages/3d/environments/museum-117927.Google ScholarGoogle Scholar
  43. [43] SteamVR SDK. n.d. https://github.com/ValveSoftware/openvr.Google ScholarGoogle Scholar
  44. [44] Opentrack tracking. n.d. https://github.com/opentrack/opentrack.Google ScholarGoogle Scholar
  45. [45] Khan M. and Chakareski J.. n.d. NJIT 6DOF VR Navigation Dataset. https://www.jakov.org.Google ScholarGoogle Scholar
  46. [46] NIST and Padova Università di. n.d. Q-D realization software. https://github.com/wigig-tools/qd-realization.Google ScholarGoogle Scholar
  47. [47] Assasa Hany, Widmer Joerg, Ropitault Tanguy, and Golmie Nada. 2019. Enhancing the Ns-3 IEEE 802.11ad model fidelity: Beam codebooks, multi-antenna beamforming training, and quasi-deterministic mmwave channel. In Proceedings of the 2019 Workshop on Ns-3 (WNS3’19). Association for Computing Machinery, New York, NY, 3340. DOI:Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. [48] Assasa Hany, Widmer Joerg, Ropitault Tanguy, Bodi Anuraag, and Golmie Nada. 2019. High fidelity simulation of IEEE 802.11ad in ns-3 using a quasi-deterministic channel model. In Proceedings of the 2019 Workshop on Next-Generation Wireless with ns-3. 2225.Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. [49] Petrangeli Stefano, Swaminathan Viswanathan, Hosseini Mohammad, and Turck Filip De. 2017. Improving virtual reality streaming using HTTP/2. In Proc. Multimedia Systems Conference. ACM, 225228.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Millimeter Wave and Free-space-optics for Future Dual-connectivity 6DOF Mobile Multi-user VR Streaming

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Multimedia Computing, Communications, and Applications
          ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 2
          March 2023
          540 pages
          ISSN:1551-6857
          EISSN:1551-6865
          DOI:10.1145/3572860
          • Editor:
          • Abdulmotaleb El Saddik
          Issue’s Table of Contents

          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 6 February 2023
          • Online AM: 16 June 2022
          • Accepted: 31 May 2022
          • Revised: 19 May 2022
          • Received: 9 February 2022
          Published in tomm Volume 19, Issue 2

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Refereed
        • Article Metrics

          • Downloads (Last 12 months)328
          • Downloads (Last 6 weeks)22

          Other Metrics

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Full Text

        View this article in Full Text.

        View Full Text

        HTML Format

        View this article in HTML Format .

        View HTML Format
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!