Abstract
Conducting user studies to quantify the Quality of Experience (QoE) of watching the increasingly more popular 360° videos in Head-Mounted Displays (HMDs) is time-consuming, tedious, and expensive. Deriving QoE models, however, is very challenging because of the diverse viewing behaviors and complex QoE features and factors. In this article, we compile a wide spectrum of QoE features and factors that may contribute to the overall QoE. We design and conduct a user study to build a dataset of the overall QoE, QoE features, and QoE factors. Using the dataset, we derive the QoE models for both the Mean Opinion Score (MOS) and Individual Score (IS), where MOS captures the aggregated QoE across all subjects, while IS captures the QoE of individual subjects. Our derived overall QoE models achieve 0.98 and 0.91 in Pearson’s Linear Correlation Coefficient (PLCC) for MOS and IS, respectively. Besides, we make several new observations on our user study results, such as (1) content factors dominate the overall QoE across all factor categories, (2) Video Multi-Method Assessment Fusion (VMAF) is the dominating factor among content factors, and (3) the perceived cybersickness is affected by human factors more among others. Our proposed user study design is useful for QoE modeling (specifically) and subjective evaluations (in general) of emerging 360° tiled video streaming to HMDs.
- [1] . 2020. Impact of the impairment in 360-degree videos on users VR involvement and machine learning-based QoE predictions. IEEE Access 8 (2020), 204585–204596.Google Scholar
- [2] . 2020. Subjective QoE of 360-degree virtual reality videos and machine learning predictions. IEEE Access (2020), 148084–148099.Google Scholar
- [3] . 2014. Survey on machine learning-based QoE-QoS correlation models. In Proc. of IEEE International Conference on Computing, Management and Telecommunications (ComManTel’14). Google Scholar
- [4] . 2007. Support vector regression. Neural Information Processing-Letters and Reviews 11, 10 (2007), 203–224.Google Scholar
- [5] . 2016. Does 3D 360 video enhance user’s VR experience: An evaluation study. In Proc. of International Conference on Human Computer Interaction (Interaction’16). Google Scholar
Digital Library
- [6] . 2020. Quality of experience for streaming services: Measurements, challenges and insights. IEEE Access 8 (2020), 13341–13361.Google Scholar
Cross Ref
- [7] . 1993. Transformations for within-subject designs: A Monte Carlo investigation.APA Psychological Bulletin 113, 3 (1993), 566–579.Google Scholar
- [8] 2008. SSIM-Optimal linear image restoration. In Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’08). Google Scholar
Cross Ref
- [9] 2019. VIVE: Discover Virtual Reality Beyond Imagination. Retrieved April 8, 2019, from https://www.vive.com/us/.Google Scholar
- [10] . 2020. Eye Tracking SDK (SRanipal). Retrieved January 22, 2020, from https://developer.vive.com/resources/knowledgebase/vive-sranipal-sdk/.Google Scholar
- [11] . 2020. VIVE PRO EYE. Retrieved January 22, 2020, from https://enterprise.vive.com/us/product/vive-pro-eye/.Google Scholar
- [12] . 2019. Voronoi-based objective quality metrics for omnidirectional video. In Proc. of International Conference on Quality of Multimedia Experience (QoMEX’19).Google Scholar
Cross Ref
- [13] . 2020. Visual attention-aware quality estimation framework for omnidirectional video using spherical Voronoi diagram. Springer Quality and User Experience 5, 1 (2020), 4:1–4:17.Google Scholar
- [14] . 2016. An evaluation of heart rate and electrodermal activity as an objective QoE evaluation method for immersive virtual reality environments. In Proc. of International Conference on Quality of Multimedia Experience (QoMEX’16).Google Scholar
Cross Ref
- [15] . 2019. A survey on 360° video streaming: Acquisition, transmission, and display. Computing Surveys 52, 4 (2019), 71:1–71:36. Google Scholar
Digital Library
- [16] . 2020. On the optimal encoding ladder of tiled 360° videos for head-mounted virtual reality. IEEE Transactions on Circuits and Systems for Video Technology 31, 4 (
July 2020), 1632–1647.Google Scholar - [17] . 2016. Combating VR sickness through subtle dynamic Field-of-View modification. In Proc. of IEEE Symposium on 3D User Interfaces (3DUI’16).Google Scholar
- [18] . 2009. Statistical Models: Theory and Practice. Cambridge University Press.Google Scholar
Cross Ref
- [19] . 2020. Subjective test dataset and meta-data-based models for 360° streaming video quality. In Proc. of IEEE International Workshop on Multimedia Signal Processing (MMSP’20).Google Scholar
- [20] . 2001. Greedy function approximation: A gradient boosting machine. JSTOR Annals of Statistics 29, 5 (2001), 1632–1647.Google Scholar
- [21] . 2002. Postural stability and sickness symptoms after HMD use. In IEEE International Conference on Systems, Man and Cybernetics, Vol. 1.Google Scholar
Cross Ref
- [22] . 2016. QoE beyond the MOS: An in-depth look at QoE via better metrics and their relation to MOS. Quality and User Experience 1, 1 (2016), 2:1–2:23.Google Scholar
Cross Ref
- [23] . 2017. Is foveated rendering perceivable in virtual reality: Exploring the efficiency and consistency of quality assessment methods. In Proc. of ACM International Conference on Multimedia (MM’17). Google Scholar
Digital Library
- [24] . 2018. Modeling the perceptual quality of immersive images rendered on head mounted displays: Resolution and compression. IEEE Transactions on Image Processing 27, 12 (2018), 6039–6050. Google Scholar
Digital Library
- [25] 2019. VMAF - Video Multi-Method Assessment Fusion. https://github.com/Netflix/vmaf.Google Scholar
- [26] 2012. Information Technology – Dynamic Adaptive Streaming Over HTTP (DASH) – Part 1: Media Presentation Description and Segment Formats. (
March 2012).Google Scholar - [27] 2017. Algorithm Descriptions of Projection Format Conversion and Video Quality Metrics in 360Lib.
Standard . International Telecommunication Union.Google Scholar - [28] . 2019. JVET - Joint Video Experts Team. https://www.itu.int/en/ITU-T/studygroups/2017-2020/16/Pages/video/jvet.aspx.Google Scholar
- [29] 2000. Final Report from the Video Quality Experts Group on the Validation of Objective Models of Video Quality Assessment.
Technical Report . VQEG.Google Scholar - [30] . 1999. Subjective video quality assessment methods for multimedia applications. ITU-T Recommendation P.910 (
September 1999).Google Scholar - [31] . 2019. A benchmark of four methods for generating 360° saliency maps from eye tracking data. World Scientific International Journal of Semantic Computing 13, 3 (2019), 329–341.Google Scholar
- [32] . 2016. Measurement of quality of experience of video-on-demand services: A survey. IEEE Communications Surveys & Tutorials 18, 1 (2016), 401–418.Google Scholar
Digital Library
- [33] . 2010. Framing the context of use for mobile HCI. IGI Global International Journal of Mobile Human Computer Interaction 2, 4 (2010), 1–28. Google Scholar
Digital Library
- [34] . 1993. Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness. International Journal of Aviation Psychology 3, 3 (1993), 203–220.Google Scholar
Cross Ref
- [35] . 2010. QoE measurement of mobile YouTube video streaming. In Proc. of Workshop on Mobile Video Delivery (MoVid’10). Google Scholar
Digital Library
- [36] . 2018. VRSA net: VR sickness assessment considering exceptional motion for 360 VR video. IEEE Transactions on Image Processing 28, 4 (2018), 1646–1660.Google Scholar
Digital Library
- [37] . 2018. Virtual reality sickness predictor: Analysis of visual-vestibular conflict and VR contents. In Proc. of IEEE International Conference on Quality of Multimedia Experience (QoMEX’18).Google Scholar
Cross Ref
- [38] . 2019. Motion sickness prediction in stereoscopic videos using 3D convolutional neural networks. IEEE Transactions on Visualization and Computer Graphics 25, 5 (2019), 1919–1927.Google Scholar
Cross Ref
- [39] . 2002. Classification and Regression by RandomForest. R News 2, 3 (2002), 18–22.Google Scholar
Cross Ref
- [40] . 2017. Performance measurements of 360° video streaming to head-mounted displays over live 4G cellular networks. In Proc. of Asia-Pacific Network Operations and Management Symposium (APNOMS’17).Google Scholar
- [41] . 1981. An iterative image registration technique with an application to stereo vision. In Proc. of the International Joint Conference on Artificial Intelligence (IJCAI’81). Google Scholar
Digital Library
- [42] 2020. MATLAB - MathWorks. Retrieved August 4, 2020, from https://www.mathworks.com/products/matlab.html.Google Scholar
- [43] . 2013. An overview of tiles in HEVC. IEEE Journal of Selected Topics in Signal Processing 7, 6 (
December 2013), 969–977.Google ScholarCross Ref
- [44] . 2014. Features of quality of experience. In Quality of Experience, and (Eds.). Springer US, Chapter 5.Google Scholar
Cross Ref
- [45] . 1996. Context and Consciousness: Activity Theory and Human-Computer Interaction. MIT Press. Google Scholar
Digital Library
- [46] 2016. NFLX Dataset. Retrieved May 17, 2020, from https://drive.google.com/drive/u/0/folders/0B3YWNICYMBIweGdJbERlUG9zc0k.Google Scholar
- [47] . 2020. QoE-Modeling-for-360-Degree-Videos-Dataset. Retrieved August 11, 2020, from https://github.com/nmsl-nthu/QoE-Modeling-for-360-Degree-Videos-Dataset.Google Scholar
- [48] . 2019. Facebook Oculus Rift. Retrieved April 8, 2019, from https://www.oculus.com/.Google Scholar
- [49] . 2019. Video multimethod assessment fusion (VMAF) on 360VR contents. IEEE Transactions on Consumer Electronics 66, 1 (2019), 22–31.Google Scholar
Digital Library
- [50] . 2018. Towards a machine-learning approach for sickness prediction in 360 stereoscopic videos. IEEE Transactions on Visualization and Computer Graphics 24, 4 (2018), 1594–1603. Google Scholar
Digital Library
- [51] . 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12 (2011), 2825–2830. Google Scholar
Digital Library
- [52] . 2020. SiSiMo: Towards simulator sickness modeling for 360° videos viewed with an HMD. In Proc. of IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW’20).Google Scholar
Cross Ref
- [53] . 2010. Augmenting context awareness by combining body sensor networks and social networks. IEEE Transactions on Instrumentation and Measurement 60, 2 (2010), 345–353.Google Scholar
Cross Ref
- [54] . 2017. Which saliency weighting for omni directional image quality assessment? In Proc. of IEEE International Conference on Quality of Multimedia Experience (QoMEX’17).Google Scholar
Cross Ref
- [55] . 2018. VRate: A Unity3D asset for integrating subjective assessment questionnaires in virtual environments. In Proc. of IEEE International Conference on Quality of Multimedia Experience (QoMEX’18).Google Scholar
Cross Ref
- [56] . 2014. Factors influencing quality of experience. In Quality of Experience, and (Eds.). Springer US, Chapter 4.Google Scholar
Cross Ref
- [57] . 2018. Assessing the QoE impact of 3D rendering style in the context of VR-based training. In Proc. of IEEE International Conference on Quality of Multimedia Experience (QoMEX’18).Google Scholar
Cross Ref
- [58] . 2008. Subjective video quality assessment methods for multimedia applications. ITU-T Recommendation P.910 (
April 2008).Google Scholar - [59] . 2016. Mean opinion score (MOS) terminology. ITU-T Recommendation P.800.1 (
July 2016).Google Scholar - [60] . 2016. Methods for the subjective assessment of video quality, audio quality and audiovisual quality of Internet video and distribution quality television in any environment. ITU-T Recommendation P.913 (
March 2016).Google Scholar - [61] . 2016. Series P: Terminals and subjective and objective assessment methods. ITU-T Recommendation P.800.2 (
July 2016).Google Scholar - [62] . 1988. Maximum Z scores and outliers. American Statistician 42, 1 (1988), 79–80.Google Scholar
- [63] . 2017. Comparison of subjective quality evaluation for HEVC encoded omnidirectional videos at different bit-rates for UHD and FHD resolution. In Proc. of ACM Multimedia Thematic Workshops. Google Scholar
Digital Library
- [64] . 2017. Measuring and comparing QoE and simulator sickness of omnidirectional videos in different head mounted displays. In Proc. of International Conference on Quality of Multimedia Experience (QoMEX’17).Google Scholar
Cross Ref
- [65] . 2019. Subjective quality evaluation of tile-based streaming for omnidirectional videos. In Proc. of ACM Conference on Multimedia Systems (MMSys’19). Google Scholar
Digital Library
- [66] . 2011. The MPEG-DASH standard for multimedia streaming over the Internet. IEEE Multimedia 18, 4 (2011), 62–67. Google Scholar
Digital Library
- [67] . 2015. QoE modeling for VP9 and H.265 videos on mobile devices. In Proc. of ACM International Conference on Multimedia (MM’15). Google Scholar
Digital Library
- [68] . 1904. The proof and measurement of association between two things. American Journal of Psychology 15, 1 (1904), 72–101.Google Scholar
Cross Ref
- [69] . 2016. A questionnaire to measure the user experience in immersive virtual environments. In Proc. of ACM Virtual Reality International Conference (VRIC’16). Google Scholar
Digital Library
- [70] . 2020. SteamVR Plugin. Retrieved January 22, 2020, from https://assetstore.unity.com/packages/tools/integration/steamvr-plugin-32647.Google Scholar
- [71] . 2020. Equi-Angular Cubemap Skybox for Unity. Retrieved August 11, 2020, from https://github.com/trejkaz/EACSkyboxShader.Google Scholar
- [72] . 2011. Perceptual Video Quality Dimensions. Master’s thesis. Technische, Universität Berlin, Berlin, Germany.Google Scholar
- [73] 2020. Unity. Retrieved May 22, 2020, from https://unity3d.com/.Google Scholar
- [74] . 2016. Testbed for subjective evaluation of omnidirectional visual content. In Proc. of Picture Coding Symposium (PCS’16).Google Scholar
- [75] . 2017. On the performance of objective metrics for omnidirectional visual content. In Proc. of International Conference on Quality of Multimedia Experience (QoMEX’17).Google Scholar
Cross Ref
- [76] . 2014. Quality of service versus quality of experience. In Quality of Experience, and (Eds.). Springer US, Chapter 6.Google Scholar
Cross Ref
- [77] . 2016. Kvazaar: Open-Source HEVC/H.265 Encoder. In Proc. of ACM International Conference on Multimedia (MM’16). Google Scholar
Digital Library
- [78] . 2018. Subjective assessment of different locomotion techniques in virtual reality environments. In Proc. of IEEE International Conference on Quality of Multimedia Experience (QoMEX’18).Google Scholar
Cross Ref
- [79] . 2014. Mixing tile resolutions in tiled video: A perceptual quality assessment. In Proc. of ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV’14). Google Scholar
Digital Library
- [80] . 2016. Foveated real-time ray tracing for head-mounted displays. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 289–298. Google Scholar
Digital Library
- [81] . 1958. Visual sensory units and the minimal angle of resolution. Elsevier American Journal of Ophthalmology 46, 1 (1958), 102–113.Google Scholar
- [82] . 2017. 360ProbDASH: Improving QoE of 360 Video streaming using Tile-Based HTTP adaptive streaming. In Proc. of ACM International Conference on Multimedia (MM’17). Google Scholar
Digital Library
- [83] . 2018. Modeling the perceptual impact of viewport adaptation for immersive video. In Proc. of IEEE International Symposium on Circuits and Systems (ISCAS’18).Google Scholar
- [84] . 2019. Towards quality-of-experience models for watching 360° videos in head-mounted virtual reality. In Proc. of International Conference on Quality of Multimedia Experience (QoMEX’19).Google Scholar
Cross Ref
- [85] . 2015. Content adaptive representations of omnidirectional videos for cinematic virtual reality. In Proc. of ACM International Workshop on Immersive Media Experiences (ImmersiveMe’15). Google Scholar
Digital Library
Index Terms
Modeling the User Experience of Watching 360° Videos with Head-Mounted Displays






Comments