Abstract
The media quality assessment research community has traditionally been focusing on developing objective algorithms to predict the result of a typical subjective experiment in terms of Mean Opinion Score (MOS) value. However, the MOS, being a single value, is insufficient to model the complexity and diversity of human opinions encountered in an actual subjective experiment. In this work we propose a complementary approach for objective media quality assessment that attempts to more closely model what happens in a subjective experiment in terms of single observers and, at the same time, we perform a qualitative analysis of the proposed approach while highlighting its suitability. More precisely, we propose to model, using neural networks (NNs), the way single observers perceive media quality. Once trained, these NNs, one for each observer, are expected to mimic the corresponding observer in terms of quality perception. Then, similarly to a subjective experiment, such NNs can be used to simulate the users’ single opinions, which can be later aggregated by means of different statistical indicators such as average, standard deviation, quantiles, etc. Unlike previous approaches that consider subjective experiments as a black box providing reliable ground truth data for training, the proposed approach is able to consider human factors by analyzing and weighting individual observers. Such a model may therefore implicitly account for users’ expectations and tendencies, that have been shown in many studies to significantly correlate with visual quality perception. Furthermore, our proposal also introduces and investigates an index measuring how much inconsistency there would be if an observer was asked to rate many times the same stimulus. Simulation experiments conducted on several datasets demonstrate that the proposed approach can be effectively implemented in practice and thus yielding a more complete objective assessment of end users’ quality of experience.
- [1] . 2019. Spatiotemporal feature integration and model fusion for full reference video quality assessment. IEEE Transactions on Circuits and Systems for Video Technology 29, 8 (
Aug 2019), 2256–2270.DOI: DOI: https://doi.org/10.1109/TCSVT.2018.2868262Google ScholarCross Ref
- [2] . 2015. Objective video quality assessment-towards large scale video database enhanced model development. IEICE Transactions on Communications E98B, 1 (2015), 2–11.
DOI: DOI: https://doi.org/10.1587/transcom.E98.B.2Google ScholarCross Ref
- [3] . 1997. Efficiency of the Ishihara test for identifying red-green colour deficiency. Ophthalmic and Physiological Optics 17, 5 (1997), 403–408.Google Scholar
Cross Ref
- [4] . 2012. Qualinet white paper on definitions of Quality of Experience. (2012).
European Network on Quality of Experience in Multimedia Systems and Services (COST Action IC 1003). Google Scholar - [5] . 2002. Methodology for the subjective assessment of the quality of television pictures. (
June 2002).Google Scholar - [6] . 2011. Objective video quality assessment methods: A classification, review, and performance comparison. IEEE Transactions on Broadcasting 57, 2 (
Feb 2011), 165–182.Google ScholarCross Ref
- [7] . 2020. Annual Internet Report: Growth in Internet users (2018-2023). (2020). https://www.cisco.com/c/en/us/solutions/collateral/executive-perspectives/annual-internet-report/white-paper-c11-741490.html.Google Scholar
- [8] . 2020. Recommender systems leveraging multimedia content. ACM Computing Surveys (CSUR) 53, 5 (2020), 1–38. Google Scholar
Digital Library
- [9] . 2019. Computing Quality-of-Experience ranges for video quality estimation. In Eleventh International Conference on Quality of Multimedia Experience (QoMEX). IEEE, Berlin, Germany, 1–3.
DOI: DOI: https://doi.org/10.1109/QoMEX.2019.8743303Google ScholarCross Ref
- [10] . 2020. Predicting single observer’s votes from objective measures using neural networks. In Proceedings of Human Vision and Electronic Imaging Conference (HVEI). Society for Imaging Science and Technology (IS&T), Burlingame, CA, USA.Google Scholar
- [11] . 2020. Full reference video quality measure improvement using neural networks. In Proc. Intl. Conf. Acoustics, Speech, and Signal Processing (ICASSP). IEEE, Barcelona, Spain, 2737–2741.
DOI: DOI: https://doi.org/10.1109/ICASSP40776.2020.9053739Google Scholar - [12] . 2016. On the influence of individual characteristics and personality traits on the user experience with multi-sensorial media: An experimental insight. Multimedia Tools and Applications 75 (
Feb 2016).DOI: DOI: https://doi.org/10.1007/s11042-016-3360-z Google ScholarCross Ref
- [13] . 2019. Extension of the ITS4S Dataset. (
Jan 2019). http://media.polito.it/its4s.Google Scholar - [14] . 2016. QoE beyond the MOS: An in-depth look at QoE via better metrics and their relation to MOS. Quality and User Experience 1, 1 (
Sep 2016).DOI: DOI: https://doi.org/10.1007/s41233-016-0002-1Google ScholarCross Ref
- [15] . 2011. SOS: The MOS is not enough!. In Third International Workshop on Quality of Multimedia Experience (QoMEX). IEEE, Mechelen, Belgium, 131–136.
DOI: DOI: https://doi.org/10.1109/QoMEX.2011.6065690Google Scholar - [16] . 2014. Personalized video recommendation through graph propagation. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 10, 4 (2014), 1–17. Google Scholar
Digital Library
- [17] . 2012. Are QoE requirements for multimedia services different for men and women? Analysis of gender differences in forming QoE in virtual acoustic environments. In Intl. Multi Topic Conference on Emerging Trends and Applications in Information Communication Technologies (IMTIC), Vol. 281. Springer, Jamshoro, Pakistan.
DOI: DOI: https://doi.org/10.1007/978-3-642-28962-0_20Google ScholarCross Ref
- [18] . 2007. Definition of Quality of Experience (QoE). (
Jan 2007).Google Scholar - [19] . 2018. QoE assessment for IoT-based multi sensorial media broadcasting. IEEE Transactions on Broadcasting 64, 2 (2018), 552–560.Google Scholar
Cross Ref
- [20] . 2009. Modeling subjective tests of quality of experience with a generalized linear model. In International Workshop on Quality of Multimedia Experience (QoMEX). IEEE, San Diego, CA, USA, 35–40.
DOI: DOI: https://doi.org/10.1109/QOMEX.2009.5246979Google Scholar - [21] . 2015. The accuracy of subjects in a quality experiment: A theoretical subject model. IEEE Transactions on Multimedia 17 (
12 2015), 2210–2224.DOI: DOI: https://doi.org/10.1109/TMM.2015.2484963Google ScholarCross Ref
- [22] . 2009. The big picture on small screens delivering acceptable video quality in mobile TV. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 5, 3 (2009), 1–27. Google Scholar
Digital Library
- [23] . 2019. Assessing personally perceived image quality via image features and collaborative filtering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Long Beach, CA, USA, 8169–8177.Google Scholar
Cross Ref
- [24] . 2020. Training objective image and video quality estimators using multiple databases. IEEE Transactions on Multimedia 22, 4 (2020), 961–969.Google Scholar
Digital Library
- [25] . 2006. A convolutional neural network approach for objective video quality assessment. IEEE Transactions on Neural Networks 17, 5 (
Sep 2006), 1316–1327. Google ScholarDigital Library
- [26] . 2016. Recent developments in visual quality monitoring by key performance indicators. Multimedia Tools and Applications 75 (2016), 10745-10767.
DOI: DOI: https://doi.org/10.1007/s11042-014-2229-2 Google ScholarCross Ref
- [27] . 2017. Recover subjective quality scores from noisy measurements. In Data Compression Conference (DCC). IEEE, Snowbird, UT, USA, 52–61.Google Scholar
Cross Ref
- [28] . 2018. Analyzing ordinal data with metric models: What could possibly go wrong?Journal of Experimental Social Psychology 79 (
Nov 2018), 328–348.DOI: DOI: https://doi.org/10.1016/j.jesp.2018.08.009Google ScholarCross Ref
- [29] . 2015. Context-aware QoE modelling, measurement and prediction in mobile computing systems. IEEE Transactions on Mobile Computing 14 (
May 2015), 920–936.DOI: DOI: https://doi.org/10.1109/TMC.2013.155Google ScholarCross Ref
- [30] . 2001. New techniques for assessing audio and video quality in real-time interactive communications. In IHM-HCI Tutorial. Lille, France.Google Scholar
- [31] . 2010. Multimodal interaction: A suitable strategy for including older users?Interacting with Computers 22, 6 (
Nov 2010), 465–474.DOI: DOI: https://doi.org/10.1016/j.intcom.2010.08.005 Google ScholarCross Ref
- [32] . 2019. VMAF - Video Multi-Method Assessment Fusion. https://github.com/Netflix/vmaf. (
Jan. 2019).Google Scholar - [33] . 2010. Likert scales, levels of measurement and the “laws” of statistics. Advances in Health Sciences Education 15 (2010), 625–632.Google Scholar
Cross Ref
- [34] . 2008. Subjective video quality assessment methods for multimedia applications. (
Apr 2008).Google Scholar - [35] . 2012. Quality of experience assessment in internet TV. In Proc. Intl. Conf. on Mobile Networks and Management. Springer, Aveiro, Portugal, 261–274.Google Scholar
- [36] . 2018. ITS4S: A Video Quality Dataset with Four-Second Unrepeated Scenes. (
Feb 2018).NTIA, Technical Memo TM-18-532. Google Scholar - [37] . 2012. When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods 17, 3 (2012), 354–373.Google Scholar
Cross Ref
- [38] . 2007. Motion based reference-free quality estimation for H.264/AVC video streaming. In 2007 2nd International Symposium on Wireless Pervasive Computing. IEEE, San Juan, Puerto Rico.
DOI: DOI: https://doi.org/10.1109/ISWPC.2007.342629Google Scholar - [39] . 2016. Do personality and culture influence perceived video quality and enjoyment?IEEE Transactions on Multimedia 18, 9 (2016), 1796–1807. Google Scholar
Digital Library
- [40] . 2019. Fundamental advantages of considering Quality of Experience distributions over mean opinion scores. In 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX). IEEE, Berlin, Germany, 1–6.
DOI: DOI: https://doi.org/10.1109/QoMEX.2019.8743296Google ScholarCross Ref
- [41] . 2006. Image information and visual quality. IEEE Transactions on Image Processing 15, 2 (
Feb 2006), 430–444. Google ScholarDigital Library
- [42] . 2016. Mean opinion score (MOS) revisited: Methods and applications, limitations and alternatives. Multimedia Systems 22, 2 (
Mar 2016), 213–227.DOI: DOI: https://doi.org/10.1007/s00530-014-0446-1 Google ScholarCross Ref
- [43] . 2007. Test distance vision using a Snellen chart.Community Eye Health 20, 63 (2007), 52.Google Scholar
- [44] . 2019. No-reference video quality assessment based on the temporal pooling of deep features. Neural Processing Letters 50, 3 (
12 Apr 2019), 2595–2608.DOI: DOI: https://doi.org/10.1007/s11063-019-10036-6Google ScholarCross Ref
- [45] . 2018. Deeprn: A content preserving deep architecture for blind image quality assessment. In 2018 IEEE International Conference on Multimedia and Expo (ICME). IEEE, San Diego, CA, USA, 1–6.
DOI: DOI: https://doi.org/10.1109/ICME.2018.8486528Google ScholarCross Ref
- [46] . 2010. Report on the Validation of Video Quality Models for High Definition Video Content (v. 2.0). (
Jun 2010). http://bit.ly/2Z7GWDIGoogle Scholar - [47] . 2003. Multiscale structural similarity for image quality assessment. In The Thirty-Seventh Asilomar Conference on Signals, Systems & Computers, Vol. 2. IEEE, Pacific Grove, CA, USA, 1398–1402.
DOI: DOI: https://doi.org/10.1109/ACSSC.2003.1292216Google ScholarCross Ref
- [48] . 2011. All users are (not) equal - the influence of user characteristics on perceived quality, modality choice and performance. In Proc. IWSDS Workshop on Paralinguistic Information and its Integration in Spoken Dialogue Systems. Springer, New York, NY, USA, 175–186.
DOI: DOI: https://doi.org/10.1007/978-1-4614-1335-6_19Google Scholar - [49] . 2018. Quality matters: Assessing cQA pair quality via transductive multi-view learning. In Proc. of International Joint Conference on Artificial Intelligence (IJCAI). Stockholm, Sweden, 4482–4488. Google Scholar
Digital Library
- [50] . 2008. The evolution of video quality measurement: From PSNR to hybrid metrics. IEEE Transactions on Broadcasting 54, 3 (
Sep 2008), 660–668.DOI: DOI: https://doi.org/10.1109/TBC.2008.2000733Google ScholarCross Ref
- [51] . 2016. Free-energy principle inspired video quality metric and its use in video coding. IEEE Transactions on Multimedia 18, 4 (
Feb 2016), 590–602.Google ScholarDigital Library
- [52] . 2019. Naturalness-aware deep no-reference image quality assessment. IEEE Transactions on Multimedia 21, 10 (
Mar 2019), 2603–2615.Google ScholarDigital Library
- [53] . 2012. Neighborhood component feature selection for high-dimensional data. Journal of Computers 7 (
Jan 2012), 161–168.Google ScholarCross Ref
- [54] . 2019. Deep neural networks for no-reference video quality assessment. In IEEE International Conference on Image Processing (ICIP). IEEE, Taipei, Taiwan, 2349–2353.Google Scholar
Cross Ref
- [55] . 2017. A probabilistic quality representation approach to deep blind image quality prediction. (2017).
arXiv:arXiv:1708.08190v2 Google Scholar - [56] . 2020. Objective video quality assessment combining transfer learning with CNN. IEEE Transactions on Neural Networks and Learning Systems 31, 8 (2020), 2716–2730.
DOI: DOI: https://doi.org/10.1109/TNNLS.2018.2890310Google ScholarCross Ref
- [57] . 2004. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (
Apr 2004), 600–612.DOI: DOI: https://doi.org/10.1109/TIP.2003.819861 Google ScholarDigital Library
- [58] . 2018. Measuring individual video QoE: A survey, and proposal for future directions using social media. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 14, 2s (2018), 1–24. Google Scholar
Digital Library
Index Terms
Mimicking Individual Media Quality Perception with Neural Network based Artificial Observers
Recommendations
Dynamic neural network-based robust observers for uncertain nonlinear systems
A dynamic neural network (DNN) based robust observer for uncertain nonlinear systems is developed. The observer structure consists of a DNN to estimate the system dynamics on-line, a dynamic filter to estimate the unmeasurable state and a sliding mode ...
Data-based adaptive neural network optimal output feedback control for nonlinear systems with actuator saturation
This paper investigates the adaptive neural network optimal output feedback control design problem for nonlinear continuous-time systems with actuator saturation. The system dynamics and states of the controlled system are unknown. A neural network ...
Observer-based adaptive neural control for a class of nonlinear pure-feedback systems
This paper focuses on the issue of adaptive neural output-feedback control for a class of nonlinear pure-feedback systems. A state observer is first constructed to estimate the unmeasurable state variables. Radial basis function (RBF) neural networks (...






Comments