skip to main content
research-article

Mimicking Individual Media Quality Perception with Neural Network based Artificial Observers

Published:27 January 2022Publication History
Skip Abstract Section

Abstract

The media quality assessment research community has traditionally been focusing on developing objective algorithms to predict the result of a typical subjective experiment in terms of Mean Opinion Score (MOS) value. However, the MOS, being a single value, is insufficient to model the complexity and diversity of human opinions encountered in an actual subjective experiment. In this work we propose a complementary approach for objective media quality assessment that attempts to more closely model what happens in a subjective experiment in terms of single observers and, at the same time, we perform a qualitative analysis of the proposed approach while highlighting its suitability. More precisely, we propose to model, using neural networks (NNs), the way single observers perceive media quality. Once trained, these NNs, one for each observer, are expected to mimic the corresponding observer in terms of quality perception. Then, similarly to a subjective experiment, such NNs can be used to simulate the users’ single opinions, which can be later aggregated by means of different statistical indicators such as average, standard deviation, quantiles, etc. Unlike previous approaches that consider subjective experiments as a black box providing reliable ground truth data for training, the proposed approach is able to consider human factors by analyzing and weighting individual observers. Such a model may therefore implicitly account for users’ expectations and tendencies, that have been shown in many studies to significantly correlate with visual quality perception. Furthermore, our proposal also introduces and investigates an index measuring how much inconsistency there would be if an observer was asked to rate many times the same stimulus. Simulation experiments conducted on several datasets demonstrate that the proposed approach can be effectively implemented in practice and thus yielding a more complete objective assessment of end users’ quality of experience.

REFERENCES

  1. [1] Bampis C. G., Li Z., and Bovik A. C.. 2019. Spatiotemporal feature integration and model fusion for full reference video quality assessment. IEEE Transactions on Circuits and Systems for Video Technology 29, 8 (Aug 2019), 22562270. DOI: DOI: https://doi.org/10.1109/TCSVT.2018.2868262Google ScholarGoogle ScholarCross RefCross Ref
  2. [2] Barkowsky M., Masala E., Van Wallendael G., Brunnström K., Staelens N., and Callet P. Le. 2015. Objective video quality assessment-towards large scale video database enhanced model development. IEICE Transactions on Communications E98B, 1 (2015), 211. DOI: DOI: https://doi.org/10.1587/transcom.E98.B.2Google ScholarGoogle ScholarCross RefCross Ref
  3. [3] Birch Jennifer. 1997. Efficiency of the Ishihara test for identifying red-green colour deficiency. Ophthalmic and Physiological Optics 17, 5 (1997), 403408.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Brunnström Kjell et al. 2012. Qualinet white paper on definitions of Quality of Experience. (2012). European Network on Quality of Experience in Multimedia Systems and Services (COST Action IC 1003).Google ScholarGoogle Scholar
  5. [5] BT.500-11 ITU-R Rec.. 2002. Methodology for the subjective assessment of the quality of television pictures. (June 2002).Google ScholarGoogle Scholar
  6. [6] Chikkerur S., Sundaram V., Reisslein M., and Karam L. J.. 2011. Objective video quality assessment methods: A classification, review, and performance comparison. IEEE Transactions on Broadcasting 57, 2 (Feb 2011), 165182.Google ScholarGoogle ScholarCross RefCross Ref
  7. [7] Cisco. 2020. Annual Internet Report: Growth in Internet users (2018-2023). (2020). https://www.cisco.com/c/en/us/solutions/collateral/executive-perspectives/annual-internet-report/white-paper-c11-741490.html.Google ScholarGoogle Scholar
  8. [8] Deldjoo Yashar, Schedl Markus, Cremonesi Paolo, and Pasi Gabriella. 2020. Recommender systems leveraging multimedia content. ACM Computing Surveys (CSUR) 53, 5 (2020), 138. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Tiotsop Lohic Fotio, Masala Enrico, Aldahdooh Ahmed, Van Wallendael Glenn, and Barkowsky Marcus. 2019. Computing Quality-of-Experience ranges for video quality estimation. In Eleventh International Conference on Quality of Multimedia Experience (QoMEX). IEEE, Berlin, Germany, 13. DOI: DOI: https://doi.org/10.1109/QoMEX.2019.8743303Google ScholarGoogle ScholarCross RefCross Ref
  10. [10] Tiotsop Lohic Fotio, Mizdos Tomas, Uhrina Miroslav, Pocta Peter, Barkowsky Marcus, and Masala Enrico. 2020. Predicting single observer’s votes from objective measures using neural networks. In Proceedings of Human Vision and Electronic Imaging Conference (HVEI). Society for Imaging Science and Technology (IS&T), Burlingame, CA, USA.Google ScholarGoogle Scholar
  11. [11] Tiotsop Lohic Fotio, Servetti Antonio, and Masala Enrico. 2020. Full reference video quality measure improvement using neural networks. In Proc. Intl. Conf. Acoustics, Speech, and Signal Processing (ICASSP). IEEE, Barcelona, Spain, 27372741. DOI: DOI: https://doi.org/10.1109/ICASSP40776.2020.9053739Google ScholarGoogle Scholar
  12. [12] Galloso Iris, Palacios Juan, Feijoo Claudio, and Santamaria Asuncion. 2016. On the influence of individual characteristics and personality traits on the user experience with multi-sensorial media: An experimental insight. Multimedia Tools and Applications 75 (Feb 2016). DOI: DOI: https://doi.org/10.1007/s11042-016-3360-z Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Group Internet Media. 2019. Extension of the ITS4S Dataset. (Jan 2019). http://media.polito.it/its4s.Google ScholarGoogle Scholar
  14. [14] Hoßfeld Tobias, Heegaard Poul E., Varela Martín, and Möller Sebastian. 2016. QoE beyond the MOS: An in-depth look at QoE via better metrics and their relation to MOS. Quality and User Experience 1, 1 (Sep 2016). DOI: DOI: https://doi.org/10.1007/s41233-016-0002-1Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Hoßfeld Tobias, Schatz Raimund, and Egger Sebastian. 2011. SOS: The MOS is not enough!. In Third International Workshop on Quality of Multimedia Experience (QoMEX). IEEE, Mechelen, Belgium, 131136. DOI: DOI: https://doi.org/10.1109/QoMEX.2011.6065690Google ScholarGoogle Scholar
  16. [16] Huang Qinghua, Chen Bisheng, Wang Jingdong, and Mei Tao. 2014. Personalized video recommendation through graph propagation. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 10, 4 (2014), 117. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. [17] Hyder Mansoor, Hoene Christian, and Crespi Noel. 2012. Are QoE requirements for multimedia services different for men and women? Analysis of gender differences in forming QoE in virtual acoustic environments. In Intl. Multi Topic Conference on Emerging Trends and Applications in Information Communication Technologies (IMTIC), Vol. 281. Springer, Jamshoro, Pakistan. DOI: DOI: https://doi.org/10.1007/978-3-642-28962-0_20Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] 1 ITU-T Rec. G.100 Amd.. 2007. Definition of Quality of Experience (QoE). (Jan 2007).Google ScholarGoogle Scholar
  19. [19] Jalal Lana, Anedda Matteo, Popescu Vlad, and Murroni Maurizio. 2018. QoE assessment for IoT-based multi sensorial media broadcasting. IEEE Transactions on Broadcasting 64, 2 (2018), 552560.Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Janowski Lucjan and Papir Zdzislaw. 2009. Modeling subjective tests of quality of experience with a generalized linear model. In International Workshop on Quality of Multimedia Experience (QoMEX). IEEE, San Diego, CA, USA, 3540. DOI: DOI: https://doi.org/10.1109/QOMEX.2009.5246979Google ScholarGoogle Scholar
  21. [21] Janowski Lucjan and Pinson Margaret. 2015. The accuracy of subjects in a quality experiment: A theoretical subject model. IEEE Transactions on Multimedia 17 (12 2015), 22102224. DOI: DOI: https://doi.org/10.1109/TMM.2015.2484963Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Knoche Hendrik and Sasse Martina Angela. 2009. The big picture on small screens delivering acceptable video quality in mobile TV. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 5, 3 (2009), 127. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. [23] Korhonen Jari. 2019. Assessing personally perceived image quality via image features and collaborative filtering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Long Beach, CA, USA, 81698177.Google ScholarGoogle ScholarCross RefCross Ref
  24. [24] Krasula L., Baveye Y., and Callet P. Le. 2020. Training objective image and video quality estimators using multiple databases. IEEE Transactions on Multimedia 22, 4 (2020), 961969.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. [25] Callet P. Le, Viard-Gaudin C., and Barba D.. 2006. A convolutional neural network approach for objective video quality assessment. IEEE Transactions on Neural Networks 17, 5 (Sep 2006), 13161327. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. [26] Leszczuk Mikolaj, Hanusiak Mateusz, Farias Mylene, Wyckens Emmanuel, and Heston George. 2016. Recent developments in visual quality monitoring by key performance indicators. Multimedia Tools and Applications 75 (2016), 10745-10767. DOI: DOI: https://doi.org/10.1007/s11042-014-2229-2 Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Li Z. and Bampis C. G.. 2017. Recover subjective quality scores from noisy measurements. In Data Compression Conference (DCC). IEEE, Snowbird, UT, USA, 5261.Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] Liddell Torrin M. and Kruschke John K.. 2018. Analyzing ordinal data with metric models: What could possibly go wrong?Journal of Experimental Social Psychology 79 (Nov 2018), 328348. DOI: DOI: https://doi.org/10.1016/j.jesp.2018.08.009Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Mitra Karan, Zaslavsky Arkady, and Ahlund Christer. 2015. Context-aware QoE modelling, measurement and prediction in mobile computing systems. IEEE Transactions on Mobile Computing 14 (May 2015), 920936. DOI: DOI: https://doi.org/10.1109/TMC.2013.155Google ScholarGoogle ScholarCross RefCross Ref
  30. [30] Mullin Jim, Smallwood Lucy, Watson Anna, and Wilson Gillian. 2001. New techniques for assessing audio and video quality in real-time interactive communications. In IHM-HCI Tutorial. Lille, France.Google ScholarGoogle Scholar
  31. [31] Naumann Anja B., Wechsung Ina, and Hurtienne Jorn. 2010. Multimodal interaction: A suitable strategy for including older users?Interacting with Computers 22, 6 (Nov 2010), 465474. DOI: DOI: https://doi.org/10.1016/j.intcom.2010.08.005 Google ScholarGoogle ScholarCross RefCross Ref
  32. [32] Netflix. 2019. VMAF - Video Multi-Method Assessment Fusion. https://github.com/Netflix/vmaf. (Jan. 2019).Google ScholarGoogle Scholar
  33. [33] Norman Geoff. 2010. Likert scales, levels of measurement and the “laws” of statistics. Advances in Health Sciences Education 15 (2010), 625632.Google ScholarGoogle ScholarCross RefCross Ref
  34. [34] P.910 ITU-T Rec.. 2008. Subjective video quality assessment methods for multimedia applications. (Apr 2008).Google ScholarGoogle Scholar
  35. [35] Palhais Joana, Cruz Rui S., and Nunes Mário S.. 2012. Quality of experience assessment in internet TV. In Proc. Intl. Conf. on Mobile Networks and Management. Springer, Aveiro, Portugal, 261274.Google ScholarGoogle Scholar
  36. [36] Pinson Margaret H.. 2018. ITS4S: A Video Quality Dataset with Four-Second Unrepeated Scenes. (Feb 2018). NTIA, Technical Memo TM-18-532.Google ScholarGoogle Scholar
  37. [37] Rhemtulla Mijke, Brosseau-Liard Patricia, and Savalei Victoria. 2012. When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods 17, 3 (2012), 354373.Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Ries M., Nemethova O., and Rupp M.. 2007. Motion based reference-free quality estimation for H.264/AVC video streaming. In 2007 2nd International Symposium on Wireless Pervasive Computing. IEEE, San Juan, Puerto Rico. DOI: DOI: https://doi.org/10.1109/ISWPC.2007.342629Google ScholarGoogle Scholar
  39. [39] Scott M. J., Guntuku S. C., Lin W., and Ghinea G.. 2016. Do personality and culture influence perceived video quality and enjoyment?IEEE Transactions on Multimedia 18, 9 (2016), 17961807. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. [40] Seufert M.. 2019. Fundamental advantages of considering Quality of Experience distributions over mean opinion scores. In 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX). IEEE, Berlin, Germany, 16. DOI: DOI: https://doi.org/10.1109/QoMEX.2019.8743296Google ScholarGoogle ScholarCross RefCross Ref
  41. [41] Sheikh H. R. and Bovik A. C.. 2006. Image information and visual quality. IEEE Transactions on Image Processing 15, 2 (Feb 2006), 430444. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. [42] Streijl Robert C., Winkler Stefan, and Hands David S.. 2016. Mean opinion score (MOS) revisited: Methods and applications, limitations and alternatives. Multimedia Systems 22, 2 (Mar 2016), 213227. DOI: DOI: https://doi.org/10.1007/s00530-014-0446-1 Google ScholarGoogle ScholarCross RefCross Ref
  43. [43] Sue Stevens. 2007. Test distance vision using a Snellen chart.Community Eye Health 20, 63 (2007), 52.Google ScholarGoogle Scholar
  44. [44] Varga Domonkos. 2019. No-reference video quality assessment based on the temporal pooling of deep features. Neural Processing Letters 50, 3 (12 Apr 2019), 25952608. DOI: DOI: https://doi.org/10.1007/s11063-019-10036-6Google ScholarGoogle ScholarCross RefCross Ref
  45. [45] Varga Domonkos, Saupe Dietmar, and Szirányi Tamás. 2018. Deeprn: A content preserving deep architecture for blind image quality assessment. In 2018 IEEE International Conference on Multimedia and Expo (ICME). IEEE, San Diego, CA, USA, 16. DOI: DOI: https://doi.org/10.1109/ICME.2018.8486528Google ScholarGoogle ScholarCross RefCross Ref
  46. [46] VQEG. 2010. Report on the Validation of Video Quality Models for High Definition Video Content (v. 2.0). (Jun 2010). http://bit.ly/2Z7GWDIGoogle ScholarGoogle Scholar
  47. [47] Wang Z., Simoncelli E. P., and Bovik A. C.. 2003. Multiscale structural similarity for image quality assessment. In The Thirty-Seventh Asilomar Conference on Signals, Systems & Computers, Vol. 2. IEEE, Pacific Grove, CA, USA, 13981402. DOI: DOI: https://doi.org/10.1109/ACSSC.2003.1292216Google ScholarGoogle ScholarCross RefCross Ref
  48. [48] Wechsung Ina, Schulz Matthias, Engelbrecht Klaus-Peter, Niemann Julia, and Möller Sebastian. 2011. All users are (not) equal - the influence of user characteristics on perceived quality, modality choice and performance. In Proc. IWSDS Workshop on Paralinguistic Information and its Integration in Spoken Dialogue Systems. Springer, New York, NY, USA, 175186. DOI: DOI: https://doi.org/10.1007/978-1-4614-1335-6_19Google ScholarGoogle Scholar
  49. [49] Wei Xiaochi, Huang Heyan, Nie Liqiang, Feng Fuli, Hong Richang, and Chua Tat-Seng. 2018. Quality matters: Assessing cQA pair quality via transductive multi-view learning. In Proc. of International Joint Conference on Artificial Intelligence (IJCAI). Stockholm, Sweden, 44824488. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. [50] Winkler S. and Mohandas P.. 2008. The evolution of video quality measurement: From PSNR to hybrid metrics. IEEE Transactions on Broadcasting 54, 3 (Sep 2008), 660668. DOI: DOI: https://doi.org/10.1109/TBC.2008.2000733Google ScholarGoogle ScholarCross RefCross Ref
  51. [51] Xu L., Lin W., Ma L., Zhang Y., Fang Y., Ngan K. N., Li S., and Yan Y.. 2016. Free-energy principle inspired video quality metric and its use in video coding. IEEE Transactions on Multimedia 18, 4 (Feb 2016), 590602.Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. [52] Yan B., Bare B., and Tan W.. 2019. Naturalness-aware deep no-reference image quality assessment. IEEE Transactions on Multimedia 21, 10 (Mar 2019), 26032615.Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. [53] Yang Wei, Wang Kuanquan, and Zuo Wangmeng. 2012. Neighborhood component feature selection for high-dimensional data. Journal of Computers 7 (Jan 2012), 161168.Google ScholarGoogle ScholarCross RefCross Ref
  54. [54] You J. and Korhonen J.. 2019. Deep neural networks for no-reference video quality assessment. In IEEE International Conference on Image Processing (ICIP). IEEE, Taipei, Taiwan, 23492353.Google ScholarGoogle ScholarCross RefCross Ref
  55. [55] Zeng Hui, Zhang Lei, and Bovik Alan C. 2017. A probabilistic quality representation approach to deep blind image quality prediction. (2017). arXiv:arXiv:1708.08190v2Google ScholarGoogle Scholar
  56. [56] Zhang Yu, Gao Xinbo, He Lihuo, Lu Wen, and He Ran. 2020. Objective video quality assessment combining transfer learning with CNN. IEEE Transactions on Neural Networks and Learning Systems 31, 8 (2020), 27162730. DOI: DOI: https://doi.org/10.1109/TNNLS.2018.2890310Google ScholarGoogle ScholarCross RefCross Ref
  57. [57] Wang Zhou, Bovik A. C., Sheikh H. R., and Simoncelli E. P.. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (Apr 2004), 600612. DOI: DOI: https://doi.org/10.1109/TIP.2003.819861 Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. [58] Zhu Yi, Guntuku Sharath Chandra, Lin Weisi, Ghinea Gheorghita, and Redi Judith A.. 2018. Measuring individual video QoE: A survey, and proposal for future directions using social media. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 14, 2s (2018), 124. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Mimicking Individual Media Quality Perception with Neural Network based Artificial Observers

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Multimedia Computing, Communications, and Applications
      ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 18, Issue 1
      January 2022
      517 pages
      ISSN:1551-6857
      EISSN:1551-6865
      DOI:10.1145/3505205
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 27 January 2022
      • Accepted: 1 May 2021
      • Revised: 1 April 2021
      • Received: 1 November 2020
      Published in tomm Volume 18, Issue 1

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Refereed

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!