skip to main content
research-article

A Bayesian Quality-of-Experience Model for Adaptive Streaming Videos

Published:11 February 2023Publication History
Skip Abstract Section

Abstract

The fundamental conflict between the enormous space of adaptive streaming videos and the limited capacity for subjective experiment casts significant challenges to objective Quality-of-Experience (QoE) prediction. Existing objective QoE models either employ pre-defined parametrization or exhibit complex functional form, achieving limited generalization capability in diverse streaming environments. In this study, we propose an objective QoE model, namely, the Bayesian streaming quality index (BSQI), to integrate prior knowledge on the human visual system and human annotated data in a principled way. By analyzing the subjective characteristics towards streaming videos from a corpus of subjective studies, we show that a family of QoE functions lies in a convex set. Using a variant of projected gradient descent, we optimize the objective QoE model over a database of training videos. The proposed BSQI demonstrates strong prediction accuracy in a broad range of streaming conditions, evident by state-of-the-art performance on four publicly available benchmark datasets and a novel analysis-by-synthesis visual experiment.

REFERENCES

  1. [1] Bampis C. G. and Bovik A. C.. 2017. Learning to predict streaming video QoE: Distortions, rebuffering and memory. ArXiv preprint arXiv:1703.00633 (Mar.2017).Google ScholarGoogle Scholar
  2. [2] Bampis C. G, Bovik A. C., and Li Z.. 2018. A simple prediction fusion improves data-driven full-reference video quality assessment models. In Picture Coding Symposium. IEEE, 298302.Google ScholarGoogle Scholar
  3. [3] Bampis C. G., Li Z., and Bovik A. C.. 2017. Continuous prediction of streaming video QoE using dynamic networks. IEEE Sig. Process. Lett. 24, 7 (Jul.2017), 10831087.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Bampis C. G., Li Z., Katsavounidis I., Huang T. Y., Ekanadham C., and Bovik A. C.. 2018. Towards perceptually optimized end-to-end adaptive video streaming. ArXiv preprint arXiv:1808.03898 (Aug.2018).Google ScholarGoogle Scholar
  5. [5] Bampis C. G., Li Z., Moorthy A. K., Katsavounidis I., Aaron A., and Bovik A. C.. 2017. Study of temporal effects on subjective video quality of experience. IEEE Trans. Image Process. 26, 11 (Nov.2017), 52175231.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. [6] Bentaleb A., Begen A. C., and Zimmermann R.. 2016. SDNDASH: Improving QoE of HTTP adaptive streaming using software defined networking. In ACM International Conference on Multimedia. ACM, 12961305.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. [7] Bishop C.. 2006. Pattern Recognition and Machine Learning. Springer-Verlag, Berlin. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. [8] Boyd S., Parikh N., Chu E., Peleato B., and Eckstein J.. 2011. Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends® Mach. Learn. 3, 1 (July2011), 1122.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Breiman L.. 2001. Random forests. Mach. Learn. 45, 1 (Oct.2001), 532.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. [10] VNI Cisco Mobile. 2017. Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update, 2016–2021 White Paper. Retrieved from https://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/mobile-white-paper-c11-520862.html.Google ScholarGoogle Scholar
  11. [11] Cock J. De, Li Z., Manohara M., and Aaron A.. 2016. Complexity-based consistent-quality encoding in the cloud. In IEEE International Conference on Image Processing.IEEE, 14841488.Google ScholarGoogle Scholar
  12. [12] Dobrian F., Sekar V., Awan A., Stoica I., Joseph D., Ganjam A., Zhan J., and Zhang H.. 2011. Understanding the impact of video quality on user engagement. ACM SIGCOMM Comput. Commun. Rev. 41, 4 (Aug.2011), 362373.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. [13] Domingos Pedro. 2000. A unified bias-variance decomposition. In International Conference on Machine Learning. 231238.Google ScholarGoogle Scholar
  14. [14] Duanmu Z., Chen D., Li Z., Liu W., Wang Z., Wang Y., and Gao W.. 2019. Waterloo Streaming Quality-of-Experience Database IV. Retrieved from http://ece.uwaterloo.ca/zduanmu/waterloosqoe4.Google ScholarGoogle Scholar
  15. [15] Duanmu Z., Liu W., Chen D., Li Z., Wang Z., Wang Y., and Gao W.. 2019. Pairwise Comparison of Objective QoE Models via Analysis-by-synthesis. Retrieved from http://ivc.uwaterloo.ca/research/KSQI/demo/.Google ScholarGoogle Scholar
  16. [16] Duanmu Z., Liu W., Li Z., Chen D., Wang Z., Wang Y., and Gao W.. 2020. Assessing the quality-of-experience of adaptive bitrate video streaming. arXiv preprint arXiv:2008.08804 (2020).Google ScholarGoogle Scholar
  17. [17] Duanmu Z., Liu W., Wang Z., and Wang Z.. 2021. To Appear. Quantifying visual image quality: A Bayesian view. Ann. Rev. Vis. Sci. (Sep. 2021, To Appear).Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Duanmu Z., Ma K., and Wang Z.. 2017. Quality-of-experience of adaptive video streaming: Exploring the space of adaptations. In ACM International Conference on Multimedia. ACM, 17521760.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. [19] Duanmu Z., Ma K., and Wang Z.. 2018. Quality-of-experience for adaptive streaming videos: An expectation confirmation theory motivated approach. IEEE Trans. Image Process. 27, 12 (Dec.2018), 61356146.Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Duanmu Z., Rehman A., and Wang Z.. 2018. A quality-of-experience database for adaptive video streaming. IEEE Trans. Broadcast. 64, 2 (June2018), 474487.Google ScholarGoogle ScholarCross RefCross Ref
  21. [21] Duanmu Z., Zeng K., Ma K., Rehman A., and Wang Z.. 2017. A quality-of-experience index for streaming video. IEEE J. Select. Topics Sig. Process. 11, 1 (Sep.2017), 154166.Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Duc Tho Nguyen, Tran Chanh Minh, Tan Phan Xuan, and Kamioka Eiji. 2019. Bidirectional LSTM for continuously predicting QoE in HTTP adaptive streaming. In International Conference on Information Science and Systems. 156160.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. [23] Encoding.com. 2016. Microsoft Smooth Streaming. Retrieved from https://www.encoding.com/microsoft-smooth-streaming/.Google ScholarGoogle Scholar
  24. [24] Eswara N., Ashique S., Panchbhai A., Chakraborty S., Sethuram H. P., Kuchi K., Kumar A., and Channappayya S. S.. 2020. Streaming video QoE modeling and prediction: A long short-term memory approach. IEEE Trans. Circ. Syst. Video Technol. 30, 3 (Mar.2020), 661673.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. [25] Gama J., Žliobaitė I., Bifet A., Pechenizkiy M., and Bouchachia A.. 2014. A survey on concept drift adaptation. ACM Comput. Surv. 46, 4 (Apr.2014), 137.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. [26] Ghadiyaram Deepti, Pan Janice, and Bovik A. C.. 2017. A subjective and objective study of stalling events in mobile streaming videos. IEEE Trans. Circ. Syst. Video Technol. 29, 1 (2017), 183197.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. [27] Ghadiyaram D., Pan J., and Bovik A. C.. 2018. Learning a continuous-time streaming video QoE model. IEEE Trans. Image Process. 27, 5 (May2018), 22572271.Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] Grafl M. and Timmerer C.. 2013. Representation switch smoothing for adaptive HTTP streaming. In IEEE International Workshop on Perceptual Quality of Systems. ISCA/DEGA, 178183.Google ScholarGoogle Scholar
  29. [29] Hagan M. T., Demuth H. B., Beale M. H., and Jesús R. De. 1996. Neural Network Design. Vol. 20. PWS Pub. Boston.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. [30] Hornik K., Stinchcombe M., and White H.. 1989. Multilayer feedforward networks are universal approximators. Neural Netw. 2, 5 (Jan.1989), 359366.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. [31] Hoßfeld T., Schatz R., Biersack E., and Plissonneau L.. 2013. Internet video delivery in YouTube: From traffic measurements to quality of experience. In Data Traffic Monitoring and Analysis. Springer, Berlin, 264301.Google ScholarGoogle ScholarCross RefCross Ref
  32. [32] Hoßfeld T., Seufert M., Hirth M., Zinner T., Tran-Gia P., and Schatz R.. 2011. Quantification of YouTube QoE via crowdsourcing. In IEEE International Symosium on Multimedia. IEEE, 494499.Google ScholarGoogle Scholar
  33. [33] Huang T., Zhou C., Zhang R., Wu C., Yao X., and Sun L.. 2019. Comyco: Quality-aware adaptive video streaming via imitation learning. In ACM International Conference on Multimedia. 429437.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. [34] Huang T. Y., Johari R., McKeown N., Trunnell M., and Watson M.. 2015. A buffer-based approach to rate adaptation: Evidence from a large video streaming service. ACM SIGCOMM Comput. Commun. Rev. 44, 4 (Feb.2015), 187198.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. [35] Ishii I., Tatebe T., Gu Q., Moriue Y., Takaki T., and Tajima K.. 2010. 2000 fps real-time vision system with high-frame-rate video recording. In IEEE International Conference on Robotics and Automation. IEEE, 15361541.Google ScholarGoogle ScholarCross RefCross Ref
  36. [36] BT.500-12 ITU-R. 1993. Recommendation: Methodology for the Subjective Assessment of the Quality of Television Pictures.Google ScholarGoogle Scholar
  37. [37] Jiang J., Sekar V., and Zhang H.. 2014. Improving fairness, efficiency, and stability in HTTP-based adaptive video streaming with FESTIVE. IEEE/ACM Trans. Netw. 22, 1 (Feb.2014), 326340.Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Kang S. B., Uyttendaele M., Winder S., and Szeliski R.. 2003. High dynamic range video. ACM Trans. Graph. 22, 3 (July2003), 319325.Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. [39] Kim H. J., Yun D. G., Kim H., Cho K. S., and Choi S. G.. 2012. QoE assessment model for video streaming service using QoS parameters in wired-wireless network. In IEEE International Conference on Advanced Communications Technology. 459464.Google ScholarGoogle Scholar
  40. [40] Feuvre J. Le, Concolato C., and Moissinac J.. 2007. GPAC: Open source multimedia framework. In ACM International Conference on Multimedia. ACM, 10091012.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. [41] Lederer S.. 2015. Optimal Adaptive Streaming Formats MPEG-DASH & HLS Segment Length. Retrieved from https://bitmovin.com/mpeg-dash-hls-segment-length/.Google ScholarGoogle Scholar
  42. [42] Li Z., Aaron A., Katsavounidis L., Moorthy A., and Manohara M.. 2016. Toward a Practical Perceptual Video Quality Metric. Retrieved from http://techblog.netflix.com/2016/06/toward-practical-perceptual-video.html.Google ScholarGoogle Scholar
  43. [43] Li Z., Duanmu Z., Liu W., and Wang Z.. To Appear. AVC, HEVC, VP9, AVS2, or AV1? - A comparative study of state-of-the-art video encoders on 4K videos. In International Conference on Image Analysis and Recognition. AIMI.Google ScholarGoogle Scholar
  44. [44] Li Z., Zhu X., Gahm J., Pan R., Hu H., Begen A. C., and Oran D.. 2014. Probe and adapt: Rate adaptation for HTTP video streaming at scale. IEEE J. Select. Areas Commun. 32, 4 (Apr.2014), 719733.Google ScholarGoogle ScholarCross RefCross Ref
  45. [45] Liu W., Duanmu Z., and Wang Z.. 2018. End-to-end blind quality assessment of compressed videos using deep neural networks. In ACM International Conference on Multimedia. ACM, 546554.Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. [46] Liu X., Dobrian F., Milner H., Jiang J., Sekar V., Stoica I., and Zhang H.. 2012. A case for a coordinated internet video control plane. ACM SIGCOMM Comput. Commun. Rev. 42, 4 (Sep.2012), 359370.Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. [47] Liu Y., Dey S., Ulupinar F., Luby M., and Mao Y.. 2015. Deriving and validating user experience model for DASH video streaming. IEEE Trans. Broadcast. 61, 4 (Dec.2015), 651665.Google ScholarGoogle ScholarCross RefCross Ref
  48. [48] Ma K., Duanmu Z., Wang Z., Wu Q., Liu W., Yong H., Li H., and Zhang L.. 2019. Group maximum differentiation competition: Model comparison with few samples. IEEE Trans. Patt. Anal. Mach. Intell. (2019), DOI: 10.1109/TPAMI.2018.2889948Google ScholarGoogle Scholar
  49. [49] Mao H., Netravali R., and Alizadeh M.. 2017. Neural adaptive video streaming with pensieve. In ACM SIGCOMM. ACM, 197210.Google ScholarGoogle Scholar
  50. [50] Mishra A. M.. 2001. Quality of Service in Communications Networks. Wiley.Google ScholarGoogle Scholar
  51. [51] Mok R. K., Luo X., Chan E. W., and Chang R. K.. 2012. QDASH: A QoE-aware DASH system. In ACM Conference on Multimedia Systems. ACM, 1122.Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. [52] Moorthy A. K., Choi L. K., Bovik A. C., and Veciana G. De. 2012. Video quality assessment on mobile devices: Subjective, behavioral and objective studies. IEEE J. Select. Topics Sig. Process. 6, 6 (Oct.2012), 652671.Google ScholarGoogle ScholarCross RefCross Ref
  53. [53] Nasiri R. M., Wang J., Rehman A., Wang S., and Wang Z.. 2015. Perceptual quality assessment of high frame rate video. In IEEE International Conference on Multimedia and Signal Processing. IEEE, 16.Google ScholarGoogle ScholarCross RefCross Ref
  54. [54] Inc. Netflix2015. Per-title Encode Optimization. Retrieved from http://techblog.netflix.com/2015/12/per-title-encode-optimization.html.Google ScholarGoogle Scholar
  55. [55] Ng R., Levoy M., Brédif M., Duval G., Horowitz M., and Hanrahan P.. 2005. Light Field Photography with a Hand-held Plenoptic Camera. Computer Science Technical Report.Google ScholarGoogle Scholar
  56. [56] Ni P., Eg R., Eichhorn A., Griwodz C., and Halvorsen P.. 2011. Flicker effects in adaptive video streaming to handheld devices. In ACM International Conference on Multimedia. ACM, 463472.Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. [57] Nightingale J., Wang Q., Grecos C., and Goma S.. 2014. The impact of network impairment on quality of experience (QoE) in H.265/HEVC video streaming. IEEE Trans. Consum. Electron. 60, 2 (2014), 242250.Google ScholarGoogle ScholarCross RefCross Ref
  58. [58] Oliver R. L.. 1980. A cognitive model of the antecedents and consequences of satisfaction decisions. J. Market. Res. 17, 4 (Nov.1980), 460469.Google ScholarGoogle ScholarCross RefCross Ref
  59. [59] P.1203 ITU-T. 2017. Parametric Bitstream-based Quality Assessment of Progressive Download and Adaptive Audiovisual Streaming Services Over Reliable Transport. Retrieved from https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-P.1203-201710-I!!PDF-E&type=items.Google ScholarGoogle Scholar
  60. [60] Pastrana-Vidal R., Gicquel J. C., Colomes C., and Cherifi H.. 2004. Sporadic frame dropping impact on quality perception. In Human Vision and Electronic Imaging IX. SPIE, 182194.Google ScholarGoogle Scholar
  61. [61] Paudyal P., Battisti F., and Carli M.. 2016. Impact of video content and transmission impairments on Quality of Experience. Multim. Tools Applic. 75, 23 (2016), 1646116485.Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. [62] Ponomarenko N., Jin L., Ieremeiev O., Lukin V., Egiazarian K., Astola J., Vozel B., Chehdi K., Carli M., and Battisti F.. 2015. Image database TID2013: Peculiarities, results and perspectives. Sig. Process.: Image Commun. 30 (Jan.2015), 5777.Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. [63] Rehman A. and Wang Z.. 2013. Perceptual experience of time-varying video quality. In IEEE International Workshop Quality of Multimedia Experience. IEEE, 218223.Google ScholarGoogle Scholar
  64. [64] Rehman A., Zeng K., and Wang Z.. 2015. Display device-adapted video quality-of-experience assessment. In SPIE. SPIE, 939406.1–939406.11.Google ScholarGoogle Scholar
  65. [65] Riiser H., Vigmostad P., Griwodz C., and Halvorsen P.. 2013. Commute path bandwidth traces from 3G networks: Analysis and applications. In ACM Conference on Multimedia Systems. ACM, 114118.Google ScholarGoogle ScholarDigital LibraryDigital Library
  66. [66] Rodríguez D. Z., Wang Z., Rosa R. L., and Bressan G.. 2014. The impact of video-quality-level switching on user quality of experience in dynamic adaptive streaming over HTTP. EURASIP J. Wirel. Commun. Netw. 2014, 1 (2014), 115.Google ScholarGoogle ScholarCross RefCross Ref
  67. [67] Sheikh H. R., Sabir M. F., and Bovik A. C.. 2006. A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Trans. Image Process. 15, 11 (Nov.2006), 34403451.Google ScholarGoogle ScholarDigital LibraryDigital Library
  68. [68] Singh K., Hadjadj-Aoul Y., and Rubino G.. 2012. Quality of experience estimation for adaptive HTTP/TCP video streaming using H.264/AVC. In IEEE Consumer Communications & Networking Conference. IEEE, 16.Google ScholarGoogle ScholarCross RefCross Ref
  69. [69] Spiteri K., Urgaonkar R., and Sitaraman R. K.. 2016. BOLA: Near-optimal bitrate adaptation for online videos. In IEEE International Conference on Computer Communications.IEEE, 19.Google ScholarGoogle ScholarDigital LibraryDigital Library
  70. [70] Stellato B., Banjac G., Goulart P., Bemporad A., and Boyd S.. 2017. OSQP: An operator splitting solver for quadratic programs. ArXiv preprint arXiv:1711.08013 (Nov.2017).Google ScholarGoogle Scholar
  71. [71] Toni L., Aparicio-Pardo R., Pires K., Simon W., Blanc A., and Frossard P.. 2015. Optimal selection of adaptive streaming representations. ACM Trans. Multim. Comput., Commun., Applic. 11, 2s (Feb.2015), 143.Google ScholarGoogle ScholarDigital LibraryDigital Library
  72. [72] Tsukida K. and Gupta M. R.. 2011. How to Analyze Paired Comparison Data. University of Washington, Technical Report. UWEETR-2011-0004.Google ScholarGoogle Scholar
  73. [73] Hooft J. van der, Petrangeli S., Wauters T., Huysegems R., Alface P. R., Bostoen T., and Turck F. De. 2016. HTTP/2-based adaptive streaming of HEVC video over 4G/LTE networks. IEEE Commun. Lett. 20, 11 (Aug.2016), 21772180.Google ScholarGoogle ScholarCross RefCross Ref
  74. [74] VQEG. 2000. Final Report from the Video Quality Experts Group on the Validation of Objective Models of Video Quality Assessment. Technical Report. Retrieved from http://www.vqeg.org/.Google ScholarGoogle Scholar
  75. [75] Wang Z.. 2001. Internet QoS: Architectures and Mechanisms for Quality of Service. Morgan Kaufmann.Google ScholarGoogle Scholar
  76. [76] Wang Z., Bovik A. C., Sheikh H. R., and Simoncelli E. P.. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13, 4 (Apr.2004), 600612.Google ScholarGoogle ScholarDigital LibraryDigital Library
  77. [77] Wang Z. and Bovik A. C.. 2009. Mean squared error: Love it or leave it? A new look at signal fidelity measures. IEEE Sig. Process. Mag. 26, 1 (Jan.2009), 98117.Google ScholarGoogle ScholarCross RefCross Ref
  78. [78] Wang Z., Duanmu Z., Rehman A., and Zeng K.. 2017. Method and system for automatic user quality-of-experience measurement of streaming video. US Patent WO/2017/152274.Google ScholarGoogle Scholar
  79. [79] Wang Z. and Rehman A.. 2017. Begin with the end in mind: A unified end-to-end quality-of-experience monitoring, optimization and management framework. In SMPTE Annual Technical Conference and Exhibition. SMPTE, 111.Google ScholarGoogle ScholarCross RefCross Ref
  80. [80] Wang Z. and Simoncelli E. P.. 2008. Maximum differentiation (MAD) competition: A methodology for comparing computational models of perceptual quantities. J. Vis. 8, 12 (2008), 88.Google ScholarGoogle ScholarCross RefCross Ref
  81. [81] Wang Z., Zeng K., and Rehman A.. 2016. Method and system for smart adaptive video streaming driven by perceptual quality-of-experience estimations. US Patent WO/2016/123721.Google ScholarGoogle Scholar
  82. [82] Watanabe K., Okamoto J., and Kurita T.. 2007. Objective video quality assessment method for evaluating effects of freeze distortion in arbitrary video scenes. In Image Quality and System Performance IV, Vol. 64940P. SPIE, 18.Google ScholarGoogle Scholar
  83. [83] Xue J., Zhang D., Yu H., and Chen C. W.. 2014. Assessing quality of experience for adaptive HTTP video streaming. In IEEE International Conference on Multimedia and Expo Workshop. IEEE, 16.Google ScholarGoogle Scholar
  84. [84] Yin X., Jindal A., Sekar V., and Sinopoli B.. 2015. A control-theoretic approach for dynamic adaptive video streaming over HTTP. ACM SIGCOMM Comput. Commun. Rev. 45, 4 (Apr.2015), 325338.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. A Bayesian Quality-of-Experience Model for Adaptive Streaming Videos

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Multimedia Computing, Communications, and Applications
          ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 18, Issue 3s
          October 2022
          381 pages
          ISSN:1551-6857
          EISSN:1551-6865
          DOI:10.1145/3567476
          • Editor:
          • Abdulmotaleb El Saddik
          Issue’s Table of Contents

          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 11 February 2023
          • Online AM: 14 July 2022
          • Accepted: 13 October 2021
          • Revised: 11 September 2021
          • Received: 7 April 2021
          Published in tomm Volume 18, Issue 3s

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Refereed
        • Article Metrics

          • Downloads (Last 12 months)241
          • Downloads (Last 6 weeks)17

          Other Metrics

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Full Text

        View this article in Full Text.

        View Full Text

        HTML Format

        View this article in HTML Format .

        View HTML Format
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!