skip to main content
research-article

A new methodology to derive objective quality assessment metrics for scalable multiview 3D video coding

Published:16 October 2012Publication History
Skip Abstract Section

Abstract

With the growing demand for 3D video, efforts are underway to incorporate it in the next generation of broadcast and streaming applications and standards. 3D video is currently available in games, entertainment, education, security, and surveillance applications. A typical scenario for multiview 3D consists of several 3D video sequences captured simultaneously from the same scene with the help of multiple cameras from different positions and through different angles. Multiview video coding provides a compact representation of these multiple views by exploiting the large amount of inter-view statistical dependencies. One of the major challenges in this field is how to transmit the large amount of data of a multiview sequence over error prone channels to heterogeneous mobile devices with different bandwidth, resolution, and processing/battery power, while maintaining a high visual quality. Scalable Multiview 3D Video Coding (SMVC) is one of the methods to address this challenge; however, the evaluation of the overall visual quality of the resulting scaled-down video requires a new objective perceptual quality measure specifically designed for scalable multiview 3D video. Although several subjective and objective quality assessment methods have been proposed for multiview 3D sequences, no comparable attempt has been made for quality assessment of scalable multiview 3D video. In this article, we propose a new methodology to build suitable objective quality assessment metrics for different scalable modalities in multiview 3D video. Our proposed methodology considers the importance of each layer and its content as a quality of experience factor in the overall quality. Furthermore, in addition to the quality of each layer, the concept of disparity between layers (inter-layer disparity) and disparity between the units of each layer (intra-layer disparity) is considered as an effective feature to evaluate overall perceived quality more accurately. Simulation results indicate that by using this methodology, more efficient objective quality assessment metrics can be introduced for each multiview 3D video scalable modalities.

References

  1. Barkowsky, M., Wang, K., Cousseau, R., Brunnstrom, K., Olsson, R., and Le Callet, P. 2010. Subjective quality assessment of error concealment strategies for 3DTV in the presence of asymmetric transmission errors. In Proceedings of the 18th International Packet Video Workshop (PV). 193--200.Google ScholarGoogle ScholarCross RefCross Ref
  2. Do, L., Zinger, S., and de With, P. H. N. 2010. Conversion of free-viewpoint 3D multiview video for stereoscopic displays. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME). 1730--1734.Google ScholarGoogle Scholar
  3. Dodgson, N. A. 2005. Autostereoscopic 3D displays. In Computer Journal Published by the IEEE Computer Society 38, 9, 31--36. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Feghali, R., Speranza, F., Wang, D., and Vincent, A. 2007. Video quality metric for bit rate control via joint adjustment of quantization and frame rate. IEEE Transactions on Broadcasting. 53, 1, 441--446.Google ScholarGoogle ScholarCross RefCross Ref
  5. Grois, D., Kaminsky, E., and Hadar, O. 2010. Dynamically adjustable and scalable ROI video coding. In Proceedings of the IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB). 1--5.Google ScholarGoogle Scholar
  6. Gutierrez, J., Perez, P., Jaureguizar, F., Cabrera, J., and Garcia, N. 2011. Subjective assessment of the impact of transmission errors in 3DTV compared to HDTV. In Proceedings of the 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON). Antalya, Turkey. 1--4.Google ScholarGoogle Scholar
  7. Ha, K. and Kim, M. 2011. A perceptual quality assessment metric using temporal complexity and disparity information for stereoscopic video. In Proceedings of the 18th IEEE International Conference on Image Processing (ICIP), Brussels, Belgium. 2525--2528.Google ScholarGoogle Scholar
  8. Hewage, C. T. E. R., Worrall, S. T., Dogan, S., and Kondoz, A. M. 2008. Prediction of stereoscopic video quality using objective quality models of 2-D video. Electron. Lett. 44, 16, 963--965.Google ScholarGoogle ScholarCross RefCross Ref
  9. Hewage, C. T. E. R. and Martini, M. G. 2010a. Reduced-reference quality evaluation for compressed depth maps associated with colour plus depth 3D video. In Proceedings of the 17th IEEE International Conference on Image Processing (ICIP). 4017--4020.Google ScholarGoogle Scholar
  10. Hewage, C. T. E. R. and Martini, M. G. 2010b. Reduced-reference quality metric for 3D depth map transmission. In Proceedings of the 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON). 1--4.Google ScholarGoogle Scholar
  11. Hewage, C. T. E. R. and Martini, M. G. 2011. Reduced-Reference quality assessment for 3D video compression and transmission. IEEE Trans. Consumer Electron. 57, 3, 1185--1193.Google ScholarGoogle ScholarCross RefCross Ref
  12. Ho, Y. S. and Oh, K. J. 2007. Overview of multi-view video coding. In Proceedings of the 14th EURASIP Conference Focused on Speech and Image Processing Metromedia Communications and Service. 5--12Google ScholarGoogle Scholar
  13. ISO/IES JTC 1/SC 29/WG11.2005. Introduction to multi-view video coding. http://www.itscj.ipsj or jp./sc29/.Google ScholarGoogle Scholar
  14. ITU-R Recommendation BT.500-10. 2000. Methodology for the subjective assessment of the quality of television picture. http://www.itu.int/rec/R-REC-BT.500/en.Google ScholarGoogle Scholar
  15. ITU-R BT.812. 1994. Subjective assessment of the quality of alphanumeric and graphic pictures in teletext and similar services.Google ScholarGoogle Scholar
  16. ITU-T Rec H.264/ISO/IEC, Annex H. 2010. Multiview video coding. http://www.itu.int/rec/T-REC-H264.Google ScholarGoogle Scholar
  17. Jia, H., Gao, W., and Lu, Y. 2003. Stereoscopic video coding based on global displacement compensated prediction. In Proceedings of the Information, Communications and Signal Processing and the Pacific Rim Conference on Multimedia. 61--65.Google ScholarGoogle Scholar
  18. Jin, L., Boev, A., Gotchev, A., Egiazarian, K. 2011. 3D-DCT based perceptual quality assessment of stereo video. In Proceedings of the 18th IEEE International Conference on Image Processing (ICIP). 2521--2524.Google ScholarGoogle ScholarCross RefCross Ref
  19. Joveluro, P., Malekmohamadi, H., Fernando., W. A. C., and Kondoz, A. M. 2010. Perceptual video quality metric for 3D video quality assessment. In Proceedings of The True Vision - Capture, Transmission and Display of 3D Video,Tampere, Finland. 1--4.Google ScholarGoogle Scholar
  20. Kim, D., Min, D., Oh, J., Jeon, S., and Sohn, K. 2009. Depth map quality metric for three-dimensional video. In Proceedings of SPIE 7237, 723719.Google ScholarGoogle Scholar
  21. Koumaras, H., Kourtis, A., Lin, C.-H., and Shieh, C.-K. 2007. A theoretical framework for end-to-end video Quality Prediction of MPEG-based Sequences. In 3rd International Conference onNetworking and Services, ICNS. Athens, Greece. 62. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Leon, G., Kalva, H., and Furht, B. 2008. 3D video quality evaluation with depth quality variations. In Proceedings of the 3DTV Conference: The True Vision-Capture, Transmission and Display of 3D Video. 301--304.Google ScholarGoogle Scholar
  23. Li, C., Yang, X., Chu, B., Lu, W., and Pang, L. 2010. A new image fusion quality assessment method based on contourlet and SSIM. In Proceedings of the 3rd IEEE International Conference on Computer Science and Information Technology. 246--249.Google ScholarGoogle Scholar
  24. Li, Q. and Wang Z. 2009. Reduced-Reference image quality assessment using divisive normalization-based image representation. IEEE J. Select. Topics Sig. Proce. 3, 2, 202--211.Google ScholarGoogle ScholarCross RefCross Ref
  25. Lin, W. and Kuo, C.-C. J. 2011. Perceptual visual quality metrics: a survey. J. Visual Commun. Image Represent. 22, 4, 297--312. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Liyuan, X., Junyong, Y., Ebrahimi, T., and Perkis, A. 2011. Objective metrics for quality of experience in stereoscopic images. In Proceedings of the 18th IEEE International Conference on Image Processing (ICIP). 3105--3108.Google ScholarGoogle Scholar
  27. Martini, M. G. and Hewage, C. T. E. R. 2011. Objective quality assessment for 3-D video delivery. In Proceedings of the IEEE ComSoc MMTC E-letter 6, 8, 8--22.Google ScholarGoogle Scholar
  28. Maalouf, A. and Larabi, M.-C. 2010. A no-reference color video quality metric based on a 3D multispectral wavelet transform. In Proceedings of the International Workshop on Quality of Multimedia Experience (QoMEX). 11--16.Google ScholarGoogle Scholar
  29. Merl, ftp://ftp.merl.com/pub/avetro/mvc-testseq/stereo-interlaced/420/. (Last accessed 1/12).Google ScholarGoogle Scholar
  30. Micallef, B. W., Debono, C. J., and Farrugia, R. A. 2010. Exploiting depth information for fast multiview video coding. In Proceedings of the Picture Coding Symposium (PCS). 38--41.Google ScholarGoogle Scholar
  31. Mittal, A., Moorthy, A. K., Ghosh, J., and Bovik, A.C. 2011. Algorithmic assessment of 3D quality of experience for images and videos. In Proceedings of the Digital Signal Processing and IEEE Signal Processing Education Workshop. 338--343.Google ScholarGoogle Scholar
  32. Muller, K., Merkle, P., and Wiegand, T. 2011. 3-D video representation using depth maps. In Proc. IEEE. 99, 4, 643--656.Google ScholarGoogle ScholarCross RefCross Ref
  33. Nur, G., Arachchi, H. K., Dogan, S., and Kondoz, A.M. 2011. Extended VQM model for predicting 3D video quality considering ambient illumination context . In Proceedings of the 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON). 1--4.Google ScholarGoogle Scholar
  34. Ozbek, N., Ertan, G., and Karakus, O. 2011. Interactive quality assessment for asymmetric coding of 3D video. In Proceedings of the 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON). 1--4.Google ScholarGoogle Scholar
  35. Pandit, S. P., Chen, Y., and Ye, S. 2008. Text of ISO/IEC 14496- 5:2001/PDAM 15 Reference software for multiview video coding, ISO/IEC JTC1/SC29/WG11 MPEG2008/W9974.Google ScholarGoogle Scholar
  36. Rodriguez, E. J., Nunome, T., and Tasaka, S. 2009. Assessment of user behavior and QoE in multiview video and audio IP transmission. In Proceedings of the 15th Asia-Pacific Conference on Communications, APCC. 790--793. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Roodaki, H., Hashemi, M. R., and Shirmohammadi, S. 2011. A new scalable multiview video coding configuration for mobile applications. In Proceedings of IEEE International Conference on Multimedia & Expo. 1--4. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Saygili, G., Gurler, C. G., and Tekalp, A. M. 2010. Quality assessment of asymmetric stereo video coding. In Proceedings of the 17th IEEE International Conference on Image Processing (ICIP). 4009--4012.Google ScholarGoogle Scholar
  39. Schwarz, H., Marpe, D., and Wiegand, T. 2007. Overview of the scalable video coding extension of the H.264/AVC standard. IEEE Trans. Circ. Syst. Video Techn. 17, 9, 1103--1120. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Shao, H., Cao, X., and Er, G. 2009. Objective quality of depth image based rendering in 3DTV system. In Proceedings of the 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video. 1--4.Google ScholarGoogle Scholar
  41. Shimizu, Sh., Kitahara, M., Kimata, H., Kamikura, K., and Yashima, Y. 2007. View scalable multiview video coding using 3-D warping with depth map. IEEE Trans. Circ. Syst. Video Techn. 17, 11, 1485--1495. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Tanimoto, M., Fujii, T., Suzuki, K., Fukushima, N., and Mori, Y. 2008. Reference softwares for depth estimation and view synthesis. ISO/IEC JTC1/SC29/WG11 MPEG2008/M15377.Google ScholarGoogle Scholar
  43. Tanimoto, M. 2009. Overview of FTV (free-viewpoint television), In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), 1552--1553. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Tanimoto Laboratory. 2012. http://www.tanimoto.nuee.nagoya-u.ac.jp/~fukushima/mpegftv/yuv/. (Last accessed 1/12).Google ScholarGoogle Scholar
  45. Umar, A. S., Swash, R. M., and Sadka, A. H. 2011. Subjective quality assessment of 3D videos. In Proceedings of the IEEE Africon, Livingstone. 1--6.Google ScholarGoogle Scholar
  46. Vetro, A., Tourapis, A. M., Muller, K., and Chen, T. 2011a. 3D-TV content storage and transmission. Trans. Broadcast. 57, 2, 384--394.Google ScholarGoogle ScholarCross RefCross Ref
  47. Vetro, A., Wiegand, T., and Sullivan, G. J. 2011b. Overview of the stereo and multiview video coding extensions of the H.264/MPEG-4 AVC standard. Proc. IEEE. 99, 4, 626--642.Google ScholarGoogle ScholarCross RefCross Ref
  48. Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. 2004. Image Quality Assessment: from Error Visibility to Structural Similarity. Ieee Trans. Image Process. 13, 4, 600--612. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Wong, K.-M., PO, L.-M., Cheung, K.-W., Ng, K.-H., and Xu, X. 2011. Stretching, compression and shearing disparity compensated prediction techniques for stereo and multiview video coding. In Proceedings of The IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 841--844.Google ScholarGoogle Scholar
  50. Yamagishi, K., Karam, L., Okamoto, J., and Hayashi, T. 2011. Subjective characteristics for stereoscopic high definition video. In Proceedings of the 3rd International Workshop on Quality of Multimedia Experience (QOME). 37--42.Google ScholarGoogle Scholar
  51. Ylxla, L., yllln, C., andna, Z. 2009. Method of ensuring eye-to-eye contact in multi-view video conference. In Proceedings of the IEEE International Conference on Broadband Network & Multimedia Technology. 882--886.Google ScholarGoogle Scholar
  52. Zhang, Y., An, P., Wu, Y., and Zhang, Z. 2010. A multiview video quality assessment method based on disparity and ssim. In Proceedings of the IEEE 10th International Conference on Signal Processing (ISCP). 1044--1047.Google ScholarGoogle Scholar
  53. Zhu, Y. and Zhen, T. 2009. 3D multi-view autostereoscopic display and its key technologies. In Proceedings of the Asia-Pacific Conference on Information Processing (AS1P). 31--35. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. In Proceedings of the ACM SIGGRAPH. 600-608. http://research.microsoft.com/en-us/um/people/sbkang/3dvideodownload. (Last accessed 1/12). Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. A new methodology to derive objective quality assessment metrics for scalable multiview 3D video coding

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Multimedia Computing, Communications, and Applications
      ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 8, Issue 3s
      Special section of best papers of ACM multimedia 2011, and special section on 3D mobile multimedia
      September 2012
      173 pages
      ISSN:1551-6857
      EISSN:1551-6865
      DOI:10.1145/2348816
      Issue’s Table of Contents

      Copyright © 2012 ACM

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 16 October 2012
      • Accepted: 1 June 2012
      • Revised: 1 April 2012
      • Received: 1 January 2012
      Published in tomm Volume 8, Issue 3s

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Research
      • Refereed

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!