skip to main content
research-article

GPU-accelerated depth codec for real-time, high-quality light field reconstruction

Published:25 July 2018Publication History
Skip Abstract Section

Abstract

Pre-calculated depth information is essential for efficient light field video rendering, due to the prohibitive cost of depth estimation from color when real-time performance is desired. Standard state-of-the-art video codecs fail to satisfy such performance requirements when the amount of data to be decoded becomes too large. In this paper, we propose a depth image and video codec based on block compression, that exploits typical characteristics of depth streams, drawing inspiration from S3TC texture compression and geometric wavelets. Our codec offers very fast hardware-accelerated decoding that also allows partial extraction for view-dependent decoding. We demonstrate the effectiveness of our codec in a number of multi-view 360 degree video datasets, with quantitative analysis of storage cost, reconstruction quality and decoding performance.

References

  1. Edward H Adelson, James R Bergen, et al. 1991. The plenoptic function and the elements of early vision. (1991).Google ScholarGoogle Scholar
  2. Filippo Bannò, Paolo Simone Gasparello, Franco Tecchia, and Massimo Bergamasco. 2012. Real-time Compression of Depth Streams Through Meshification and Valence-based Encoding. In Proceedings of the 11th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry (VRCAI '12). ACM, New York, NY, USA, 263--270. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan. 2015. High-quality Streamable Free-viewpoint Video. ACM Trans. Graph. 34, 4, Article 69 (July 2015), 13 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Piotr Didyk Tobias Ritschel, Elmar Eisemann, Karol Myszkowski, and Hans-Peter Seidel. 2010. Adaptive Image-space Stereo View Synthesis.. In VMV. 299--306.Google ScholarGoogle Scholar
  5. C Goktug Gurler, Anil Aksay, Gozde Bozdagi Akar, and A Murat Tekalp. 2009. Multi-threaded architectures and benchmark tests for real-time multi-view video decoding. In Multimedia and Expo, 2009. ICME 2009. IEEE International Conference on. IEEE, 237--240. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Fabian Jäger. 2011. Contour-based segmentation and coding for depth map compression. In Visual Communications and Image Processing (VCIP), 2011 IEEE. IEEE, 1--4.Google ScholarGoogle Scholar
  7. Xiaoran Jiang, Mikaël Le Pendu, and Christine Guillemot. 2017. Light field compression using depth image based view synthesis. In Multimedia 8 Expo Workshops (ICMEW), 2017IEEE International Conference on. IEEE, 19--24.Google ScholarGoogle Scholar
  8. Vahid Kiani, Ahad Harati, and Abedin Vahedian. 2017. Planelets - A Piecewise Linear Fractional Model for Preserving Scene Geometry in Intra-Coding of Indoor Depth Images. IEEE Transactions on Image Processing 26 (2017), 590--602. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Babis Koniaris, Maggie Kosek, David Sinclair, and Kenny Mitchell. 2017. Real-time Rendering with Compressed Animated Light Fields. In Proceedings of the 43rd Graphics Interface Conference (GI '17). Canadian Human-Computer Communications Society, School of Computer Science, University of Waterloo, Waterloo, Ontario, Canada, 33--40. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Jianjun Lei, Shuai Li, Ce Zhu, Ming-Ting Sun, and Chunping Hou. 2015. Depth coding based on depth-texture motion and structure similarities. IEEE Transactions on Circuits and Systems for Video Technology 25, 2 (2015), 275--286.Google ScholarGoogle ScholarCross RefCross Ref
  11. Marc Levoy and Pat Hanrahan. 1996. Light field rendering. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. ACM, 31--42. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Shujie Liu, PoLin Lai, Dong Tian, and Chang Wen Chen. 2011. New depth coding techniques with utilization of corresponding video. IEEE Transactions on broadcasting 57, 2 (2011), 551--561.Google ScholarGoogle ScholarCross RefCross Ref
  13. Yunpeng Liu, Stephan Beck, Renfang Wang, Jin Li, Huixia Xu, Shijie Yao, Xiaopeng Tong, and Bernd Froehlich. 2015. Hybrid Lossless-Lossy Compression for Real-Time Depth-Sensor Streams in 3D Telepresence Applications.. In PCM (1) (Lecture Notes in Computer Science), Yo-Sung Ho, Jitao Sang, Yong Man Ro, Junmo Kim, and Fei Wu (Eds.), Vol. 9314. Springer, 442--452.Google ScholarGoogle Scholar
  14. Morgan McGuire, Mike Mara, Derek Nowrouzezahrai, and David Luebke. 2017. Real-time global illumination using precomputed light field probes. In Proceedings of the 21st ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games. ACM, 2. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Philipp Merkle, Aljoscha Smolic, Karsten Muller, and Thomas Wiegand. 2007. Multi-view video plus depth representation and coding. In Image Processing, 2007. ICIP 2007. IEEE International Conference on, Vol. 1. IEEE, I-201.Google ScholarGoogle ScholarCross RefCross Ref
  16. Yannick Morvan, Dirk Farin, et al. 2006. Platelet-based coding of depth maps for the transmission of multiview images. In Proceedings of SPIE, Stereoscopic Displays and Applications, Vol. 6055. 93--100.Google ScholarGoogle Scholar
  17. Karsten Muller, Heiko Schwarz, Detlev Marpe, Christian Bartnik, Sebastian Bosse, Heribert Brust, Tobias Hinz, Haricharan Lakshman, Philipp Merkle, Franz Hunn Rhee, Gerhard Tech, Martin Winken, and Thomas Wiegand. 2013. 3D High-Efficiency Video Coding for Multi-View Video and Depth Data. Trans. Img. Proc. 22, 9 (Sept. 2013), 3366--3378. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Dawid Pająk, Robert Herzog, Radosław Mantiuk, Piotr Didyk, Elmar Eisemann, Karol Myszkowski, and Kari Pulli. 2014. Perceptual depth compression for stereo applications. In Computer Graphics Forum, Vol. 33. Wiley Online Library, 195--204. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Fabrizio Pece, Jan Kautz, and Tim Weyrich. 2011. Adapting Standard Video Codecs for Depth Streaming. In Joint Virtual Reality Conference of EGVE - EuroVR, Sabine Coquillart, Anthony Steed, and Greg Welch (Eds.). The Eurographics Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Jeff Pool, Anselmo Lastra, and Montek Singh. 2012. Lossless compression of variable-precision floating-point buffers on GPUs. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games. ACM, 47--54. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Jacob Ström, Per Wennersten, Jim Rasmusson, Jon Hasselgren, Jacob Munkberg, Petrik Clarberg, and Tomas Akenine-Möller. 2008. Floating-point buffer compression in a unified codec architecture. In Proceedings of the 23rd ACM SIGGRAPH/EUROGRAPHICS symposium on Graphics hardware. Eurographics Association, 75--84. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Krzysztof Wegner, Olgierd Stankiewicz, and Marek Domanski. 2014. Fast View Synthesis using platelet-based depth representation. In Systems, Signals and Image Processing (IWSSIP), 2014 International Conference on. IEEE, 55--58.Google ScholarGoogle Scholar
  23. M. O. Wildeboer, T. Yendo, M. P. Tehrani, T. Fujii, and M. Tanimoto. 2010. Color based depth up-sampling for depth compression. In 28th Picture Coding Symposium. 170--173.Google ScholarGoogle Scholar
  24. Andrew D. Wilson. 2017. Fast Lossless Depth Image Compression. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (ISS '17). ACM, New York, NY, USA, 100--105. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. GPU-accelerated depth codec for real-time, high-quality light field reconstruction

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image Proceedings of the ACM on Computer Graphics and Interactive Techniques
          Proceedings of the ACM on Computer Graphics and Interactive Techniques  Volume 1, Issue 1
          July 2018
          378 pages
          EISSN:2577-6193
          DOI:10.1145/3242771
          Issue’s Table of Contents

          Copyright © 2018 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 25 July 2018
          Published in pacmcgit Volume 1, Issue 1

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Research
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!