Abstract
Pre-calculated depth information is essential for efficient light field video rendering, due to the prohibitive cost of depth estimation from color when real-time performance is desired. Standard state-of-the-art video codecs fail to satisfy such performance requirements when the amount of data to be decoded becomes too large. In this paper, we propose a depth image and video codec based on block compression, that exploits typical characteristics of depth streams, drawing inspiration from S3TC texture compression and geometric wavelets. Our codec offers very fast hardware-accelerated decoding that also allows partial extraction for view-dependent decoding. We demonstrate the effectiveness of our codec in a number of multi-view 360 degree video datasets, with quantitative analysis of storage cost, reconstruction quality and decoding performance.
- Edward H Adelson, James R Bergen, et al. 1991. The plenoptic function and the elements of early vision. (1991).Google Scholar
- Filippo Bannò, Paolo Simone Gasparello, Franco Tecchia, and Massimo Bergamasco. 2012. Real-time Compression of Depth Streams Through Meshification and Valence-based Encoding. In Proceedings of the 11th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry (VRCAI '12). ACM, New York, NY, USA, 263--270. Google Scholar
Digital Library
- Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan. 2015. High-quality Streamable Free-viewpoint Video. ACM Trans. Graph. 34, 4, Article 69 (July 2015), 13 pages. Google Scholar
Digital Library
- Piotr Didyk Tobias Ritschel, Elmar Eisemann, Karol Myszkowski, and Hans-Peter Seidel. 2010. Adaptive Image-space Stereo View Synthesis.. In VMV. 299--306.Google Scholar
- C Goktug Gurler, Anil Aksay, Gozde Bozdagi Akar, and A Murat Tekalp. 2009. Multi-threaded architectures and benchmark tests for real-time multi-view video decoding. In Multimedia and Expo, 2009. ICME 2009. IEEE International Conference on. IEEE, 237--240. Google Scholar
Digital Library
- Fabian Jäger. 2011. Contour-based segmentation and coding for depth map compression. In Visual Communications and Image Processing (VCIP), 2011 IEEE. IEEE, 1--4.Google Scholar
- Xiaoran Jiang, Mikaël Le Pendu, and Christine Guillemot. 2017. Light field compression using depth image based view synthesis. In Multimedia 8 Expo Workshops (ICMEW), 2017IEEE International Conference on. IEEE, 19--24.Google Scholar
- Vahid Kiani, Ahad Harati, and Abedin Vahedian. 2017. Planelets - A Piecewise Linear Fractional Model for Preserving Scene Geometry in Intra-Coding of Indoor Depth Images. IEEE Transactions on Image Processing 26 (2017), 590--602. Google Scholar
Digital Library
- Babis Koniaris, Maggie Kosek, David Sinclair, and Kenny Mitchell. 2017. Real-time Rendering with Compressed Animated Light Fields. In Proceedings of the 43rd Graphics Interface Conference (GI '17). Canadian Human-Computer Communications Society, School of Computer Science, University of Waterloo, Waterloo, Ontario, Canada, 33--40. Google Scholar
Digital Library
- Jianjun Lei, Shuai Li, Ce Zhu, Ming-Ting Sun, and Chunping Hou. 2015. Depth coding based on depth-texture motion and structure similarities. IEEE Transactions on Circuits and Systems for Video Technology 25, 2 (2015), 275--286.Google Scholar
Cross Ref
- Marc Levoy and Pat Hanrahan. 1996. Light field rendering. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. ACM, 31--42. Google Scholar
Digital Library
- Shujie Liu, PoLin Lai, Dong Tian, and Chang Wen Chen. 2011. New depth coding techniques with utilization of corresponding video. IEEE Transactions on broadcasting 57, 2 (2011), 551--561.Google Scholar
Cross Ref
- Yunpeng Liu, Stephan Beck, Renfang Wang, Jin Li, Huixia Xu, Shijie Yao, Xiaopeng Tong, and Bernd Froehlich. 2015. Hybrid Lossless-Lossy Compression for Real-Time Depth-Sensor Streams in 3D Telepresence Applications.. In PCM (1) (Lecture Notes in Computer Science), Yo-Sung Ho, Jitao Sang, Yong Man Ro, Junmo Kim, and Fei Wu (Eds.), Vol. 9314. Springer, 442--452.Google Scholar
- Morgan McGuire, Mike Mara, Derek Nowrouzezahrai, and David Luebke. 2017. Real-time global illumination using precomputed light field probes. In Proceedings of the 21st ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games. ACM, 2. Google Scholar
Digital Library
- Philipp Merkle, Aljoscha Smolic, Karsten Muller, and Thomas Wiegand. 2007. Multi-view video plus depth representation and coding. In Image Processing, 2007. ICIP 2007. IEEE International Conference on, Vol. 1. IEEE, I-201.Google Scholar
Cross Ref
- Yannick Morvan, Dirk Farin, et al. 2006. Platelet-based coding of depth maps for the transmission of multiview images. In Proceedings of SPIE, Stereoscopic Displays and Applications, Vol. 6055. 93--100.Google Scholar
- Karsten Muller, Heiko Schwarz, Detlev Marpe, Christian Bartnik, Sebastian Bosse, Heribert Brust, Tobias Hinz, Haricharan Lakshman, Philipp Merkle, Franz Hunn Rhee, Gerhard Tech, Martin Winken, and Thomas Wiegand. 2013. 3D High-Efficiency Video Coding for Multi-View Video and Depth Data. Trans. Img. Proc. 22, 9 (Sept. 2013), 3366--3378. Google Scholar
Digital Library
- Dawid Pająk, Robert Herzog, Radosław Mantiuk, Piotr Didyk, Elmar Eisemann, Karol Myszkowski, and Kari Pulli. 2014. Perceptual depth compression for stereo applications. In Computer Graphics Forum, Vol. 33. Wiley Online Library, 195--204. Google Scholar
Digital Library
- Fabrizio Pece, Jan Kautz, and Tim Weyrich. 2011. Adapting Standard Video Codecs for Depth Streaming. In Joint Virtual Reality Conference of EGVE - EuroVR, Sabine Coquillart, Anthony Steed, and Greg Welch (Eds.). The Eurographics Association. Google Scholar
Digital Library
- Jeff Pool, Anselmo Lastra, and Montek Singh. 2012. Lossless compression of variable-precision floating-point buffers on GPUs. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games. ACM, 47--54. Google Scholar
Digital Library
- Jacob Ström, Per Wennersten, Jim Rasmusson, Jon Hasselgren, Jacob Munkberg, Petrik Clarberg, and Tomas Akenine-Möller. 2008. Floating-point buffer compression in a unified codec architecture. In Proceedings of the 23rd ACM SIGGRAPH/EUROGRAPHICS symposium on Graphics hardware. Eurographics Association, 75--84. Google Scholar
Digital Library
- Krzysztof Wegner, Olgierd Stankiewicz, and Marek Domanski. 2014. Fast View Synthesis using platelet-based depth representation. In Systems, Signals and Image Processing (IWSSIP), 2014 International Conference on. IEEE, 55--58.Google Scholar
- M. O. Wildeboer, T. Yendo, M. P. Tehrani, T. Fujii, and M. Tanimoto. 2010. Color based depth up-sampling for depth compression. In 28th Picture Coding Symposium. 170--173.Google Scholar
- Andrew D. Wilson. 2017. Fast Lossless Depth Image Compression. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (ISS '17). ACM, New York, NY, USA, 100--105. Google Scholar
Digital Library
Index Terms
GPU-accelerated depth codec for real-time, high-quality light field reconstruction
Recommendations
Multiple light field rendering
GRAPHITE '03: Proceedings of the 1st international conference on Computer graphics and interactive techniques in Australasia and South East AsiaA light field is a 4D function describing the radiance across a boundary between the volume containing a scene, and the disjoint volume in which the eyepoint may be placed. Light field rendering is the process of rendering novel views of a scene ...
Real-time Rendering with Compressed Animated Light Fields
GI '17: Proceedings of the 43rd Graphics Interface ConferenceWe propose an end-to-end solution for presenting movie quality animated graphics to the user while still allowing the sense of presence afforded by free viewpoint head motion. By transforming offline rendered movie content into a novel immersive ...
Learning based compression of surface light fields for real-time rendering of global illumination scenes
SA '13: SIGGRAPH Asia 2013 Technical BriefsWe present an algorithm for compression and real-time rendering of surface light fields (SLF) encoding the visual appearance of objects in static scenes with high frequency variations. We apply a non-local clustering in order to exploit spatial ...






Comments