skip to main content
research-article

An Adaptive Two-Layer Light Field Compression Scheme Using GNN-Based Reconstruction

Authors Info & Claims
Published:21 June 2020Publication History
Skip Abstract Section

Abstract

As a new form of volumetric media, Light Field (LF) can provide users with a true six degrees of freedom immersive experience because LF captures the scene with photo-realism, including aperture-limited changes in viewpoint. But uncompressed LF data is too large for network transmission, which is the reason why LF compression has become an important research topic. One of the more recent approaches for LF compression is to reduce the angular resolution of the input LF during compression and to use LF reconstruction to recover the discarded viewpoints during decompression. Following this approach, we propose a new LF reconstruction algorithm based on Graph Neural Networks; we show that it can achieve higher compression and better quality compared to existing reconstruction methods, although suffering from the same problem as those methods—the inability to deal effectively with high-frequency image components. To solve this problem, we propose an adaptive two-layer compression architecture that separates high-frequency and low-frequency components and compresses each with a different strategy so that the performance can become robust and controllable. Experiments with multiple datasets1 show that our proposed scheme is capable of providing a decompression quality of above 40 dB, and can significantly improve compression efficiency compared with similar LF reconstruction schemes.

References

  1. Edward H. Adelson and James R. Bergen. 1991. The Plenoptic Function and the Elements of Early Vision. Vol. 2. Vision and Modeling Group, Media Laboratory, Massachusetts Institute of Technology.Google ScholarGoogle Scholar
  2. Amar Aggoun. 2011. Compression of 3D integral images using 3D wavelet transform. Journal of Display Technology 7, 11 (2011), 586--592.Google ScholarGoogle ScholarCross RefCross Ref
  3. H. Amirpour, M. Pereira, and A. Pinheiro. 2018. High efficient snake order pseudo-sequence based light field image compression. In 2018 Data Compression Conference. 397--397. DOI:https://doi.org/10.1109/DCC.2018.00050Google ScholarGoogle ScholarCross RefCross Ref
  4. Computer Graphics Laboratory, Stanford University 2008. Light Field Datasets. Retrieved from http://lightfield.stanford.edu/lfs.html.Google ScholarGoogle Scholar
  5. Caroline Conti, Luís Ducla Soares, and Paulo Nunes. 2016. HEVC-based 3D holoscopic video coding using self-similarity compensated prediction. Signal Processing Image Communication 42 (2016), 59--78.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. D. G. Dansereau, O. Pizarro, and S. B. Williams. 2013. Decoding, calibration and rectification for lenselet-based plenoptic cameras. In 2013 IEEE Conference on Computer Vision and Pattern Recognition. 1027--1034. DOI:https://doi.org/10.1109/CVPR.2013.137Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. 1996. The lumigraph. In 23rd Annual Conference on Computer Graphics and Interactive Techniques. 43--54.Google ScholarGoogle Scholar
  8. Bichuan Guo, Yuxing Han, and Jiangtao Wen. 2018. Convex optimization based bit allocation for light field compression under weighting and consistency constraints. In Data Compression Conference. IEEE.Google ScholarGoogle ScholarCross RefCross Ref
  9. Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems. 1024--1034.Google ScholarGoogle Scholar
  10. Harini Priyadarshini Hariharan, Tobias Lange, and Thorsten Herfet. 2017. Low complexity light field compression based on pseudo-temporal circular sequencing. In IEEE International Symposium on Broadband Multimedia Systems and Broadcasting. 1--5.Google ScholarGoogle ScholarCross RefCross Ref
  11. Fatma Hawary, Christine Guillemot, Dominique Thoreau, and Guillaume Boisson. 2017. Scalable light field compression scheme using sparse reconstruction and restoration. In 2017 IEEE International Conference on Image Processing (ICIP). IEEE, 3250--3254.Google ScholarGoogle ScholarCross RefCross Ref
  12. Xinjue Hu, Jingming Shan, Yu Liu, and Lin Zhang. 2019. Adaptive two-layer light field compression scheme based on sparse reconstruction. In 10th ACM Multimedia Systems Conference. 74--85.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Xiaoran Jiang, Mikaël Le Pendu, Reuben A. Farrugia, and Christine Guillemot. 2017. Light field compression with homography-based low rank approximation. IEEE Journal of Selected Topics in Signal Processing PP, 99 (2017), 1--1.Google ScholarGoogle Scholar
  14. Shinjini Kundu. 2012. Light field compression using homography and 2D warping. In IEEE International Conference on Acoustics, Speech and Signal Processing. 1349--1352.Google ScholarGoogle ScholarCross RefCross Ref
  15. Marc Levoy and Pat Hanrahan. 1996. Light field rendering. In 23rd Annual Conference on Computer Graphics and Interactive Techniques. 31--42.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Li Li, Zhu Li, Bin Li, Dong Liu, and Houqiang Li. 2017. Pseudo-sequence-based 2-D hierarchical coding structure for light-field image compression. IEEE Journal of Selected Topics in Signal Processing 11, 7 (2017), 1107--1119.Google ScholarGoogle ScholarCross RefCross Ref
  17. Yun Li, Marten Sjostrom, Roger Olsson, and Ulf Jennehag. 2014. Efficient intra prediction scheme for light field image compression. In IEEE International Conference on Acoustics, Speech and Signal Processing. 539--543.Google ScholarGoogle ScholarCross RefCross Ref
  18. Dong Liu, Lizhi Wang, Li Li, Zhiwei Xiong, Feng Wu, and Wenjun Zeng. 2016. Pseudo-sequence-based light field image compression. In IEEE International Conference on Multimedia and Expo Workshops. 1--4.Google ScholarGoogle ScholarCross RefCross Ref
  19. Luís F. R. Lucas, Caroline Conti, Paulo Nunes, Luís Ducla Soares, Nuno M. M. Rodrigues, Carla L. Pagliari, Eduardo A. B. Da Silva, and Sérgio M. M. De Faria. 2014. Locally linear embedding-based prediction for 3D holoscopic image coding using HEVC. In Signal Processing Conference. 11--15.Google ScholarGoogle Scholar
  20. Ricardo Monteiro, Luis Lucas, Caroline Conti, Paulo Nunes, Nuno Rodrigues, Sergio Faria, Carla Pagliari, Eduardo Da Silva, and Luis Soares. 2016. Light field HEVC-based image coding using locally linear embedding and self-similarity compensated prediction. In IEEE International Conference on Multimedia and Expo Workshops. 1--4.Google ScholarGoogle ScholarCross RefCross Ref
  21. Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz, Pat Hanrahan, et al. 2005. Light field photography with a hand-held plenoptic camera. Computer Science Technical Report CSTR 2, 11 (2005), 1--11.Google ScholarGoogle Scholar
  22. Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. 2016. Learning convolutional neural networks for graphs. In International Conference on Machine Learning.Google ScholarGoogle Scholar
  23. C. Perra and P. Assuncao. 2016. High efficiency coding of light field images based on tiling and pseudo-temporal data arrangement. In IEEE International Conference on Multimedia and Expo Workshops. 1--4.Google ScholarGoogle Scholar
  24. Cristian Perra and Daniele Giusto. 2017. JPEG 2000 compression of unfocused light field images based on lenslet array slicing. In IEEE International Conference on Consumer Electronics. 27--28.Google ScholarGoogle Scholar
  25. Reza Rassool. 2017. VMAF reproducibility: Validating a perceptual practical video quality metric. In IEEE International Symposium on Broadband Multimedia Systems and Broadcasting.Google ScholarGoogle Scholar
  26. Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008. The graph neural network model. IEEE Transactions on Neural Networks 20, 1 (2008), 61--80.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Lixin Shi, Haitham Hassanieh, Abe Davis, Dina Katabi, and Fredo Durand. 2014. Light field reconstruction using sparsity in the continuous fourier domain. ACM Transactions on Graphics (TOG) 34, 1 (2014), 12.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Irene Viola, Hermina Petric Maretic, Pascal Frossard, and Touradj Ebrahimi. 2018. A graph learning approach for light field image compression. Applications of Digital Image Processing XLISpie-Int Soc Optical Engineering (2018), 12.Google ScholarGoogle Scholar
  29. Gaochang Wu, Belen Masia, Adrian Jarabo, Yuchen Zhang, Liangyong Wang, Qionghai Dai, Tianyou Chai, and Yebin Liu. 2017. Light field image processing: An overview. IEEE Journal of Selected Topics in Signal Processing 11, 7 (2017), 926--954.Google ScholarGoogle ScholarCross RefCross Ref
  30. Shan Xu, Zhi-Liang Zhou, and Nicholas Devaney. 2014. Multi-view image restoration from plenoptic raw images. In Asian Conference on Computer Vision. Springer, 3--15.Google ScholarGoogle Scholar
  31. Wei Zhang, Dong Liu, Zhiwei Xiong, and Jizheng Xu. 2018. SIFT-based adaptive prediction structure for light field compression. In Visual Communications and Image Processing. 1--4.Google ScholarGoogle Scholar
  32. Xiang Zhang, Philip A. Chou, Ming Ting Sun, Maolong Tang, Shanshe Wang, Siwei Ma, and Wen Gao. 2018. Surface light field compression using a point cloud codec. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 9, 1 (2018), 163--176.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. An Adaptive Two-Layer Light Field Compression Scheme Using GNN-Based Reconstruction

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Multimedia Computing, Communications, and Applications
        ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 16, Issue 2s
        Special Issue on Smart Communications and Networking for Future Video Surveillance and Special Section on Extended MMSYS-NOSSDAV 2019 Best Papers
        April 2020
        291 pages
        ISSN:1551-6857
        EISSN:1551-6865
        DOI:10.1145/3407689
        Issue’s Table of Contents

        Copyright © 2020 ACM

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 21 June 2020
        • Online AM: 7 May 2020
        • Accepted: 1 April 2020
        • Revised: 1 March 2020
        • Received: 1 December 2019
        Published in tomm Volume 16, Issue 2s

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Research
        • Refereed

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!