skip to main content
research-article

Over- and Under-Exposure Reconstruction of a Single Plenoptic Capture

Published:22 May 2018Publication History
Skip Abstract Section

Abstract

Light field images, for example, taken with plenoptic cameras, offer interesting post-processing opportunities, including depth-of-field management, depth estimation, viewpoint selection, and 3D image synthesis. Like most capture devices, however, plenoptic cameras have a limited dynamic range, so that over- and under-exposed areas in plenoptic images are commonplace. We therefore present a straightforward and robust plenoptic reconstruction technique based on the observation that vignetting causes peripheral views to receive less light than central views. Thus, corresponding pixels in different views can be used to reconstruct illumination, especially in areas where information missing in one view is present in another. Our algorithm accurately reconstructs under- and over-exposed regions (known as declipping), additionally affording an increase in peak luminance by up to two f-stops, and a comparable lowering of the noise floor. The key advantages of this approach are that no hardware modifications are necessary to improve the dynamic range, that no multiple exposure techniques are required, and therefore that no ghosting or other artifacts are introduced.

References

  1. Mekides Assefa Abebe, Tania Pouli, Jonathan Kervec, and Chaker Larabi. 2015. Color clipping and over-exposure correction. In Eurographics Symposium on Rendering—Experimental Ideas 8 Implementations, Jaakko Lehtinen and Derek Nowrouzezahrai (eds.). The Eurographics Association.Google ScholarGoogle Scholar
  2. Edward H. Adelson and John Y. A. Wang. 1992. Single lens stereo with a plenoptic camera. IEEE Trans. Pattern Anal. Mach. Intell. 14, 2 (1992), 99--106. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Kurt Barton Akeley, Brian Cabral, Colvin Pitts, Chia-Kai Liang, Bennett Wilburn, Timothy James Knight, and Yi-Ren Ng. 2014. Compensation for sensor saturation and microlens modulation during light-field image processing. (2014). Patent No. US 9386288 B2, filed December 15, 2014, issued July 5, 2016.Google ScholarGoogle Scholar
  4. Tom E. Bishop and Paolo Favaro. 2012. The light field camera: Extended depth of field, aliasing, and superresolution. IEEE Trans. Pattern Anal. Mach. Intell. 34, 5 (2012), 972--986. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Tom E. Bishop, Sara Zanetti, and Paolo Favaro. 2009. Light field superresolution. In Proceedings of the IEEE International Conference on Computational Photography. 1--9.Google ScholarGoogle ScholarCross RefCross Ref
  6. D. Cho, M. Lee, S. Kim, and Y.-W. Tai. 2013. Modeling the calibration pipeline of the Lytro camera for high quality light-field image reconstruction. In IEEE International Conference on Computer Vision. 3280--3287. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. F. R. Chung. 1997. Spectral Graph Theory. American Mathematical Society.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13, 9 (2004), 1200--1212. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Donald Dansereau, Oscar Pizarro, and Stefan Williams. 2013. Decoding, calibration and rectification for lenselet-based plenoptic cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1027--1034. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Paul Debevec and Jitendra Malik. 1997. Recovering high dynamic range radiance maps from photographs. In SIGGRAPH’97: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques. 369--378. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Alexei A. Efros and Thomas K. Leung. 1999. Texture synthesis by non-parametric sampling. In Proceedings of the Seventh IEEE International Conference on Computer Vision, 1999, Vol. 2. 1033--1038. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Juliet Fiss, Brian Curless, and Richard Szeliski. 2014. Refocussing plenoptic images using depth-adaptive splatting. In Proceedings of the IEEE International Conference on Computational Photography. 1--9.Google ScholarGoogle Scholar
  13. Orazio Gallo, N. Gelfandz, Wei-Chao Chen, Marius Tico, and Kari Pulli. 2009. Artifact-free high dynamic range imaging. In Proceedings of the 2009 IEEE International Conference on Computational Photography (ICCP’09). IEEE, 1--7.Google ScholarGoogle ScholarCross RefCross Ref
  14. Elena Garces, Jose I. Echevarria, Wen Zhang, Hongzhi Wu, Kun Zhou, and Diego Gutierrez. 2016. Intrinsic light fields. https://arxiv.org/abs/1608.04342 (2016).Google ScholarGoogle Scholar
  15. Todor Georgiev and Andrew Lumsdaine. 2010. Rich image capture with plenoptic cameras. In Proceedings of the IEEE International Conference on Computational Photography. 1--8.Google ScholarGoogle ScholarCross RefCross Ref
  16. Todor Georgiev, Andrew Lumsdaine, and Sergio Goma. 2009. High dynamic range image capture with plenoptic 2.0 camera. In Proceedings of Signal Recovery and Synthesis. Optical Society of America, SWA7P.Google ScholarGoogle ScholarCross RefCross Ref
  17. Todor Georgiev, Ke Colin Zheng, Brian Curless, David Salesin, Shree Nayar, and Chintan Intwala. 2006. Spatio-angular resolution tradeoff in integral photography. In Proceedings of the Eurographics Symposium on Rendering. 263--272. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. 1996. The lumigraph. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. ACM, 43--54. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Miguel Granados, Hans-Peter Seidel, and Hendrik P. A. Lensch. 2008. Background estimation from non-time sequence images. In Proceedings of Graphics Interface 2008. 33--40. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Christopher Hahne, Amar Aggoun, Vladan Velisavljevic, Susanne Fiebig, and Matthias Pesch. 2016. Refocusing distance of a standard plenoptic camera. Optics Express 24, 19 (2016), 21521--21540.Google ScholarGoogle ScholarCross RefCross Ref
  21. Roarke Horstmeyer, Gary Euliss, Ravindra Athale, and Marc Levoy. 2009. Flexible multimodal camera using a light field architecture. In Proceedings of the IEEE Conference on Computational Photography. 1--8.Google ScholarGoogle ScholarCross RefCross Ref
  22. W. Hu, G. Cheung, and M. Kazui. 2016. Graph-based dequantization of block-compressed piecewise smooth images. IEEE Signal Process. Lett. 23, 2 (2016), 242--246.Google ScholarGoogle ScholarCross RefCross Ref
  23. W. Hu, X. Li, G. Cheung, and O. Au. 2013. Depth map denoising using graph-based transform and group sparsity. In Proceedings of the IEEE International Workshop on Multimedia Signal Processing.Google ScholarGoogle Scholar
  24. Wei Hu, Mozhdeh Seifi, and Erik Reinhard. 2017. Optical center estimation for lenslet-based plenoptic cameras. In 19th IEEE International Workshop on Multimedia Signal Processing (MMSP’17).Google ScholarGoogle ScholarCross RefCross Ref
  25. Ivo Ihrke, Timo Stich, Heiko Gottschlich, Marcus Magnor, and Hans-Peter Seidel. 2008. Fast incident light field acquisition and rendering. J. WSCG 16, 1 (2008), 25--32.Google ScholarGoogle Scholar
  26. Aaron Isaksen, Leonard McMillan, and Steven J Gortler. 2000. Dynamically reparameterized light fields. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. 297--306. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Frederic E. Ives. 1903. Parallax Stereogram and Process of Making Same. (Filed Sept. 25, 1902, issued Apr. 14, 1903). U.S. Patent US725567 A.Google ScholarGoogle Scholar
  28. E. A. Khan, A. O. Akyuz, and E. Reinhard. 2006. Ghost removal in high dynamic range images. In IEEE International Conference on Image Processing. 2005--2008.Google ScholarGoogle Scholar
  29. Seon Joo Kim and Marc Pollefeys. 2008. Robust radiometric calibration and vignetting correction. IEEE Trans. Pattern Anal. Mach. Intell. 30, 4 (2008), 562--576. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Marc Levoy and Pat Hanrahan. 1996. Light field rendering. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. ACM, 31--42. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Marc Levoy, Ren Ng, Andrew Adams, Matthew Footer, and Mark Horowitz. 2006. Light field microscopy. ACM Trans. Graph. 25, 3 (2006), 924--934. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Chia-Kai Liang and Ravi Ramamoorthi. 2015. A light transport framework for lenslet light field cameras. ACM Trans. Graph. 34, 2 (March 2015), 16:1--16:19. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Andrew Lumsdaine and Todor Georgiev. 2008. Full Resolution Lightfield Rendering.Technical report, Indiana University and Adobe Systems.Google ScholarGoogle Scholar
  34. Andrew Lumsdaine and Todor Georgiev. 2009. The focused plenoptic camera. In IEEE International Conference on Computational Photography (ICCP’09). 1--8.Google ScholarGoogle ScholarCross RefCross Ref
  35. U. Luxburg. 2007. A tutorial on spectral clustering. In Statistics and Computing, Vol. 17, No. 4. 395--416. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Henrique S. Malvar, Li-Wei He, and Ross Cutler. 2004. High-quality linear interpolation for demosaicking of bayer-patterned color images. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Vol. 3. iii--485.Google ScholarGoogle Scholar
  37. Alkhazur Manakov, John Restrepo, Oliver Klehm, Ramon Hegedus, Elmar Eisemann, Hans-Peter Seidel, and Ivo Ihrke. 2013. A reconfigurable camera add-on for high dynamic range, multispectral, polarization, and light-field imaging. ACM Trans. Graph. 32, 4 (2013), 47--1. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. S. Mann and R. W. Picard. 1995. On being ‘undigital’ with digital cameras: Extending dynamic range by combining differently exposed pictures. In Proceedings of IS&T 48th Annual Conference. 422--428.Google ScholarGoogle Scholar
  39. Syed Z. Masood, Jiejie Zhu, and Marshall F. Tappen. 2009. Automatic correction of saturated regions in photographs using cross-channel correlation. Comput. Graph. Forum 28, 7 (2009), 1861--1869.Google ScholarGoogle ScholarCross RefCross Ref
  40. Vincent Masselus, Pieter Peers, Philip Dutré, and Yves D. Willems. 2003. Relighting with 4D incident light fields. ACM Trans. Graph. 22, 3 (2003), 613--620. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Peyman Milanfar. 2013. A tour of modern image filtering: New insights and methods, both practical and theoretical. IEEE Signal Process. Mag. 30, 1 (2013), 106--128.Google ScholarGoogle ScholarCross RefCross Ref
  42. T. Mitsunaga and S. K. Nayar. 1999. Radiometric self calibration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vol. 1. 374--380.Google ScholarGoogle Scholar
  43. Shree K. Nayar and Srinivasa G. Narasimhan. 2002. Assorted pixels: Multisampled imaging with structural models. In Proceedings of the European Conference on Computer Vision. 636--652. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz, and Pat Hanrahan. 2005. Light field photography with a hand-held plenoptic camera. Comput. Sci. Tech. Rep. 2, 11 (2005), 1--11.Google ScholarGoogle Scholar
  45. Christian Perwass and Lennart Wietzke. 2012. Single lens 3D-camera with extended depth-of-field. In IS&T/SPIE Electronic Imaging. International Society for Optics and Photonics, 829108--829108.Google ScholarGoogle Scholar
  46. Colvin Pitts, Timothy James Knight, Chia-Kai Liang, and Yi-Ren Ng. 2015. Generating dolly zoom effect using light field image data. (2015). Patent No. US 8971625 B2, filed June 23, 2014, issued March 3, 2015.Google ScholarGoogle Scholar
  47. Ramesh Raskar, Amit Agrawal, Cyrus A. Wilson, and Ashok Veeraraghavan. 2008. Glare aware photography: 4D Ray sampling for reducing glare effects of camera lenses. ACM Trans. Graph. 27, 3 (2008), 56:1--56:10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. E. Reinhard and K. Devlin. 2005. Dynamic range reduction inspired by photoreceptor physiology. IEEE Trans. Vis. Comput. Graph. 11, 1 (2005), 13--24. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. E. Reinhard, W. Heidrich, P. Debevec, S. Pattanaik, G. Ward, and K. Myszkowski. 2010. High Dynamic Range Imaging: Acquisition, Display and Image-Based Lighting (2nd ed.). Morgan Kaufmann.Google ScholarGoogle Scholar
  50. Erik Reinhard, Erum Arif Khan, Ahmet Oğuz Akyüz, and Garrett M. Johnson. 2008. Color Imaging: Fundamentals and Applications. A K Peters, Wellesley. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Erik Reinhard, Michael Stark, Peter Shirley, and James Ferwerda. 2002. Photographic tone reproduction for digital images. ACM Trans. Graph. 21, 3 (2002), 267--276. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. M. A. Robertson, S. Borman, and R. L. Stevenson. 2003. Estimation-theoretic approach to dynamic range enhancement using multiple exposures. Journal of Electronic Imaging 12 (2003), 219--228. Issue 2.Google ScholarGoogle ScholarCross RefCross Ref
  53. Neus Sabater, Mozhdeh Seifi, Valter Drazic, Gustavo Sandri, and Patrick Pérez. 2014. Accurate disparity estimation for plenoptic images. In Computer Vision — ECCV 2014 Workshops. Springer, 548--560.Google ScholarGoogle Scholar
  54. Yoav Y. Schechner and Shree K. Nayar. 2003. Generalized mosaicing: High dynamic range in a wide field of view. International Journal of Computer Vision 53, 3 (2003), 245--267. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. David C. Schedl, Clemens Birklbauer, and Oliver Bimber. 2014. Coded exposure HDR light-field video recording. Computer Graphics Forum 33, 2 (2014), 33--42. Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. Pradeep Sen, Nima Khademi Kalantari, Maziar Yaesoubi, Soheil Darabi, Dan B. Goldman, and Eli Shechtman. 2012. Robust patch-based HDR reconstruction of dynamic scenes. ACM Trans. Graph. 31, 6 (2012), 203:1--203:11. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. D. I. Shuman, S. K. Narang, P. Frossard, A. Ortega, and P. Vandergheynst. 2013. The emerging field of signal processing on graphs. IEEE Signal Processing Magazine 30, 3 (2013), 83--98.Google ScholarGoogle ScholarCross RefCross Ref
  58. Thiow Keng Tan, Choong Seng Boon, and Yuya Suzuki. 2006. Intra prediction by template matching. In Proceedings of the IEEE International Conference on Image Processing. 1693--1696.Google ScholarGoogle ScholarCross RefCross Ref
  59. Michael Tao, Sunil Hadap, Jitendra Malik, and Ravi Ramamoorthi. 2013. Depth from combining defocus and correspondence using light-field cameras. In Proceedings of the IEEE International Conference on Computer Vision. 673--680. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. J. Unger, S. Gustavson, P. Larsson, and A. Ynnerman. 2008. Free form incident light fields. Computer Graphics Forum 27, 4 (2008), 1293--1301. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. J. Unger, A. Wenger, T. Hawtins, A. Gardner, and P. Debevec. 2003. Capturing and rendering with incident light fields. In Proceedings of the Eurographics Symposium on Rendering. 141--149. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Ashok Veeraraghavan, Ramesh Raskar, Amit Agrawal, Ankit Mohan, and Jack Tumblin. 2007. Dappled photography: Mask enhanced cameras for hyterodyned light fields and coded aperture refocusing. ACM Trans. Graph. 26, 3 (2007), 69. Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. Lvdi Wang, Li-Yi Wei, Kun Zhou, Baining Guo, and Heung-Yeung Shum. 2007. High dynamic range image hallucination. In Proceedings of the 18th Eurographics conference on Rendering Techniques. Eurographics Association, 321--326. Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. Ting-Chun Wang, Jun-Yan Zhu, Ebi Hiroaki, Manmohan Chandraker, Alexei A Efros, and Ravi Ramamoorthi. 2016. A 4D light-field dataset and CNN architecture for material recognition. In Proceedings of the European Conference on Computer Vision. 121--138.Google ScholarGoogle ScholarCross RefCross Ref
  65. Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13, 4 (2004), 600--612. Google ScholarGoogle ScholarDigital LibraryDigital Library
  66. Sven Wanner and Bastian Goldluecke. 2014. Variational light field analysis for disparity estimation and super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. 36, 3 (2014), 606--619. Google ScholarGoogle ScholarDigital LibraryDigital Library
  67. Greg Ward. 2003. Fast, robust image registration for compositing high dynamic range photographs from hand-held exposures. J. Graph. Tools 8, 2 (2003), 17--30.Google ScholarGoogle ScholarCross RefCross Ref
  68. Li-Yi Wei and Marc Levoy. 2001. Texture synthesis over arbitrary manifold surfaces. In SIGGRAPH’01: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. 355--360. Google ScholarGoogle ScholarDigital LibraryDigital Library
  69. Z. Yu, X. Guo, H. Ling, A. Lumsdaine, and J. Yu. 2013. Line assisted light field triangulation and stereo matching. In Proceedings of the IEEE International Conference on Computer Vision. 2792--2799. Google ScholarGoogle ScholarDigital LibraryDigital Library
  70. Zhan Yu, Jingji Yu, Andrew Lumsdaine, and Todor Georgiev. 2012. An analysis of color demosaicing in plenoptic cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 901--908. Google ScholarGoogle ScholarDigital LibraryDigital Library
  71. Yuanjie Zheng, Stephen Lin, Chandra Kambhamettu, Jingyi Yu, and Sing Bing Kang. 2009. Single-image vignetting correction. IEEE Trans. Pattern Anal. Mach. Intell. 31, 12 (2009), 2243--2256. Google ScholarGoogle ScholarDigital LibraryDigital Library
  72. Shan Zhu and Kai-Kuang Ma. 2000. A new diamond search algorithm for fast block-matching motion estimation. IEEE Trans. Image Process. 9, 2 (2000), 287--290. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Over- and Under-Exposure Reconstruction of a Single Plenoptic Capture

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!