Programmable aperture photography: multiplexed light field acquisition

Abstract

In this paper, we present a system including a novel component called programmable aperture and two associated post-processing algorithms for high-quality light field acquisition. The shape of the programmable aperture can be adjusted and used to capture light field at full sensor resolution through multiple exposures without any additional optics and without moving the camera. High acquisition efficiency is achieved by employing an optimal multiplexing scheme, and quality data is obtained by using the two post-processing algorithms designed for self calibration of photometric distortion and for multi-view depth estimation. View-dependent depth maps thus generated help boost the angular resolution of light field. Various post-exposure photographic effects are given to demonstrate the effectiveness of the system and the quality of the captured light field.

References

1. Adelson, E. H., and Wang, J. Y. A. 1992. Single lens stereo with a plenoptic camera. IEEE Trans. Pattern Anal. Mach. Intell. 14, 2, 99--106.
2. Aggarwal, M., Hua, H., and Ahuja, N. 2001. On cosinefourth and vignetting effects in real lenses. In Proc. ICCV '01: Proc. the Eighth IEEE International Conference on Computer Vision, vol. 1, 472--479.
3. Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23, 11, 1222--1239.
4. Chai, J.-X., Chan, S.-C., and Tong, H.-Y. S. X. 2000. Plenoptic sampling. In SIGGRAPH '00: Proceedings of the 27th annual conference on Computer graphics and interactive techniques, ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 307--318.
5. Debevec, P. E., Taylor, C. J., and Malik, J. 1996. Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach. In SIGGRAPH '96: Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, ACM, New York, NY, USA, 11--20.
6. Donoho, D. 2006. Compressed Sensing. IEEE Trans. Information Theory 52, 4, 1289--1306.
7. Durand, F., Holzschuch, N., Soler, C., Chan, E., and Sillion, F. X. 2005. A frequency analysis of light transport. In SIGGRAPH '05: ACM SIGGRAPH 2005 Papers, ACM Press, New York, NY, USA, 1115--1126.
8. Farid, H., and Simoncelli, E. P. 1998. Range estimation by optical differentiation. Journal of the Optical Society of America A 15, 7, 1777--1786.
9. Georgiev, T., Zheng, K. C., Curless, B., Salesin, D., Nayar, S., and Intwala, C. 2006. Spatio-angular resolution tradeoff in integral photography. In EGRW '06: Proc. the 17th Eurographics workshop on Rendering.
10. Georgiev, T., Intwala, C., and Babacan, D. 2007. Lightfield capture by multiplexing in the frequency domain. Adobe technical report, Adobe Systems Incorporated.
11. Goldman, D. B., and Chen, J.-H. 2005. Vignette and exposure calibration and compensation. In Proc. ICCV '05: Proc. the 10th IEEE International Conference on Computer Vision, 899--906.
12. Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen, M. F. 1996. The lumigraph. In SIGGRAPH '96: Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, ACM Press, New York, NY, USA, 43--54.
13. Green, P., Sun, W., Matusik, W., and Durand, F. 2007. Multi-aperture photography. ACM Trans. Graph. 26, 3, 68.
14. Harwit, M., and Sloane, N. J. 1979. Hadamard Transform Optics. Academic Press, New York.
15. HP components group. 1998. Noise sources in CMOS image sensors. Technical report, Hewlett-Packard Company.
16. Isaksen, A., McMillan, L., and Gortler, S. J. 2000. Dynamically reparameterized light fields. In SIGGRAPH '00: Proceedings of the 27th annual conference on Computer graphics and interactive techniques, ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 297--306.
17. Ive, H. E. 1930. Parallax panoramagrams made with a large diameter lens. Journal of the Optical Society of America 20, 6 (June), 332--342.
18. Joshi, N., Matusik, W., and Avidan, S. 2006. Natural video matting using camera arrays. ACM Trans. Graph. 25, 3, 779--786.
19. Kang, S. B., and Szeliski, R. 2004. Extracting view-dependent depth maps from a collection of images. International Journal of Computer Vision 58, 2, 139--163.
20. Kolmogorov, V., and Zabih, R. 2002. Multi-camera scene reconstruction via graph cuts. Proc. European Conference on Computer Vision 3, 82--96.
21. Kolmogorov, V. 2006. Convergent tree-reweighted message passing for energy minimization. IEEE Trans. Pattern Anal. Mach. Intell. 28, 10, 1568--1583.
22. Levin, A., Fergus, R., Durand, F., and Freeman, W. T. 2007. Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph. 26, 3, 70.
23. Levoy, M., and Hanrahan, P. 1996. Light field rendering. In SIGGRAPH '96: Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, ACM Press, New York, NY, USA, 31--42.
24. Liang, C.-K., Liu, G., and Chen, H. H. 2007. Light field acquisition using programmable aperture camera. In Proc. IEEE International Conference on Image Processing, vol. 5, 233--236.
25. Lippmann, M. G. 1908. Epreuves reversible donnant la sensation du relief. J. Phys 7, 821--825.
26. Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60, 2, 91--110.
27. Nayar, S. K., and Branzoi, V. 2003. Adaptive dynamic range imaging: Optical control of pixel exposures over space and time. In Proc. ICCV '03: Proc. the Ninth IEEE International Conference on Computer Vision, vol. 2, 1168--1175.
28. Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., and Hanrahan, P. 2005. Light field photography with a hand-held plenoptic camera. CSTR 2005-02, Stanford University, April.
29. Ng, R. 2005. Fourier slice photography. In SIGGRAPH '05: ACM SIGGRAPH 2005 Papers, ACM, New York, NY, USA, 735--744.
30. Okoshi, T. 1976. Three-dimensional imaging techniques. Academic Press New York.
31. Raskar, R., Tan, K.-H., Feris, R., Yu, J., and Turk, M. 2004. Non-photorealistic camera: depth edge detection and stylized rendering using multi-flash imaging. In SIGGRAPH '04: ACM SIGGRAPH 2004 Papers, ACM Press, New York, NY, USA, 679--688.
32. Raskar, R., Agrawal, A., and Tumblin, J. 2006. Coded exposure photography: motion deblurring using fluttered shutter. ACM Trans. Graph. 25, 3, 795--804.
33. Ratner, N., and Schechner, Y. Y. 2007. Illumination multiplexing within fundamental limits. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 1--8.
34. Scharstein, D., and Szeliski, R. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision 47, 1--3, 7--42.
35. Schechner, Y. Y., and Nayar, S. K. 2004. Uncontrolled modulation imaging. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, 197--204.
36. Schechner, Y. Y., Nayar, S. K., and Belhumeur, P. N. 2003. A theory of multiplexed illumination. In Proc. ICCV '03: Proc. the Ninth IEEE International Conference on Computer Vision, vol. 2, 808--815.
37. Senkichi, C., Toshio, M., Toshinori, H., Yuichi, M., and Hidetoshi, K., 2003. Device and method for correcting camera-shake and device for detecting camera shake. JP Patent No. 2003-138436.
38. Sun, J., Li, Y., Kang, S. B., and Shum, H.-Y. 2005. Symmetric stereo matching for occlusion handling. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, 399--406.
39. Szeliski, R., Zabih, R., Scharstein, D., Veksler, O., Kolmogorov, V., Agarwala, A., Tappen, M., and Rother, C. 2006. A comparative study of energy minimization methods for markov random fields. In Proc. European Conference on Computer Vision, vol. 2, 16--29.
40. Tsin, Y., Ramesh, V., and Kanade, T. 2001. Statistical calibration of CCD imaging process. In Proc. ICCV '01: Proc. the Eighth IEEE International Conference on Computer Vision, 480--487.
41. Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., and Tumblin, J. 2007. Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Trans. Graph. 26, 3, 69.
42. Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., and Debevec, P. 2005. Performance relighting and reflectance transformation with time-multiplexed illumination. In ACM SIGGRAPH '05: ACM SIGGRAPH 2005 Papers, ACM Press, New York, NY, USA, 756--764.
43. Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. In SIGGRAPH '05: ACM SIGGRAPH 2005 Papers, ACM, New York, NY, USA, 765--776.
44. Yang, J. C., Everett, M., Buehler, C., and McMillan, L. 2002. A real-time distributed light field camera. In EGRW '02: Proc. the 13th Eurographics workshop on Rendering, 77--86.
45. Yang, Q., Yang, R., Davis, J., and Nister, D. 2007. Spatial-depth super resolution for range images. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 1--8.
46. Zomet, A., and Nayar, S. K. 2006. Lensless imaging with a controllable aperture. Proc. IEEE Conference on Computer Vision and Pattern Recognition, 339--346.
47. Zwicker, M., Matusik, W., Durand, F., and Pfister, H. 2006. Antialiasing for automultiscopic 3d displays. In EGSR'06: Proc. the 17th Eurographics Symposium on Rendering.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

      • Computer vision tasks

        • Scene understanding

        • Vision for robotics

      • Image and video acquisition

  • Modeling and simulation

    • Model development and analysis

      • Model verification and validation

      • Modeling methodologies