A reconfigurable camera add-on for high dynamic range, multispectral, polarization, and light-field imaging

Abstract

We propose a non-permanent add-on that enables plenoptic imaging with standard cameras. Our design is based on a physical copying mechanism that multiplies a sensor image into a number of identical copies that still carry the plenoptic information of interest. Via different optical filters, we can then recover the desired information. A minor modification of the design also allows for aperture sub-sampling and, hence, light-field imaging. As the filters in our design are exchangeable, a reconfiguration for different imaging purposes is possible. We show in a prototype setup that high dynamic range, multispectral, polarization, and light-field imaging can be achieved with our design.

References

1. Adelson, E. H., and Bergen, J. R. 1991. The Plenoptic Function and the Elements of Early Vision. In Computational Models of Visual Processing, MIT Press, 3--20.
2. Adelson, E., and Wang, J. 1992. Single Lens Stereo with a Plenoptic Camera. IEEE Trans. PAMI 14, 2, 99--106.
3. Aggarwal, M., and Ahuja, N. 2001. Split aperture imaging for high dynamic range. In Computer Vision, 2001. ICCV 2001. Proceedings. Eighth IEEE International Conference on, vol. 2, 10--17 vol.2.
4. Bando, Y., Chen, B.-Y., and Nishita, T. 2008. Extracting Depth and Matte using a Color-Filtered Aperture. ACM TOG 27, 5, 134.
5. Bayer, B. E., 1976. Color Imaging Array. US Patent 3,971,065.
6. Bonnet, H., Abuter, R., Baker, A., et al. 2004. First Light of SINFONI at the VLT. ESO Messenger 117, 17, 17--24.
7. Cao, X., Tong, X., Dai, Q., and Lin, S. 2011. High-Resolution Multi-Spectral Video Capture with a Hybrid Camera System. In Proc. CVPR, 297--304.
8. Cook, R. L., and DeRose, T. 2005. Wavelet Noise. ACM TOG 24, 3, 735--744.
9. Debevec, P. E., and Malik, J. 1997. Recovering High Dynamic Range Radiance Maps from Photographs. In Proc. SIGGRAPH, 369--378.
10. Descour, M., and Dereniak, E. 1995. Computed-tomography Imaging Spectrometer: Experimental Calibration and Reconstruction Results. Appl. Optics 34, 22, 4817--4826.
11. Du, H., Tong, X., Cao, X., and Lin, S. 2009. A Prism-Based System for Multispectral Video Acquisition. In Proc. ICCV, 175--182.
12. Gehm, M. E., John, R., Brady, D. J., Willett, R. M., and Schulz, T. J. 2007. Single-Shot Compressive Spectral Imaging with a Dual-Disperser Architecture. Optics Exp. 15, 21, 14013--14027.
13. Georgiev, T., Lumsdaine, A., and Chunev, G. 2011. Using Focused Plenoptic Cameras for Rich Image Capture. IEEE CG&A 31, 1, 62--73.
14. Goldstein, D. H. 2003. Polarized Light, 2nd edition. CRC Press, New York, US.
15. Gortler, S., Grzeszczuk, R., Szelinski, R., and Cohen, M. 1996. The Lumigraph. In Proc. SIGGRAPH, 43--54.
16. Habel, R., Kudenov, M., and Wimmer, M. 2012. Practical Spectral Photography. CGF 31, 2 (May), 449--458.
17. Han, J. Y., and Perlin, K. 2003. Measuring Bidirectional Texture Reflectance with a Kaleidoscope. In Proc. SIGGRAPH, 741--748.
18. Hegedus, R., Szel, G., and Horvath, G. 2006. Imaging polarimetry of the circularly polarizing cuticle of scarab beetles (Coleoptera: Rutelidae, Cetoniidae). Vision Research 46, 2786--2797.
19. Horn, B. K. P., and Schunck, B. G. 1981. Determining Optical Flow. Artif. Intell. 17, (1--3), 185--203.
20. Horstmeyer, R., Euliss, G., Athale, R., and Levoy, M. 2009. Flexible Multimodal Camera Using a Light Field Architecture. In Proc. ICCP, 1--8.
21. Ihrke, I., Wetzstein, G., and Heidrich, W. 2010. A Theory of Plenoptic Multiplexing. In Proc. CVPR, 1--8.
22. Ihrke, I., Reshetouski, I., Manakov, A., Tevs, A., Wand, M., and Seidel, H.-P. 2012. A Kaleidoscopic Approach to Surround Geometry and Reflectance Acquisition. In Proceedings of IEEE International Workshop on Computational Cameras and Displays, 1--8.
23. Ihrke, I., 2012. Color Calibration Toolbox for MATLAB v2.0. http://giana.mmci.uni-saarland.de/software.html.
24. Isaksen, A., McMillan, L., and Gortler, S. J. 2000. Dynamically Reparameterized Light Fields. In Proc. SIGGRAPH, 297--306.
25. Ives, H., 1903. Parallax Stereogram and Process of Making Same. US patent 725,567.
26. Kudenov, M., and Dereniak, E. 2011. Compact Snapshot Real-Time Imaging Spectrometer. In SPIE Conf. on Elec.-Opt. Rem. Sens., Phot. Tech., and Appl. V, 81860W-1--81860W-12.
27. Lanman, D., Raskar, R., Agrawal, A., and Taubin, G. 2008. Shield Fields: Modeling and Capturing 3D Occluders. ACM TOG 27, 5, 131.
28. Levoy, M., and Hanrahan, P. 1996. Light Field Rendering. In Proc. SIGGRAPH, 31--42.
29. Levoy, M., Chen, B., Vaish, V., Horowitz, M., McDowall, I., and Bolas, M. 2004. Synthetic Aperture Confocal Imaging. ACM TOG 23, 3, 825--834.
30. Lippmann, G. 1908. La Photographie Intégrale. Academie des Sciences 146, 446--451.
31. Mann, S., and Picard, R. W. 1995. Being 'Undigital' with Digital Cameras: Extending Dynamic Range by Combining Differently Exposed Pictures. In Proc. IS&T, 442--448.
32. McGuire, M., Matusik, W., Pfister, H., Chen, B., Hughes, J. F., and Nayar, S. K. 2007. Optical Splitting Trees for High-Precision Monocular Imaging. IEEE CG&A 27, 2, 32--42.
33. Mitsunaga, T., and Nayar, S. K. 1999. Radiometric Self Calibration. In Proc. CVPR, 374--380.
34. Narasimhan, S., and Nayar, S. 2005. Enhancing Resolution along Multiple Imaging Dimensions using Assorted Pixels. IEEE Trans. PAMI 27, 4, 518--530.
35. Nayar, S., and Mitsunaga, T. 2000. High Dynamic Range Imaging: Spatially Varying Pixel Exposures. In Proc. CVPR, vol. 1, 472--479.
36. Neumann, L., Hegedus, R., Horvath, G., and Garcia, R. 2008. Applications of High Precision Imaging Polarimetry. In Proc. Computational Aesthetics in Graphics, Visualization and Imaging, 89--97.
37. Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., and Hanrahan, P. 2005. Light Field Photography with a Hand-Held Plenoptic Camera. Tech. Rep. Computer Science CSTR 2005-02, Stanford University.
38. Ng, R. 2005. Fourier Slice Photography. ACM TOG 24, 3, 735--744.
39. Okamoto, T., and Yamaguchi, I. 1991. Simultaneous Acquisition of Spectral Image Information. Optics Lett. 16, 16, 1277--1279.
40. Park, J.-I., Lee, M.-H., Grossberg, M. D., and Nayar, S. K. 2007. Multispectral Imaging Using Multiplexed Illumination. In Proc. ICCV, 1--8.
41. Pezzaniti, J. L., Chenault, D., Roche, M., Reinhardt, J., Pezzaniti, J. P., and Schultz, H. 2008. Four Camera Complete Stokes Imaging Polarimeter. In Proc. SPIE 6972, Polarization: Measurement, Analysis, and Remote Sensing VIII, 69720J-1--69720J-12.
42. Reinhard, E., Ward, G., Debevec, P., Pattanaik, S., Heidrich, W., and Myszkowski, K. 2010. High Dynamic Range Imaging: Acquisition, Display and Image-Based Lighting. Morgan Kaufmann Publishers.
43. Reshetouski, I., Manakov, A., Seidel, H.-P., and Ihrke, I. 2011. Three-Dimensional Kaleidoscopic Imaging. In Proc. CVPR, 353--360.
44. Rump, M., and Klein, R. 2010. Spectralization: Reconstructing spectra from sparse data. In Proc. EGSR, 1347--1354.
45. Schechner, Y., and Nayar, S. 2005. Generalized Mosaicing: Polarization Panorama. IEEE Trans. PAMI 27, 4, 631--636.
46. Schechner, Y., Narasimhan, S. G., and Nayar, S. K. 2001. Instant Dehazing of Images using Polarization. In Proc. CVPR, 325--332.
47. Spiering, B. A., 1999. Multispectral Imaging System. US Patent 5,900,942.
48. Tocci, M. D., Kiser, C., Tocci, N., and Sen, P. 2011. A Versatile HDR Video Production System. ACM TOG 30, 4.
49. Toyooka, S., and Hayasaka, N. 1997. Two-Dimensional Spectral Analysis using Broad-Band Filters. Optical Communications 137 (Apr), 22--26.
50. Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., and Tumblin, J. 2007. Dappled Photography: Mask Enhanced Cameras For Heterodyned Light Fields and Coded Aperture Refocussing. ACM TOG 26, 3, 69.
51. Wanner, S., and Goldluecke, B. 2012. Globally Consistent Depth Labeling of 4D Lightfields. In Proc. CVPR, 41--48.
52. Wanner, S., and Goldluecke, B. 2012. Spatial and Angular Variational Super-Resolution of 4D Light Fields. In Proc. ECCV, 608--621.
53. Wetzstein, G., Ihrke, I., Lanman, D., and Heidrich, W. 2011. Computational Plenoptic Imaging. CGF 30, 8, 2397--2426.
54. Wilburn, B., Joshi, N., Vaish, V., et al. 2005. High Performance Imaging using Large Camera Arrays. ACM TOG 24, 3, 765--776.
55. Zhou, C., and Nayar, S. 2011. Computational Cameras: Convergence of Optics and Processing. IEEE Trans. IP 20, 12 (Dec), 3322--3340.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition