Primal-dual coding to probe light transport

Abstract

We present primal-dual coding, a photography technique that enables direct fine-grain control over which light paths contribute to a photo. We achieve this by projecting a sequence of patterns onto the scene while the sensor is exposed to light. At the same time, a second sequence of patterns, derived from the first and applied in lockstep, modulates the light received at individual sensor pixels. We show that photography in this regime is equivalent to a matrix probing operation in which the elements of the scene's transport matrix are individually re-scaled and then mapped to the photo. This makes it possible to directly acquire photos in which specific light transport paths have been blocked, attenuated or enhanced. We show captured photos for several scenes with challenging light transport effects, including specular inter-reflections, caustics, diffuse inter-reflections and volumetric scattering. A key feature of primal-dual coding is that it operates almost exclusively in the optical domain: our results consist of directly-acquired, unprocessed RAW photos or differences between them.

References

1. Bekas, C., Kokiopoulou, E., and Saad, Y. 2007. An estimator for the diagonal of a matrix. Appl. Numer. Math. 57, 11--12, 1214--1229.
2. Chandraker, M., Ng, T., and Ramamoorthi, R. 2010. A dual theory of inverse and forward light transport. In Proc. ECCV.
3. Clark, R. N. 2007. Canon 1D Mark II analysis. In http://wwwclarkvisioncom/articles/evaluation-1d2/.
4. Corle, T. R., and Kino, G. S. 1996. Confocal scanning optical microscopy and related imaging systems. Academic Press.
5. Debevec, P., Hawkins, T., Tchou, C., Duiker, H., Sarokin, W., and Sagar, M. 2000. Acquiring the reflectance field of a human face. ACM SIGGRAPH, 145--156.
6. Fuchs, C., Heinz, M., Levoy, M., Scidel, H.-P., and Lensch, H. P. A. 2008. Combining confocal imaging and descattering. Computer Graphics Forum 27, 4, 1245--1253.
7. Garg, G., Talvala, E.-V., Levoy, M., and Lensch, H. P. A. 2006. Symmetric photography: Exploiting data-sparseness in reflectance fields. In Proc. EGSR, 251--262.
8. Ghosh, A., Chen, T., Peers, P., Wilson, C. A., and Debevec, P. 2010. Circularly polarized spherical illumination reflectometry. ACM SIGGRAPH Asia.
9. Gupta, M., Agrawal, A., Veeraraghavan, A., and Narasimhan, S. 2011. Structured light 3D scanning in the presence of global illumination. In Proc. CVPR, 713--720.
10. Heintzmann, R., Hanley, Q., Arndt-Jovin, D., and Jovin, T. 2001. A dual path programmable array microscope (PAM): simultaneous acquisition of conjugate and non-conjugate images. J. Microscopy 204, 119--135.
11. Hitomi, Y., Gu, J., Gupta, M., Mitsunaga, T., and Nayar, S. K. 2011. Video from a single coded exposure photograph using a learned over-complete dictionary. In Proc. ICCV.
12. Iso 2721:1982. Photography---Cameras---Automatic controls of exposure.
13. Kirmani, A., Hutchison, T., Davis, J., and Raskar, R. 2011. Looking around the corner using ultrafast transient imaging. Int. J. Computer Vision 95, 1, 13--28.
14. Levoy, M., Chen, B., Vaish, V., Horowitz, M., Mcdowall, I., and Bolas, M. 2004. Synthetic aperture confocal imaging. ACM SIGGRAPH, 825--834.
15. Levoy, M., Ng, R., Adams, A., Footer, M., and Horowitz, M. 2006. Light field microscopy. ACM SIGGRAPH, 924--934.
16. Mertz, J. 2011. Optical sectioning microscopy with planar or structured illumination. Nat. Meth. 8, 10, 811--819.
17. Nayar, S. K., Branzoi, V., and Boult, T. 2004. Programmable imaging using a digital micromirror array. In Proc. CVPR, 436--443.
18. Nayar, S. K., Krishnan, G., Grossberg, M. D., and Raskar, R. 2006. Fast separation of direct and global components of a scene using high frequency illumination. ACM SIGGRAPH, 935--944.
19. Ng, R., Ramamoorthi, R., and Hanrahan, P. 2003. All-frequency shadows using non-linear wavelet lighting approximation. ACM SIGGRAPH, 376--381.
20. O'Toole, M., and Kutulakos, K. N. 2010. Optical computing for fast light transport analysis. ACM SIGGRAPH Asia.
21. Peers, P., Mahajan, D. K., Lamond, B., Ghosh, A., Matusik, W., Ramamoorthi, R., and Debevec, P. 2009. Compressive light transport sensing. ACM Trans. on Graphics 28, 1.
22. Popoff, S. M., Lerosey, G., Carminati, R., Fink, M., Boccara, A. C., and Gigan, S. 2010. Measuring the transmission matrix in optics. Phys. Rev. Lett. 104, 10.
23. Schechner, Y. Y., Nayar, S. K., and Belhumeur, P. N. 2007. Multiplexing for optimal lighting. IEEE T-PAMI 29, 8, 1339--1354.
24. Seitz, S. M., Matsushita, Y., and Kutulakos, K. N. 2005. A theory of inverse light transport. In Proc. ICCV, 1440--1447.
25. Sen, P., and Darabi, S. 2009. Compressive dual photography. Computer Graphics Forum 28, 2, 609--618.
26. Sen, P., Chen, B., Garg, G., Marschner, S., Horowitz, M., Levoy, M., and Lensch, H. P. A. 2005. Dual photography. ACM SIGGRAPH, 745--755.
27. Sloan, P.-P., Kautz, J., and Snyder, J. 2002. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. ACM SIGGRAPH, 527--536.
28. Tang, J. M., and Saad, Y. 2012. A probing method for computing the diagonal of a matrix inverse. Numer. Linear Algebra Appl. 19, 3, 485--501.
29. Veeraraghavan, A., Reddy, D., and Raskar, R. 2011. Coded strobing photography: compressive sensing of high speed periodic videos. IEEE T-PAMI 33, 4, 671--686.
30. Wang, J., Dong, Y., Tong, X., Lin, Z., and Guo, B. 2009. Kernel Nyström method for light transport. ACM SIGGRAPH.
31. Wetzstein, G., Heidrich, W., and Luebke, D. 2010. Optical image processing using light modulation displays. Computer Graphics Forum 29, 6, 1934--1944.
32. Wilson, T., Juškaitis, R., and Neil, M. 1996. Confocal microscopy by aperture correlation. Optics Letters 21, 23.
33. Zhang, L., and Nayar, S. K. 2006. Projection defocus analysis for scene capture and image display. ACM SIGGRAPH, 907--915.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

        • Epipolar geometry

  • Computer graphics

    • Image manipulation

      • Texturing