Low-budget transient imaging using photonic mixer devices

Abstract

Transient imaging is an exciting a new imaging modality that can be used to understand light propagation in complex environments, and to capture and analyze scene properties such as the shape of hidden objects or the reflectance properties of surfaces.

Unfortunately, research in transient imaging has so far been hindered by the high cost of the required instrumentation, as well as the fragility and difficulty to operate and calibrate devices such as femtosecond lasers and streak cameras.

In this paper, we explore the use of photonic mixer devices (PMD), commonly used in inexpensive time-of-flight cameras, as alternative instrumentation for transient imaging. We obtain a sequence of differently modulated images with a PMD sensor, impose a model for local light/object interaction, and use an optimization procedure to infer transient images given the measurements and model. The resulting method produces transient images at a cost several orders of magnitude below existing methods, while simultaneously simplifying and speeding up the capture process.

References

1. Abramson, N. 1978. Light-in-flight recording by holography. Optics Letters 3, 4, 121--123.
2. Abramson, N. 1983. Light-in-flight recording: high-speed holographic motion pictures of ultrafast phenomena. Applied optics 22, 2, 215--232.
3. Audet, C., and Dennis Jr, J. 2006. Mesh adaptive direct search algorithms for constrained optimization. SIAM Journal on optimization 17, 1, 188--217.
4. Beck, A., and Teboulle, M. 2009. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imag. Sci. 2, 183--202.
5. Chambolle, A., and Pock, T. 2011. A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40, 120--145.
6. Conroy, R., Dorrington, A., Künnemeyer, R., and Cree, M. 2009. Range imager performance comparison in homo-dyne and heterodyne operating modes. In Proc. SPIE, vol. 7239, 723905.
7. Dorrington, A., Cree, M., Payne, A., Conroy, R., and Carnegie, D. 2007. Achieving sub-millimetre precision with a solid-state full-field heterodyning range imaging camera. Measurement Science and Technology 18, 9, 2809.
8. Dorrington, A., Godbaz, J., Cree, M., Payne, A., and Streeter, L. 2011. Separating true range measurements from multi-path and scattering interference in commercial range cameras. In Proc. SPIE, vol. 7864.
9. Hassan, A., Künnemeyer, R., Dorrington, A., and Payne, A. 2010. Proof of concept of diffuse optical tomography using time-of-flight range imaging cameras. In Proc. Electronics New Zealand Conference, 115--120.
10. Huber, P. 1973. Robust regression: asymptotics, conjectures and monte carlo. The Annals of Statistics 1, 5, 799--821.
11. Kirmani, A., Hutchison, T., Davis, J., and Raskar, R. 2009. Looking around the corner using transient imaging. In Proc. ICCV, 159--166.
12. Lange, R., and Seitz, P. 2001. Solid-state time-of-flight range camera. IEEE J. Quantum Electronics 37, 3, 390--397.
13. Lange, R., Seitz, P., Biber, A., and Lauxtermann, S. 2000. Demodulation pixels in CCD and CMOS technologies for time-of-flight ranging. Sensors and camera systems for scientific, industrial, and digital photography applications, 177--188.
14. Naik, N., Zhao, S., Velten, A., Raskar, R., and Bala, K. 2011. Single view reflectance capture using multiplexed scattering and time-of-flight imaging. ACM Trans. Graph. 30, 6, 171.
15. Nilsson, B., and Carlsson, T. E. 1998. Direct three-dimensional shape measurement by digital light-in-flight holography. Applied Optics 37, 34, 7954--7959.
16. Pandharkar, R., Velten, A., Bardagjy, A., Lawson, E., Bawendi, M., and Raskar, R. 2011. Estimating motion and size of moving non-line-of-sight objects in cluttered environments. In Proc. CVPR, 265--272.
17. Schmidt, M., 2013. minConf (retrieved Jan 7, 2013). http://www.di.ens.fr/~mschmidt/Software/minConf.html.
18. Schwarte, R., Xu, Z., Heinol, H., Olk, J., Klein, R., Buxbaum, B., Fischer, H., and Schulte, J. 1997. New electro-optical mixing and correlating sensor: facilities and applications of the photonic mixer device. In Proc. SPIE, vol. 3100, 245--253.
19. Schwarte, R., 1997. Verfahren und Vorrichtung zur Bestimmung der Phasen und/oder Amplitudeninformation einer elektromagnetischen Welle. German Patent DE 19704496.
20. Smith, A., Skorupski, J., and Davis, J. 2008. Transient rendering. Tech. Rep. UCSC-SOE-08-26, University of California Santa Cruz, School of Engineering.
21. Velten, A., Raskar, R., and Bawendi, M. 2011. Picosecond camera for time-of-flight imaging. In OSA Imaging Systems and Applications.
22. Velten, A., Willwacher, T., Gupta, O., Veeraraghavan, A., Bawendi, M., and Raskar, R. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature Communications 3, 745.
23. Velten, A., Wu, D., Jarabo, A., Masia, B., Barsi, C., Joshi, C., Lawson, E., Bawendi, M., Gutierrez, D., and Raskar, R. 2013. Femto-photography: Capturing and visualizing the propagation of light. ACM Trans. Graph. 32.
24. Wan, G., Li, X., Agranov, G., Levoy, M., and Horowitz, W. 2012. CMOS image sensors with multi-bucket pixels for computational photography. IEEE J. Solid-State Circuits 47, 4, 1031--1042.
25. Wu, D., O'Toole, M., Velten, A., Agrawal, A., and Raskar, R. 2012. Decomposing global light transport using time of flight imaging. In Prov. CVPR, 366--373.

Index Terms

• Applied computing

  • Document management and text processing

    • Document capture

      • Document scanning

      • Graphics recognition and interpretation

• Computing methodologies

  • Computer graphics

    • Image manipulation