Low-cost SPAD sensing for non-line-of-sight tracking, material classification and depth imaging

Abstract

Time-correlated imaging is an emerging sensing modality that has been shown to enable promising application scenarios, including lidar ranging, fluorescence lifetime imaging, and even non-line-of-sight sensing. A leading technology for obtaining time-correlated light measurements are single-photon avalanche diodes (SPADs), which are extremely sensitive and capable of temporal resolution on the order of tens of picoseconds. However, the rare and expensive optical setups used by researchers have so far prohibited these novel sensing techniques from entering the mass market. Fortunately, SPADs also exist in a radically cheaper and more power-efficient version that has been widely deployed as proximity sensors in mobile devices for almost a decade. These commodity SPAD sensors can be obtained at a mere few cents per detector pixel. However, their inferior data quality and severe technical drawbacks compared to their high-end counterparts necessitate the use of additional optics and suitable processing algorithms. In this paper, we adopt an existing evaluation platform for commodity SPAD sensors, and modify it to unlock time-of-flight (ToF) histogramming and hence computational imaging. Based on this platform, we develop and demonstrate a family of hardware/software systems that, for the first time, implement applications that had so far been limited to significantly more advanced, higher-priced setups: direct ToF depth imaging, non-line-of-sight object tracking, and material classification.

References

1. Nils Abramson. 1978. Light-in-flight recording by holography. Optics letters 3, 4 (1978), 121--123.
2. Yoann Altmann, Stephen McLaughlin, Miles J Padgett, Vivek K Goyal, Alfred O Hero, and Daniele Faccio. 2018. Quantum-inspired computational imaging. Science 361, 6403 (2018).
3. Victor Arellano, Diego Gutierrez, and Adrian Jarabo. 2017. Fast back-projection for non-line of sight reconstruction. Opt. Express 25, 10 (2017), 11574--11583.
4. David Patrick Baxter. 2015. Application using a single photon avalanche diode (SPAD). (June 16 2015). US Patent 9,058,081.
5. Katherine L Bouman, Vickie Ye, Adam B Yedidia, Frédo Durand, Gregory W Wornell, Antonio Torralba, and William T Freeman. 2017. Turning corners into cameras: Principles and methods. In IEEE International Conference on Computer Vision (ICCV). 2289--2297.
6. Samuel Burri, Harald Homulle, Claudio Bruschini, and Edoardo Charbon. 2016. LinoSPAD: a time-resolved 256x1 CMOS SPAD line sensor system featuring 64 FPGA-based TDC channels running at up to 8.5 giga-events per second. Proc. SPIE 9899 (2016), 98990D-10.
7. Mauro Buttafava, Jessica Zeman, Alberto Tosi, Kevin Eliceiri, and Andreas Velten. 2015. Non-line-of-sight imaging using a time-gated single photon avalanche diode. Optics express 23, 16 (2015), 20997--21011.
8. Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. 1995. A limited memory algorithm for bound constrained optimization. SIAM Journal on scientific computing 16, 5 (1995), 1190--1208.
9. C. Callenberg, A. Lyons, D. den Brok, A. Fatima, A. Turpin, V. Zickus, L. Machesky, J. Whitelaw, D. Faccio, and M. B. Hullin. 2021. Super-resolution time-resolved imaging using computational sensor fusion. Scientific Reports 11, 1 (Jan. 2021), 1689.
10. Barbara Caputo, Eric Hayman, and P Mallikarjuna. 2005. Class-specific material categorisation. In Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, Vol. 2. IEEE, 1597--1604.
11. Edoardo Charbon. 2008. Towards large scale CMOS single-photon detector arrays for lab-on-chip applications. Journal of Physics D: Applied Physics 41, 9 (2008), 094010.
12. Wenzheng Chen, Simon Daneau, Fahim Mannan, and Felix Heide. 2019. Steady-state non-line-of-sight imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6790--6799.
13. Wenzheng Chen, Fangyin Wei, Kiriakos N Kutulakos, Szymon Rusinkiewicz, and Felix Heide. 2020. Learned feature embeddings for non-line-of-sight imaging and recognition. ACM Trans. Graph. 39, 6 (2020), 1--18.
14. Liang Gao, Jinyang Liang, Chiye Li, and Lihong V Wang. 2014. Single-shot compressed ultrafast photography at one hundred billion frames per second. Nature 516, 7529 (2014), 74.
15. Genevieve Gariepy, Nikola Krstajić, Robert Henderson, Chunyong Li, Robert R Thomson, Gerald S Buller, Barmak Heshmat, Ramesh Raskar, Jonathan Leach, and Daniele Faccio. 2015. Single-photon sensitive light-in-fight imaging. Nature communications 6 (2015), 6021.
16. Javier Grau Chopite, Matthias B. Hullin, Michael Wand, and Julian Iseringhausen. 2020. Deep Non-Line-of-Sight Reconstruction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
17. Felix Heide, Matthias B. Hullin, James Gregson, and Wolfgang Heidrich. 2013. Low-Budget Transient Imaging using Photonic Mixer Devices. ACM Trans. Graph. 32, 4 (2013), 45:1--45:10.
18. Felix Heide, Matthew O'Toole, Kai Zang, David B Lindell, Steven Diamond, and Gordon Wetzstein. 2019. Non-line-of-sight imaging with partial occluders and surface normals. ACM Trans. Graph. 38, 3 (2019), 1--10.
19. Felix Heide, Lei Xiao, Wolfgang Heidrich, and Matthias B Hullin. 2014. Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3222--3229.
20. Robert K. Henderson, Nick Johnston, Haochang Chen, David Day Uei Li, Graham Hungerford, Richard Hirsch, David McLoskey, Philip Yip, and David J.S. Birch. 2018. A 192X128 Time Correlated Single Photon Counting Imager in 40nm CMOS Technology. ESSCIRC 2018 - IEEE 44th European Solid State Circuits Conference (2018), 54--57.
21. Quercus Hernandez, Diego Gutierrez, and Adrian Jarabo. 2017. A Computational Model of a Single-Photon Avalanche Diode Sensor for Transient Imaging. (2017). arXiv:physics.ins-det/1703.02635
22. Atul Ingle, Andreas Velten, and Mohit Gupta. 2019. High Flux Passive Imaging With Single-Photon Sensors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
23. Julian Iseringhausen and Matthias Hullin. 2020. Non-line-of-sight reconstruction using efficient transient rendering. ACM Trans. Graph. 39, 1 (2020), 1--14.
24. Adrian Jarabo, Belen Masia, Julio Marco, and Diego Gutierrez. 2017. Recent advances in transient imaging: A computer graphics and vision perspective. Visual Informatics 1, 1 (2017), 65--79.
25. A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar. 2013. Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles. ACM Trans. Graph. 32, 6 (2013).
26. Jonathan Klein, Christoph Peters, Jaime Martín, Martin Laurenzis, and Matthias B Hullin. 2016. Tracking objects outside the line of sight using 2D intensity images. Scientific reports 6 (2016), 32491.
27. Dilip Krishnan and Rob Fergus. 2009. Fast Image Deconvolution using Hyper-Laplacian Priors. In Advances in Neural Information Processing Systems 22, Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, and A. Culotta (Eds.). Curran Associates, Inc., 1033--1041.
28. Jinyang Liang, Cheng Ma, Liren Zhu, Yujia Chen, Liang Gao, and Lihong V Wang. 2017. Single-shot real-time video recording of a photonic Mach cone induced by a scattered light pulse. Science advances 3, 1 (2017), e1601814.
29. David B Lindell, Matthew O'Toole, and Gordon Wetzstein. 2018. Single-photon 3D imaging with deep sensor fusion. ACM Trans. Graph. 37, 4 (2018), 113.
30. David B. Lindell, Gordon Wetzstein, and Matthew O'Toole. 2019. Wave-based non-line-of-sight imaging using fast f-k migration. ACM Trans. Graph. 38, 4 (2019), 116.
31. Ce Liu, Lavanya Sharan, Edward H Adelson, and Ruth Rosenholtz. 2010. Exploring features in a bayesian framework for material recognition. In 2010 ieee computer society conference on computer vision and pattern recognition. IEEE, 239--246.
32. Xiaochun Liu, Ibón Guillén, Marco La Manna, Ji Hyun Nam, Syed Azer Reza, Toan Huu Le, Adrian Jarabo, Diego Gutierrez, and Andreas Velten. 2019. Non-line-of-sight imaging using phasor-field virtual wave optics. Nature (2019), 1--4.
33. Andreas Meuleman, Seung-Hwan Baek, Felix Heide, and Min H Kim. 2020. Single-Shot Monocular RGB-D Imaging Using Uneven Double Refraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2465--2474.
34. Nikhil Naik, Shuang Zhao, Andreas Velten, Ramesh Raskar, and Kavita Bala. 2011. Single View Reflectance Capture Using Multiplexed Scattering and Time-of-flight Imaging. ACM Trans. Graph. 30, 6 (2011), 171:1--171:10.
35. Desmond O'Connor. 2012. Time-correlated single photon counting. Academic Press.
36. Matthew O'Toole, Felix Heide, Lei Xiao, Matthias B. Hullin, Wolfgang Heidrich, and Kiriakos N. Kutulakos. 2014. Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport. ACM Trans. Graph. 33, 4 (Aug. 2014).
37. Matthew O'Toole, David B. Lindell, and Gordon Wetzstein. 2018. Confocal Non-line-of-sight imaging based on the light cone transform. Nature (2018), 338--341. Issue 555.
38. Christoph Peters, Jonathan Klein, Matthias B. Hullin, and Reinhard Klein. 2015. Solving Trigonometric Moment Problems for Fast Transient Imaging. ACM Trans. Graph. 34, 6 (Nov. 2015).
39. Corneliu Rablau. 2019. LIDAR-A new (self-driving) vehicle for introducing optics to broader engineering and non-engineering audiences. In Education and Training in Optics and Photonics. Optical Society of America, 11143_138.
40. Sabbir Rangwala. 2020. The iPhone 12 - LiDAR At Your Fingertips. Forbes (12 November 2020). https://www.forbes.com/sites/sabbirrangwala/2020/11/12/the-iphone-12lidar-at-your-fingertips/
41. Justin A. Richardson, Lindsay A. Grant, and Robert K. Henderson. 2009. Low dark count single-photon avalanche diode structure compatible with standard nanometer scale CMOS technology. IEEE Photonics Technology Letters 21, 14 (2009), 1020--1022.
42. N. Scheiner, F. Kraus, F. Wei, B. Phan, F. Mannan, N. Appenrodt, W. Ritter, J. Dickmann, K. Dietmayer, B. Sick, et al. 2020. Seeing Around Street Corners: Non-Line-of-Sight Detection and Tracking In-the-Wild Using Doppler Radar. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2068--2077.
43. Brent Schwarz. 2010. LIDAR: Mapping the world in 3D. Nat. Photonics 4, 7 (2010), 1749--4885.
44. Nikolai Smolyanskiy, Alexey Kamenev, and Stan Birchfield. 2018. On the importance of stereo for accurate depth estimation: An efficient semi-supervised deep neural network approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1007--1015.
45. STMicroelectronics. 2019. STMicroelectronics Ships 1 Billionth Time-of-Flight Module. (26 November 2019). https://www.st.com/content/st_com/en/about/media-center/press-item.html/t4210.html Press release.
46. Shuochen Su, Felix Heide, Robin Swanson, Jonathan Klein, Clara Callenberg, Matthias Hullin, and Wolfgang Heidrich. 2016. Material Classification Using Raw Time-Of-Flight Measurements. In Proc. IEEE CVPR.
47. Kenichiro Tanaka, Yasuhiro Mukaigawa, Takuya Funatomi, Hiroyuki Kubo, Yasuyuki Matsushita, and Yasushi Yagi. 2017. Material classification using frequency-and depth-dependent time-of-flight distortion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 79--88.
48. Chia-Yin Tsai, Aswin C Sankaranarayanan, and Ioannis Gkioulekas. 2019. Beyond Volumetric Albedo-A Surface Optimization Framework for Non-Line-Of-Sight Imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1545--1555.
49. Manik Varma and Andrew Zisserman. 2008. A statistical approach to material classification using image patch exemplars. IEEE transactions on pattern analysis and machine intelligence 31, 11 (2008), 2032--2047.
50. Andreas Velten, Thomas Willwacher, Otkrist Gupta, Ashok Veeraraghavan, Moungi G. Bawendi, and Ramesh Raskar. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature Communications 3 (2012), 745.
51. Andreas Velten, Di Wu, Adrian Jarabo, Belen Masia, Christopher Barsi, Chinmaya Joshi, Everett Lawson, Moungi Bawendi, Diego Gutierrez, and Ramesh Raskar. 2013. Femto-photography: capturing and visualizing the propagation of light. ACM Trans. Graph. 32, 4 (2013), 44.
52. Junko Yoshida. 2018. ST & Apple, through Thick and Thin. EETimes (22 August 2018). https://www.eetimes.com/st-apple-through-thick-and-thin/
53. Franco Zappa, Simone Tisa, Alberto Tosi, and Sergio Cova. 2007. Principles and features of single-photon avalanche diode arrays. Sensors and Actuators A: Physical 140, 1 (2007), 103--112.
54. Vytautas Zickus, Ming-Lo Wu, Kazuhiro Morimoto, Valentin Kapitany, Areeba Fatima, Alex Turpin, Robert Insall, Jamie Whitelaw, Laura Machesky, Claudio Bruschini, et al. 2020. Fluorescence lifetime imaging with a megapixel SPAD camera and neural network lifetime estimation. Scientific Reports 10, 1 (2020), 1--10.

Index Terms

• Computing methodologies

  • Computer graphics

    • Image manipulation

      • Computational photography

  • Machine learning

    • Machine learning approaches

      • Neural networks

• Hardware

  • Communication hardware, interfaces and storage

    • Sensor applications and deployments