Coding Scheme Optimization for Fast Fluorescence Lifetime Imaging

Abstract

Fluorescence lifetime imaging (FLIM) is used for measuring material properties in a wide range of applications, including biology, medical imaging, chemistry, and material science. In frequency-domain FLIM (FD-FLIM), the object of interest is illuminated with a temporally modulated light source. The fluorescence lifetime is measured by computing the correlations of the emitted light with a demodulation function at the sensor. The signal-to-noise ratio (SNR) and the acquisition time of a FD-FLIM system is determined by the coding scheme (modulation and demodulation functions). In this article, we develop theory and algorithms for designing high-performance FD-FLIM coding schemes that can achieve high SNR and short acquisition time, given a fixed source power budget. Based on a geometric analysis of the image formation and noise model, we propose a novel surrogate objective for the performance of a given coding scheme. The surrogate objective is extremely fast to compute, and can be used to efficiently explore the entire space of coding schemes. Based on this objective, we design novel, high-performance coding schemes that achieve up to an order of magnitude shorter acquisition time as compared to existing approaches. We demonstrate the performance advantage of the proposed schemes in a variety of imaging conditions, using a modular hardware prototype that can implement various coding schemes.

References

1. Diego Airado-Rodríguez, Teresa Galeano-Díaz, Isabel Durán-Merás, and Jens Petter Wold. 2009. Usefulness of fluorescence excitation-emission matrices in combination with PARAFAC, as fingerprints of red wines. Journal of Agricultural and Food Chemistry 57, 5 (2009), 1711--172
2. Syed Abdullah Aljunid, Gleb Maslennikov, Yimin Wang, Hoang Lan Dao, Valerio Scarani, and Christian Kurtsiefer. 2013. Excitation of a single atom with exponentially rising light pulses. Physical Review Letters 111, 10 (2013), 103001.
3. Philippe I. H. Bastiaens and Anthony Squire. 1999. Fluorescence lifetime imaging microscopy: Spatial resolution of biochemical processes in the cell. Trends in Cell Biology 9, 2 (1999), 48--52.
4. Wolfgang Becker. 2014. The bh TCSPC Handbook. Becker 8 Hickl.
5. Oscar Beijbom, Tali Treibitz, David I. Kline, Gal Eyal, Adi Khen, Benjamin Neal, Yossi Loya, B. Greg Mitchell, and David Kriegman. 2016. Improving automated annotation of benthic survey images using wide-band fluorescence. Scientific Reports 6 (2016), 23166.
6. Mikhail Y. Berezin and Samuel Achilefu. 2010. Fluorescence lifetime measurements and biological imaging. Chemical Reviews 110, 5 (2010), 2641--2684.
7. Ayush Bhandari, Christopher Barsi, and Ramesh Raskar. 2015a. Blind and reference-free fluorescence lifetime estimation via consumer time-of-flight sensors. Optica 2, 11 (Nov. 2015), 965--973.
8. Ayush Bhandari, Christopher Barsi, and Ramesh Raskar. 2015b. Blind and reference-free fluorescence lifetime estimation via consumer time-of-flight sensors. Optica 2, 11 (2015), 965--973.
9. Ayush Bhandari and Ramesh Raskar. 2016. Signal processing for time-of-flight imaging sensors: An introduction to inverse problems in computational 3-D imaging. IEEE Signal Processing Magazine 33, 5 (2016), 45--58.
10. M. J. Booth and T. Wilson. 2004. Low-cost, frequency-domain, fluorescence lifetime confocal microscopy. Journal of Microscopy 214, 1 (2004), 36--42.
11. A. Colasanti, A. Kisslinger, G. Fabbrocini, R. Liuzzi, M. Quarto, P. Riccio, G. Roberti, and F. Villani. 2000. MS-2 fibrosarcoma characterization by laser induced autofluorescence. Lasers in Surgery and Medicine 26, 5 (2000), 441--448.
12. Daniela Comelli, Cosimo D’Andrea, Gianluca Valentini, Rinaldo Cubeddu, Chiara Colombo, and Lucia Toniolo. 2004. Fluorescence lifetime imaging and spectroscopy as tools for nondestructive analysis of works of art. Applied Optics 43, 10 (2004), 2175--2183.
13. Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation 6, 2 (2002), 182--197.
14. Michelle A. Digman, Valeria R. Caiolfa, Moreno Zamai, and Enrico Gratton. 2008. The phasor approach to fluorescence lifetime imaging analysis. Biophysical Journal 94, 2 (2008), L14--L16.
15. L. A. Donaldson and K. Radotic. 2013. Fluorescence lifetime imaging of lignin autofluorescence in normal and compression wood. Journal of Microscopy 251, 2 (2013), 178--187.
16. Alan Elder, Simon Schlachter, and Clemens F. Kaminski. 2008. Theoretical investigation of the photon efficiency in frequency-domain fluorescence lifetime imaging microscopy. JOSA A 25, 2 (2008), 452--462.
17. Dan Elson, Jose Requejo-Isidro, Ian Munro, Fred Reavell, Jan Siegel, Klaus Suhling, Paul Tadrous, Richard Benninger, Peter Lanigan, James McGinty, et al. 2004. Time-domain fluorescence lifetime imaging applied to biological tissue. Photochemical 8 Photobiological Sciences 3, 8 (2004), 795--801.
18. Alessandro Esposito, Hans Gerritsen, Thierry Oggier, Felix Lustenberger, and Fred S. Wouters. 2006. Innovating lifetime microscopy: A compact and simple tool for life sciences, screening, and diagnostics. Journal of Biomedical Optics 11, 3 (2006), 034016.
19. Alessandro Esposito, Hans C. Gerritsen, and Fred S. Wouters. 2007. Optimizing frequency-domain fluorescence lifetime sensing for high-throughput applications: Photon economy and acquisition speed. JOSA A 24, 10 (2007), 3261--3273.
20. A. Esposito, T. Oggier, H. C. Gerritsen, F. Lustenberger, and F. S. Wouters. 2005. All-solid-state lock-in imaging for wide-field fluorescence lifetime sensing. Optics Express 13, 24 (2005), 9812--9821.
21. Ying Fu, Antony Lam, Yasuyuki Matsushita, Imari Sato, and Yoichi Sato. 2014. Interreflection removal using fluorescence. In European Conference on Computer Vision. Springer, 203--217.
22. Mohit Gupta, Andreas Velten, Shree K. Nayar, and Eric Breitbach. 2018. What are optimal coding functions for time-of-flight imaging? ACM Transactions on Graphics (TOG) 37, 2 (2018), 13.
23. Shuai Han, Yasuyuki Matsushita, Imari Sato, Takahiro Okabe, and Yoichi Sato. 2012. Camera spectral sensitivity estimation from a single image under unknown illumination by using fluorescence. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). IEEE, 805--812.
24. Miles Hansard, Seungkyu Lee, Ouk Choi, and Radu Patrice Horaud. 2012. Time-of-Flight Cameras: Principles, Methods and Applications. Springer Science 8 Business Media.
25. Samuel W. Hasinoff, Frédo Durand, and William T. Freeman. 2010. Noise-optimal capture for high dynamic range photography. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’10). IEEE, 553--560.
26. Felix Heide, Matthias B. Hullin, James Gregson, and Wolfgang Heidrich. 2013. Low-budget transient imaging using photonic mixer devices. ACM Transactions on Graphics (ToG) 32, 4 (2013), 45.
27. Felix Heide, Lei Xiao, Andreas Kolb, Matthias B. Hullin, and Wolfgang Heidrich. 2014. Imaging in scattering media using correlation image sensors and sparse convolutional coding. Optics Express 22, 21 (2014), 26338--26350.
28. J. Michael Hollas. 2004. Modern Spectroscopy. John Wiley 8 Sons.
29. Matthias B. Hullin, Martin Fuchs, Ivo Ihrke, Hans-Peter Seidel, and Hendrik P. A. Lensch. 2008. Fluorescent immersion range scanning. ACM Transactions on Graphics 27, 3 (Aug. 2008), Article 87, 10 pages.
30. Matthias B. Hullin, Johannes Hanika, Boris Ajdin, Hans-Peter Seidel, Jan Kautz, and Hendrik P. A. Lensch. 2010. Acquisition and analysis of bispectral bidirectional reflectance and reradiation distribution functions. ACM Transactions on Graphics 29, 4 (July 2010), Article 97, 7 pages.
31. Moon S. Kim, Byoung-Kwan Cho, Alan M. Lefcourt, Yud-Ren Chen, and Sukwon Kang. 2008. Multispectral fluorescence lifetime imaging of feces-contaminated apples by time-resolved laser-induced fluorescence imaging system with tunable excitation wavelengths. Applied Optics 47, 10 (2008), 1608--1616.
32. Joseph R. Lakowicz. 2006. Principles of Fluorescence Spectroscopy. Springer.
33. R. Lange. 2000. 3D Time of Flight Distance Measurement with Custom Solid State Image Sensors in CMOS, CCD Technology. https://books.google.com/books?id=upMuHwAACAAJ
34. Lea Lenhardt, Rasmus Bro, Ivana Zeković, Tatjana Dramićanin, and Miroslav D. Dramićanin. 2015. Fluorescence spectroscopy coupled with PARAFAC and PLS DA for characterization and classification of honey. Food Chemistry 175 (2015), 284--291.
35. Yuxiang Lin and Arthur F. Gmitro. 2010. Statistical analysis and optimization of frequency-domain fluorescence lifetime imaging microscopy using homodyne lock-in detection. JOSA A 27, 5 (2010), 1145--1155.
36. Douglas Magde, Gail E. Rojas, and Paul G. Seybold. 1999. Solvent dependence of the fluorescence lifetimes of xanthene dyes. Photochemistry and Photobiology 70, 5 (1999), 737--744.
37. John A. Nelder and Roger Mead. 1965. A simplex method for function minimization. The Computer Journal 7, 4 (1965), 308--313.
38. Austin Nevin, Daniela Comelli, Gianluca Valentini, Demetrios Anglos, Aviva Burnstock, Sharon Cather, and Rinaldo Cubeddu. 2007. Time-resolved fluorescence spectroscopy and imaging of proteinaceous binders used in paintings. Analytical and Bioanalytical Chemistry 388, 8 (2007), 1897--1905.
39. Lucio Pancheri, Nicola Massari, and David Stoppa. 2013. SPAD image sensor with analog counting pixel for time-resolved fluorescence detection. IEEE Transactions on Electron Devices 60, 10 (2013), 3442--3449.
40. Johan Philip and Kjell Carlsson. 2003. Theoretical investigation of the signal-to-noise ratio in fluorescence lifetime imaging. JOSA A 20, 2 (2003), 368--379.
41. Asima Pradhan, Prabir Pal, Gilles Durocher, Luc Villeneuve, Antonia Balassy, Féridoun Babai, Louis Gaboury, and Louise Blanchard. 1995. Steady state and time-resolved fluorescence properties of metastatic and non-metastatic malignant cells from different species. Journal of Photochemistry and Photobiology B: Biology 31, 3 (1995), 101--112.
42. J. Requejo-Isidro, J. McGinty, I. Munro, D. S. Elson, N. P. Galletly, M. J. Lever, M. A. A. Neil, G. W. H. Stamp, P. M. W. French, P. A. Kellett, et al. 2004. High-speed wide-field time-gated endoscopic fluorescence-lifetime imaging. Optics Letters 29, 19 (2004), 2249--2251.
43. Imari Sato, Takahiro Okabe, and Yoichi Sato. 2012. Bispectral photometric stereo based on fluorescence. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). IEEE, 270--277.
44. S. Schlachter, A. D. Elder, A. Esposito, G. S. Kaminski, J. H. Frank, L. K. Van Geest, and C. F. Kaminski. 2009. mhFLIM: Resolution of heterogeneous fluorescence decays in widefield lifetime microscopy. Optics Express 17, 3 (2009), 1557--1570.
45. L. K. Seah, U. S. Dinish, W. F. Phang, Z. X. Chao, and V. M. Murukeshan. 2005. Fluorescence optimisation and lifetime studies of fingerprints treated with magnetic powders. Forensic Science International 152, 2–3 (2005), 249--257.
46. L. K. Seah, P. Wang, V. M. Murukeshan, and Z. X. Chao. 2006. Application of fluorescence lifetime imaging (FLIM) in latent finger mark detection. Forensic Science International 160, 2–3 (2006), 109--114.
47. Nathan C. Shaner, Gerard G. Lambert, Andrew Chammas, Yuhui Ni, Paula J. Cranfill, Michelle A. Baird, Brittney R. Sell, John R. Allen, Richard N. Day, Maria Israelsson, et al. 2013. A bright monomeric green fluorescent protein derived from Branchiostoma lanceolatum. Nature Methods 10, 5 (2013), 407.
48. Wen Shi, Xiaohua Li, and Huimin Ma. 2014. Fluorescent probes and nanoparticles for intracellular sensing of pH values. Methods and Applications in Fluorescence 2, 4 (2014), 042001.
49. Ewa Sikorska, Tomasz Górecki, Igor V. Khmelinskii, Marek Sikorski, and Jacek Kozioł. 2005. Classification of edible oils using synchronous scanning fluorescence spectroscopy. Food Chemistry 89, 2 (2005), 217--225.
50. Shiwen Sun, Birgit Ungerböck, and Torsten Mayr. 2015. Imaging of oxygen in microreactors and microfluidic systems. Methods and Applications in Fluorescence 3, 3 (2015), 034002.
51. Virginia Torczon. 1997. On the convergence of pattern search algorithms. SIAM Journal on Optimization 7, 1 (1997), 1--25.
52. Tali Treibitz, Zak Murez, B. Greg Mitchell, and David Kriegman. 2012. Shape from fluorescence. In European Conference on Computer Vision. Springer, 292--306.
53. Andreas Velten, Thomas Willwacher, Otkrist Gupta, Ashok Veeraraghavan, Moungi G. Bawendi, and Ramesh Raskar. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature Communications 3 (2012), 745.
54. Charles W. Wilkerson Jr., Peter M. Goodwin, W. Patrick Ambrose, John C. Martin, and Richard A. Keller. 1993. Detection and lifetime measurement of single molecules in flowing sample streams by laser-induced fluorescence. Applied Physics Letters 62, 17 (1993), 2030--2032.
55. Di Wu, Matthew O’Toole, Andreas Velten, Amit Agrawal, and Ramesh Raskar. 2012. Decomposing global light transport using time of flight imaging. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). IEEE, 366--373.
56. Qiaole Zhao, Ian T. Young, and Jan Geert Sander De Jong. 2011. Photon budget analysis for fluorescence lifetime imaging microscopy. Journal of Biomedical Optics 16, 8 (2011), 086007.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

        • Computational photography