Light field microscopy

Abstract

By inserting a microlens array into the optical train of a conventional microscope, one can capture light fields of biological specimens in a single photograph. Although diffraction places a limit on the product of spatial and angular resolution in these light fields, we can nevertheless produce useful perspective views and focal stacks from them. Since microscopes are inherently orthographic devices, perspective views represent a new way to look at microscopic specimens. The ability to create focal stacks from a single photograph allows moving or light-sensitive specimens to be recorded. Applying 3D deconvolution to these focal stacks, we can produce a set of cross sections, which can be visualized using volume rendering. In this paper, we demonstrate a prototype light field microscope (LFM), analyze its optical performance, and show perspective views, focal stacks, and reconstructed volumes for a variety of biological specimens. We also show that synthetic focusing followed by 3D deconvolution is equivalent to applying limited-angle tomography directly to the 4D light field.

References

1. Adelson, T., Wang, J. Y. A. 1992. Single lens stereo with a plenoptic camera. IEEE Transactions on Pattern Analysis and Machine Intelligence 14, 2, 99--106.
2. Agard, D. A. 1984. Optical sectioning microscopy: Cellular architecture in three dimensions. Ann. Rev. Biophys. Bioeng 13, 191--219.
3. Andersen, A. H., Kak, A. C., 1984. Simultaneous algebraic reconstruction technique (SART): A superior implementation of the ART algorithm. Ultrasonic Imaging 6, 81--94.
4. Arridge, S. R. 2001. Methods for the inverse problem in optical tomography. Proc. Waves and Imaging Through Complex Media. Kluwer, 307--329.
5. Castleman, K. R. 1979. Digital Image Processing. Prentice Hall.
6. Chamgoulov, R. O., Lane, P. M., Macaulay, C. E. 2004. Optical computed-tomography microscope using digital spatial light modulation. Proc. SPIE 5324, 182--190.
7. Colsher, J. G. 1980. Fully three-dimensional positron emission tomography. Phys. Med. Biol. 25, 1, 103--115.
8. Corle, T. R., Kino, G. S. 1996. Confocal Scanning Optical Microscopy and Related Imaging Systems. Academic Press.
9. Ellis, G. W. 1966. Holomicrography: transformation of image during reconstruction a posteriori. Science 143, 1195--1196.
10. Goldberg, N. 1992. Camera technology: the dark side of the lens. Academic Press.
11. Goodman, J. 1996. Introduction to Fourier optics. 2nd edition, McGraw-Hill.
12. Gustafsson, M. G. L. 2005. Nonlinear structured-illumination microscopy: Wide-field fluorescence imaging with theoretically unlimited resolution. Proc. National Academy of Sciences 102, 37.
13. Holmes, T. J., Bhattacharyya, S., et al. 1995. Light microscopic images reconstructed by maximum likelihood deconvolution. In Handbook of Biological Confocal Microscopy, ed. J. B. Pawley, Plenum Press, 389--402.
14. Inoue, S., Oldenbourg, R. 1995. Microscopes. In Handbook of Optics, 2nd edition, McGraw-Hill.
15. Inoue, S. and Spring, K. R. 1997. Video Microscopy. 2nd edition, Plenum Press.
16. Gabor, D. 1948. A new microscopic principle. Nature 161, 777--778.
17. Isaksen, A., Mcmillan, L., Gortler, S. J. 2000. Dynamically reparameterized light fields. Proc. SIGGRAPH 2000.
18. Javidi, B., Okano, F., eds. 2002. Three-Dimensional Television, Video and Display Technologies. Springer-Verlag.
19. Kak, A. C., Slaney, M. 1988. Principles of Computerized Tomographic Imaging. IEEE Press.
20. Kawata, S., Nakamura, 0., Minami, S. 1987. Optical microscope tomography. I. Support constraint. J. Opt. Soc. Am. A 4, 1, 292--297.
21. Kingslake, R. 1983. Optical system design. Academic Press.
22. Levoy, M., Hanrahan, P. 1996. Light field rendering. Proc. SIGGRAPH 1996.
23. Levoy, M., Chen, B., Vaish, V., Horowitz, M., Mcdowall, I., Bolas, M. 2004. Synthetic aperture confocal imaging. ACM Transactions on Graphics (Proc. SIGGRAPH) 23, 3, 825--834.
24. Markham, J., Conchello, J.-A. 2001. Artefacts in restored images due to intensity loss in three-dimensional fluorescence microscopy. J. Microscopy 204, 2, 93--98.
25. Mcnally, J. G., Preza, C., Conchello, J. A., Thomas, L. J. Jr. 1994. Artifacts in computational optical-sectioning microscopy. J. Opt. Soc. Am. A 11, 3, 1056--67.
26. Nayar, S. K., Nakagawa, Y. 1990. Shape from focus: An effective approach for rough surfaces. Proc. International Conference on Robotics and Automation (ICRA), Vol. 2, 218--225.
27. Nayar, S. K., Narasimhan, S. G. 2002. Assorted pixels: Multi-sampled imaging with structural models. Proc. ECCV.
28. Ng, R. 2005. Fourier slice photography. ACM Transactions on Graphics (Proc. SIGGRAPH) 24, 3, 735--744.
29. Ng, R., Levoy, M., Bredif, M., Duval, G., Horowitz, M., Hanrahan, P. 2005. Light Field Photography with a Hand-Held Plenoptic Camera. Stanford Tech Report CTSR 2005-02.
30. Ng, R. 2006. Digital Light Field Photography. PhD dissertation, Stanford University.
31. Noguchi, M., Nayar, S. 1994. Microscopic shape from focus using active illumination. Proc. IAPR International Conference on Pattern Recognition (ICPR), Vol. A, 147--152.
32. Okoshi, T. 1976. Three-Dimensional Imaging Techniques. Academic Press.
33. Piller, H. 1977. Microscope Photometry. Springer-Verlag.
34. Pluta, M. 1988. Advanced Light Microscopy (in 3 volumes). Elsevier.
35. Schechner, Y., Kiryati, N. 1999. The optimal axial interval for estimating depth from defocus. Proc. ICCV.
36. Schechner, Y., Kiryati, N. 2000. Depth from defocus vs. stereo: How different really are they? IJCV 39, 2, 141--162.
37. Schechner, Y. Y., Kiryati, N., Basri, R. 2000. Separation of transparent layers using focus. IJCV 39, 1, 25--39.
38. Schechner, Y., Nayar, S. 2001. Generalized Mosaicing. Proc. ICCV.
39. Shah, U. B., Nayar, S. K. 1992. Extracting 3-D structure and focused images using an optical microscope. Proc. IEEE Symposium on Computer-Based Medical Systems.
40. Streibl, N. 1984. Depth transfer by an imaging system. Optica Acta 31, 11, 1233--1241.
41. Streibl, N. 1985. Three-dimensional imaging by a microscope. J. Opt. Soc. Am. A 2, 2, 121--127.
42. Swedlow, J. R., Sedat, J. W., Agard, D. A. 1997. Deconvolution in optical microscopy. In Deconvolution of Images and Spectra, ed. P. A. Jansson, Academic Press, 284--309.
43. Vaish, V., Garg, G., Talvala, E., Antunez, E., Wilburn, B., Horowitz, M., Levoy, M., Synthetic aperture focusing using a shear-warp factorization of the viewing transform. Proc. Workshop on Advanced 3D Imaging for Safety and Security, in conjunction with CVPR 2005.
44. Weinstein, R. S., Descour, M. R., et al. 2004. An array microscope for ultrarapid virtual slide processing and telepathology. Design, fabrication, and validation study. Human Pathology 35, 11, 1303--1314.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition