Compact snapshot hyperspectral imaging with diffracted rotation

Abstract

Traditional snapshot hyperspectral imaging systems include various optical elements: a dispersive optical element (prism), a coded aperture, several relay lenses, and an imaging lens, resulting in an impractically large form factor. We seek an alternative, minimal form factor of snapshot spectral imaging based on recent advances in diffractive optical technology. We thereupon present a compact, diffraction-based snapshot hyperspectral imaging method, using only a novel diffractive optical element (DOE) in front of a conventional, bare image sensor. Our diffractive imaging method replaces the common optical elements in hyperspectral imaging with a single optical element. To this end, we tackle two main challenges: First, the traditional diffractive lenses are not suitable for color imaging under incoherent illumination due to severe chromatic aberration because the size of the point spread function (PSF) changes depending on the wavelength. By leveraging this wavelength-dependent property alternatively for hyperspectral imaging, we introduce a novel DOE design that generates an anisotropic shape of the spectrally-varying PSF. The PSF size remains virtually unchanged, but instead the PSF shape rotates as the wavelength of light changes. Second, since there is no dispersive element and no coded aperture mask, the ill-posedness of spectral reconstruction increases significantly. Thus, we propose an end-to-end network solution based on the unrolled architecture of an optimization procedure with a spatial-spectral prior, specifically designed for deconvolution-based spectral reconstruction. Finally, we demonstrate hyperspectral imaging with a fabricated DOE attached to a conventional DSLR sensor. Results show that our method compares well with other state-of-the-art hyperspectral imaging methods in terms of spectral accuracy and spatial resolution, while our compact, diffraction-based spectral imaging method uses only a single optical element on a bare image sensor.

References

1. M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zheng. 2016.
2. TensorFlow: A System for Large-scale Machine Learning. In Proc. USENIX Conf. Operating Systems Design and Implementation (OSDI'16). 265--283.
3. Nick Antipa, Grace Kuo, Reinhard Heckel, Ben Mildenhall, Emrah Bostan, Ren Ng, and Laura Waller. 2018. DiffuserCam: lensless single-exposure 3D imaging. Optica 5, 1 (2018), 1--9.
4. Nicholas Antipa, Sylvia Necula, Ren Ng, and Laura Waller. 2016. Single-shot diffuser-encoded light field imaging. In Proc. IEEE Int. Conf. Computational Photography (ICCP 2016). IEEE, 1--11.
5. Boaz Arad and Ohad Ben-Shahar. 2016. Sparse Recovery of Hyperspectral Signal from Natural RGB Images. In Proc. European Conference on Computer Vision (ECCV 2016). Springer, 19--34.
6. M Salman Asif, Ali Ayremlou, Aswin Sankaranarayanan, Ashok Veeraraghavan, and Richard G Baraniuk. 2017. FlatCam: Thin, lensless cameras using coded aperture and computation. IEEE Transactions on Computational Imaging (TCI) 3, 3 (2017), 384--397.
7. Seung-Hwan Baek, Incheol Kim, Diego Gutierrez, and Min H. Kim. 2017. Compact Single-Shot Hyperspectral Imaging Using a Prism. ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2017) 36, 6 (2017).
8. Jose M. Bioucas-Dias and Mario A. T. Figueiredo. 2007. A new TwIST: two-step iterative shrinkage/thresholding for image restoration. IEEE Trans. Image Processing (TIP) 16, 12 (2007), 2992--3004.
9. Nicola Brusco, S Capeleto, M Fedel, Anna Paviotti, Luca Poletto, Guido Maria Cortelazzo, and G Tondello. 2006. A system for 3D modeling frescoed historical buildings with multispectral texture information. Machine Vision and Applications 17, 6 (2006), 373--393.
10. Ayan Chakrabarti and Todd Zickler. 2011. Statistics of real-world hyperspectral images. In Proc. Conference on Computer Vision and Pattern Recognition (CVPR 2011). IEEE, 193--200.
11. Inchang Choi, Daniel S. Jeon, Giljoo Nam, Diego Gutierrez, and Min H. Kim. 2017. High-Quality Hyperspectral Reconstruction Using a Spectral Prior. ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2017) 36, 6 (2017).
12. B. Choudhury, R. Swanson, F. Heide, G. Wetzstein, and W. Heidrich. 2017. Consensus Convolutional Sparse Coding. In Proc. International Conference on Computer Vision (ICCV 2017). 4290--4298.
13. Weisheng Dong, Peiyao Wang, Wotao Yin, and Guangming Shi. 2018. Denoising Prior Driven Deep Neural Network for Image Restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2018), 1--1.
14. M E Gehm, R John, D J Brady, R M Willett, and T J Schulz. 2007. Single-shot compressive spectral imaging with a dual-disperser architecture. OSA OE 15, 21 (2007), 14013--27.
15. Adam Greengard, Yoav Y Schechner, and Rafael Piestun. 2006. Depth from diffracted rotation. Optics letters 31, 2 (2006), 181--183.
16. Ralf Habel, Michael Kudenov, and Michael Wimmer. 2012. Practical spectral photography. In Computer graphics forum, Vol. 31. Wiley Online Library, 449--458.
17. Felix Heide, Qiang Fu, Yifan Peng, and Wolfgang Heidrich. 2016. Encoded diffractive optics for full-spectrum computational imaging. Scientific Reports 6 (2016), 33543.
18. Daniel S Jeon, Inchang Choi, and Min H Kim. 2016. Multisampling Compressive Video Spectroscopy. Computer Graphics Forum 35, 2 (2016), 467--477.
19. William R Johnson, Daniel W Wilson, Wolfgang Fink, Mark Humayun, and Greg Bearman. 2007. Snapshot hyperspectral imaging in ophthalmology. Journal of biomedical optics 12, 1 (2007), 014036--014036.
20. Min H Kim. 2013. 3D Graphics Techniques for Capturing and Inspecting Hyperspectral Appearance. In Ubiquitous Virtual Reality (ISUVR), 2013 Int. Symp. on. IEEE, 15--18.
21. Min H Kim, Todd Alan Harvey, David S Kittle, Holly Rushmeier, Julie Dorsey, Richard O Prum, and David J Brady. 2012a. 3D imaging spectroscopy for measuring hyperspectral patterns on solid objects. ACM Transactions on Graphics 31, 4 (2012), 38.
22. Min H. Kim and Holly Rushmeier. 2011. Radiometric Characterization of Spectral Imaging for Textual Pigment Identification. In Proc. International Symposium on Virtual Reality, Archaeology and Cultural Heritage (VAST 2011). Eurographics, Tuscany, Italy, 57--64.
23. Min H Kim, Holly Rushmeier, John ffrench, and Irma Passeri. 2012b. Developing Open-Source Software for Art Conservators. In VAST12: The 13th International Symposium on Virtual Reality, Archaeology and Intelligent Cultural Heritage. Eurographics Association, Brighton, England, 97--104.
24. Min H Kim, Holly Rushmeier, John ffrench, Irma Passeri, and David Tidmarsh. 2014. Hyper3D: 3D Graphics Software for Examining Cultural Artifacts. ACM Journal on Computing and Cultural Heritage 7, 3 (2014), 1:1--19.
25. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. In The International Conference on Learning Representations (ICLR).
26. David Kittle, Kerkil Choi, Ashwin Wagadarikar, and David J Brady. 2010. Multiframe image estimation for coded aperture snapshot spectral imagers. Applied Optics 49, 36 (2010), 6824--6833.
27. Fred A Kruse, AB Lefkoff, JW Boardman, KB Heidebrecht, AT Shapiro, PJ Barloon, and AFH Goetz. 1993. The spectral image processing system (SIPS)---interactive visualization and analysis of imaging spectrometer data. Remote sensing of environment 44, 2--3 (1993), 145--163.
28. Haebom Lee and Min H. Kim. 2014. Building a Two-Way Hyperspectral Imaging System with Liquid Crystal Tunable Filters. In Proc. Int. Conf. Image and Signal Processing (ICISP 2014) (Lecture Notes in Computer Science (LNCS)), Vol. 8509. Springer, Normandy, France, 26--34.
29. Chengbo Li, Wotao Yin, and Yin Zhang. 2009. User's guide for TVAL3: TV minimization by augmented lagrangian and alternating direction algorithms. CAAM report 20, 46--47 (2009), 4.
30. Xing Lin, Yebin Liu, Jiamin Wu, and Qionghai Dai. 2014. Spatial-spectral encoded compressive hyperspectral imaging. ACM Transactions on Graphics 33, 6 (2014), 233.
31. Giljoo Nam and Min H. Kim. 2014. Multispectral Photometric Stereo for Acquiring High-Fidelity Surface Normals. IEEE Computer Graphics and Applications 34, 6 (2014), 57--68.
32. Takayuki Okamoto, Akinori Takahashi, and Ichirou Yamaguchi. 1993. Simultaneous Acquisition of Spectral and Spatial Intensity Distribution. Appl. Spectrosc. 47, 8 (Aug 1993), 1198--1202.
33. Donald C. O'Shea, Thomas J. Suleski, Alan D. Kathman, and Dennis W. Prather. 2003. Diffractive Optics: Design, Fabrication, and Test. SPIE Press.
34. Yifan Peng, Xiong Dun, Qilin Sun, Felix Heide, and Wolfgang Heidrich. 2018. Focal sweep imaging with multi-focal diffractive optics. In IEEE Proc. Int. Conf. Computational Photography (ICCP). IEEE, 1--8.
35. Yifan Peng, Qiang Fu, Hadi Amata, Shuochen Su, Felix Heide, and Wolfgang Heidrich. 2015. Computational imaging using lightweight diffractive-refractive optics. Optics Express 23, 24 (2015), 31393--31407.
36. Yifan Peng, Qiang Fu, Felix Heide, and Wolfgang Heidrich. 2016. The diffractive achromat full spectrum computational imaging with diffractive optics. ACM Transactions on Graphics (Proc. SIGGRAPH 2016) (2016), 1--11.
37. Wallace M Porter and Harry T Enmark. 1987. A system overview of the airborne visible/infrared imaging spectrometer (AVIRIS). In 31st Annual Technical Symposium. International Society for Optics and Photonics, 22--31.
38. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234--241.
39. K. Simonyan and A. Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proc. Int. Conf. Learning Representation (ICLR).
40. Vincent Sitzmann, Steven Diamond, Yifan Peng, Xiong Dun, Stephen Boyd, Wolfgang Heidrich, Felix Heide, and Gordon Wetzstein. 2018. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Transactions on Graphics (Proc. SIGGRAPH 2018) 37, 4 (2018), 114.
41. Gary J Swanson. 1991. Binary optics technology: theoretical limits on the diffraction efficiency of multilevel diffractive optical elements. Technical Report. MASSACHUSETTS INST OF TECH LEXINGTON LINCOLN LAB.
42. Kazuyuki Tajima, Takeshi Shimano, Yusuke Nakamura, Mayu Sao, and Taku Hoshizawa. 2017. Lensless light-field imaging with multi-phased Fresnel zone aperture. In Proc. IEEE Int. Conf. Computational Photography (ICCP). IEEE, 1--7.
43. Ashwin Wagadarikar, Renu John, Rebecca Willett, and David Brady. 2008. Single disperser design for coded aperture snapshot spectral imaging. Applied optics 47, 10 (2008), B44--B51.
44. Lizhi Wang, Chen Sun, Ying Fu, Min H. Kim, and Huang Hua. 2019. Hyperspectral Image Reconstruction Using a Deep Spatial-Spectral Prior. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2019). TBD.
45. Peng Wang and Rajesh Menon. 2015. Ultra-high-sensitivity color imaging via a transparent diffractive-filter array and computational optics. Optica 2, 11 (Nov 2015), 933--939.
46. Peng Wang and Rajesh Menon. 2018. Computational multispectral video imaging. J. Opt. Soc. Am. A 35, 1 (Jan 2018), 189--199.
47. Jian Zhang and Bernard Ghanem. 2018. ISTA-Net: Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR 2018). 1828--1837.
48. Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. 2017. Learning deep CNN denoiser prior for image restoration. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Vol. 2.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

        • Hyperspectral imaging