skip to main content
research-article

Hyperspectral Reconstruction with Redundant Camera Spectral Sensitivity Functions

Published:22 May 2020Publication History
Skip Abstract Section

Abstract

High-resolution hyperspectral (HS) reconstruction has recently achieved significantly progress, among which the method based on the fusion of the RGB and HS images of the same scene can greatly improve the reconstruction performance compared with those based on the individually spectral or spatial enhancement. It is well known that the HS image is obtained only via the costly hypersoectral sensor, whereas the RGB images can be provided by low-price RGB cameras and the spectral sensitivity (SS) functions of RGB cameras are usually different. Thus, this study proposes a HS reconstruction, which fuses merely two RGB images with redundant spectral responses. In this work, we design a new RGB camera via shifting the SS of an existed RGB camera, which can provide similar strength of spectral response with different spectral centers of SS, and fuse the new achieved color image with an existed RGB image by a deep ResNet. Experiments validate that fusion of two existed RGB images can provide impressive HS reconstruction performance and further improvement can be achieved by integrating the color image of the simulated SS with the RGB image.

References

  1. J. Aeschbacher, J. Wu, and R. Timofte. 2017. In defense of shallow learned spectral reconstruction from RGB images. In Proceedings of the IEEE International Conference on Computer Vision Workshop (ICCVW’17).Google ScholarGoogle Scholar
  2. N. Akhtar, F. Shafait, and A. Mian. 2015. Bayesian sparse representation for hyperspectral image super resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). 3631--3640.Google ScholarGoogle Scholar
  3. Aitor Alvarez-Gila, Joost van de Weijer, and Estibaliz Garrote. 2017. Adversarial networks for spatial context-aware spectral image reconstruction from RGB. In Proceedings of the IEEE International Conference on Computer Vision Workshop (ICCVW’17).Google ScholarGoogle Scholar
  4. B. Arad and O. Ben-Shahar. 2016. Sparse recovery of hyperspectral signal from natural RGB images. In Proceedings of the European Conference on Computer Vision.19--34.Google ScholarGoogle Scholar
  5. J. M. Bioucas-Dias, A. Plaza, G. Camps-Valls, P. Scheunders, N. M. Nasrabadi, and J. Chanussot. 2013. Hyperspectral remote sensing data analysis and future challenges. IEEE Geoscience and Remote Sensing Magazine 1, 2 (2013), 6--36.Google ScholarGoogle ScholarCross RefCross Ref
  6. A. Chakrabarti and T. Zickler. 2011. Statistics of real-world hyperspectral images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11). 193--200.Google ScholarGoogle Scholar
  7. H. Chang, D. Yeung, and Y. Xiong. 2004. Super-resolution through neighbor embedding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’04).Google ScholarGoogle Scholar
  8. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L Yuille. 2016. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. arXiv:1606.00915.Google ScholarGoogle Scholar
  9. R. W. Dian, S. Li, A. Guo, and L. Fang. 2018. Deep hyperspectral image sharpening. IEEE Transactions on Neural Networks and Learning Systems 29, 11 (2018), 5345--5355.Google ScholarGoogle ScholarCross RefCross Ref
  10. C. Dong, C. C. Loy, K. M. He, and X. O. Tang. 2015. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 38, 2 (2015), 295--307.Google ScholarGoogle Scholar
  11. C. Dong, C. C. Loy, and X. O. Tang. 2016. Accelerating the super-resolution convolutional neural network. In Proceedings of the European Conference on Computer Vision (ECCV’16).Google ScholarGoogle Scholar
  12. W. S. Dong, F. Z. Fu, G. M. Shi, X. Cao, J. J. Wu, G. Y. Li, and X. Li. 2016. Hyperspectral image super-resolution via non-negative strutured sparse representation. IEEE Transactions on Image Processing 25, 3 (2016), 2337--2352.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. M. Fauvel, Y. Tarabalka, J. Benediktsson, J. Chanusssot, and J. Tilton. 2013. Advances in spectral-spatial classification of hyperspectral images. Proceedings of the IEEE 101, 3 (2013), 652--675.Google ScholarGoogle ScholarCross RefCross Ref
  14. S. Galliani, C. Lanaras, D. Marmanis, E. Baltsavias, and K. Schindler. 2017. Learned spectral super-resolution. arXiv:1703.09470.Google ScholarGoogle Scholar
  15. X. L. Han, J. Yu, J. Q. Luo, and W. D. Sun. 2019. Hyperspectral and multispectral image fusion using cluster-based multi-branch BP neural networks. Remote Sensing 11, 10 (2019), 1--13.Google ScholarGoogle ScholarCross RefCross Ref
  16. X.-H. Han and Y.-W. Chen. 2019. Deep residual network of spectral and spatial fusion for hyperspectral image super-resolution. In Proceedings of the IEEE International Conference on Multimedia Big Data (BigMM’19). 266--270.Google ScholarGoogle ScholarCross RefCross Ref
  17. X.-H. Han, B. X. Shi, and Y. Q. Zheng. 2018. Self-similarity constrained sparse representation for hyperspectral image super-resolution. IEEE Transactions on Image Processing 27, 11 (2018), 5625--5637.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. X.-H. Han, B. X. Shi, and Y. Q. Zheng. 2018. SSF-CNN: Spatial and spectral fusion with CNN for hyperspectral image super-resolution. In Proceedings of the 25th IEEE International Conference on Image Processing (ICIP’18). 2506--2510.Google ScholarGoogle Scholar
  19. R. C. Hardie, M. T. Eismann, and G. L. Wilson. 2004. Map estimation for hyperspectral image resolution enhancement using an auxiliary sensor. IEEE Transactions on Image Processing 13, 9 (2004), 1174--1184.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. K. M. He, X. Y. Zhang, S. Q. Ren, and J. Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16).Google ScholarGoogle Scholar
  21. A. Hore and D. Ziou. 2010. Image quality metrics: PSNR vs. SSIM. In Proceedings of the 20th International Conference on Pattern Recognition (ICPR’10). 2366--2369.Google ScholarGoogle Scholar
  22. J. Jiang, D. Y. Liu, J. W. Gu, and S. Susstrunk. 2013. What is the space of spectral sensitivity functions for digital color cameras? In Proceedings of the IEEE Workshop on Applications of Computer Vision (WACV’13). 4321--4328.Google ScholarGoogle Scholar
  23. J. Kim, J. K. Lee, and K. M. Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16).Google ScholarGoogle Scholar
  24. C. Kwan, J. H. Choi, S. H. Chan, J. Zhou, and B. Budavari. 2018. A super-resolution and fusion approach to enhancing hyperspectral images. Remote Sensing 10, 9 (2018), 1--28.Google ScholarGoogle ScholarCross RefCross Ref
  25. C. Lanaras, E. Baltsavias, and K. Schindler. 2015. Hyperspectral super-resolution by coupled spectral unmixing. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’15). 3586--3595.Google ScholarGoogle Scholar
  26. Y. S. Li, J. Hua, X. Zhao, W. Y. Xie, and J. J. Li. 2017. Hyperspectral image super-resolution using deep convolutional neural network. Neurocomputing 266 (2017), 29--41.Google ScholarGoogle ScholarCross RefCross Ref
  27. B. Lim, A. Son, H. Kim, S. Nah, and K. M. Lee. 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17).Google ScholarGoogle Scholar
  28. L. Loncan, L. B. Almeida, J. M. Bioucas-Dias, X. Briottet, J. Chanussot, N. Dobigeon, S. Fabre, et al. 2015. Hyperspectral pansharpening: A review. IEEE Geoscience and Remote Sensing Magazine 3, 3 (2015), 27--46.Google ScholarGoogle ScholarCross RefCross Ref
  29. S. H. Mei, X. Yuan, J. Y. Ji, and Y. F. Zhang. 2017. Hyperspectral image spatial super-resolution via 3D full convolutional neural network. Neurocomputing 9, 11 (2017), 1139.Google ScholarGoogle Scholar
  30. Y. Tarabalka, J. Chanusssot, and J. Benediktsson. 2010. Segmentation and classification of hyperspectral images using minimum spanning forest grown from automatically selected markers. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 40, 5 (2010), 1267--1279.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Q. Wei, J. Bioucas-Dias, N. Dobigeon, and J. Y. Toureret. 2015. Hyperspectral and multispectral image fusion based on a sparse representation. IEEE Transactions on Geoscience and Remote Sensing 53, 7 (2015), 3658--3668.Google ScholarGoogle ScholarCross RefCross Ref
  32. J. Yang, J. Wright, T. Huang, and Y. Ma. 2008. Image super-resolution as sparse representation of raw image patches. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’08), Vol. 2. 1--8.Google ScholarGoogle Scholar
  33. F. Yasuma, D. Iso, and S. Nayar. 2010. Generalized assorted pixel camera: Post-capture control of resolution, dynamic range and spectrum. IEEE Transactions on Image Processing 19, 9 (2010), 2241--2253.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. C. Yi, Y. Q. Zhao, J. C. Chan, F. Yasuma, T. Mitsunaga, D. Iso, and S. K. Nayar. 2019. Spectral super-resolution for multispectral image based on spectral improvement strategy and spatial preservation strategy. IEEE Transactions on Geoscience and Remote Sensing 57, 11 (2019), 9010--9024.Google ScholarGoogle ScholarCross RefCross Ref
  35. N. Yokoya, T. Yairi, and A. Iwasaki. 2012. Coupled nonnegative matrix factorization for hyperspectral and multispectral data fusion. IEEE Transactions on Geoscience and Remote Sensing 50, 2 (2012), 528--537.Google ScholarGoogle ScholarCross RefCross Ref
  36. J. Zhou, C. Kwan, and B. Budavari. 2016. Hyperspectral image super-resolution: A hybrid color mapping approach. Journal of Applied Remote Sensing 10, 31 (2016), 035024-1--035024-20.Google ScholarGoogle ScholarCross RefCross Ref
  37. Y. Zhou, H. Chang, K. Barner, P. Spellman, and B. Parvin. 2014. Classification of histology sections via multispectral convolutional sparse coding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’14). 3081--3088.Google ScholarGoogle Scholar

Index Terms

  1. Hyperspectral Reconstruction with Redundant Camera Spectral Sensitivity Functions

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format .

        View HTML Format
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!