skip to main content
research-article

Hyperspectral Image Reconstruction Using Multi-scale Fusion Learning

Authors Info & Claims
Published:27 January 2022Publication History
Skip Abstract Section

Abstract

Hyperspectral imaging is a promising imaging modality that simultaneously captures several images for the same scene on narrow spectral bands, and it has made considerable progress in different fields, such as agriculture, astronomy, and surveillance. However, the existing hyperspectral (HS) cameras sacrifice the spatial resolution for providing the detail spectral distribution of the imaged scene, which leads to low-resolution (LR) HS images compared with the common red-green-blue (RGB) images. Generating a high-resolution HS (HR-HS) image via fusing an observed LR-HS image with the corresponding HR-RGB image has been actively studied. Existing methods for this fusing task generally investigate hand-crafted priors to model the inherent structure of the latent HR-HS image, and they employ optimization approaches for solving it. However, proper priors for different scenes can possibly be diverse, and to figure it out for a specific scene is difficult. This study investigates a deep convolutional neural network (DCNN)-based method for automatic prior learning, and it proposes a novel fusion DCNN model with multi-scale spatial and spectral learning for effectively merging an HR-RGB and LR-HS images. Specifically, we construct an U-shape network architecture for gradually reducing the feature sizes of the HR-RGB image (Encoder-side) and increasing the feature sizes of the LR-HS image (Decoder-side), and we fuse the HR spatial structure and the detail spectral attribute in multiple scales for tackling the large resolution difference in spatial domain of the observed HR-RGB and LR-HS images. Then, we employ multi-level cost functions for the proposed multi-scale learning network to alleviate the gradient vanish problem in long-propagation procedure. In addition, for further improving the reconstruction performance of the HR-HS image, we refine the predicted HR-HS image using an alternating back-projection method for minimizing the reconstruction errors of the observed LR-HS and HR-RGB images. Experiments on three benchmark HS image datasets demonstrate the superiority of the proposed method in both quantitative values and visual qualities.

REFERENCES

  1. [1] Aiazzi B., Baronti S., Lotti F., and Selva M.. 2009. A comparison between global and context-adaptive pansharpening of multispectral images. IEEE Geosci. Remote Sens. Lett. 6, 2 (2009), 302306.Google ScholarGoogle ScholarCross RefCross Ref
  2. [2] Akhtar N., Shafait F., and Mian A.. 2014. SUnGP: A greedy sparse approximation algorithm for hyperspectral unmixing. In Proceedings of the ICPR. 37263731. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. [3] Akhtar N., Shafait F., and Mian A.. 2014. Sparse spatio-spectral representation for hyperspectral image super-resolution. In Proceedings of the ECCV. 6378.Google ScholarGoogle Scholar
  4. [4] Akhtar N., Shafait F., and Mian A.. 2015. Bayesian sparse representation for hyperspectral image super resolution. In Proceedings of the CVPR. 36313640.Google ScholarGoogle Scholar
  5. [5] Alvarez-Gila Aitor, van de Weijer Joost, and Garrote Estibaliz. 2017. Adversarial networks for spatial context-aware spectral image reconstruction from RGB. In Proceedings of the ICCVW.Google ScholarGoogle Scholar
  6. [6] Arad B. and Ben-Shahar O.. 2016. Sparse recovery of hyperspectral signal from natural rgb images. In Proceedings of the ECCV. 1934.Google ScholarGoogle Scholar
  7. [7] Bioucas-Dias J. M., Plaza A., Camps-Valls G., Scheunders P., Nasrabadi N. M., and Chanussot J.. 2013. Hyperspectral remote sensing data analysis and future challenges. IEEE Geosci. Remote Sens. Mag. 1, 2 (2013), 636.Google ScholarGoogle ScholarCross RefCross Ref
  8. [8] Cetin M. and Musaoglu N.. 2009. Merfing hyperspectral and panchromatic image data: Qualitative and quantitative analysis. Int. J. Remote Sens. 30, 7 (2009), 17791804. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Chakrabarti A. and Zickler T.. 2011. Statistics of real-world hyperspectral images. In Proceedings of the CVPR. 193200. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. [10] Chavez P. S., Sides S. C., and Anderson J. A.. 1991. Comparison of three different methods to merge multiresolution and multispectral data: Landsat TM and SPOT panchromatic. Photogramm. Eng. Rem. S. 30, 7 (1991), 17791804.Google ScholarGoogle Scholar
  11. [11] Dian R. and Li S.. 2019. Hyperspectral image super-resolution via subspace based low tensor multi-rank regularization. IEEE Trans. Image Process. 28, 10 (2019), 51355146.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. [12] Dian R. W., Li S. T., Guo A. J., and Fang L. Y.. 2018. Deep hyperspectral image sharpening. IEEE Trans. Neural Netw. Learn. Syst. (2018), 111.Google ScholarGoogle Scholar
  13. [13] Dong C., Loy C. C., He K. M., and Tang X. O.. 2015. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38, 2 (2015), 295307. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. [14] Dong C., Loy C. C., and Tang X. O.. 2016. Accelerating the super-resolution convolutional neural network. In Proceedings of the ECCV.Google ScholarGoogle Scholar
  15. [15] Dong W. S., Fu F. Z., Shi G. M., Cao X., Wu J. J., Li G. Y., and Li X.. 2016. Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Trans. Image Process. 25, 3 (2016), 23372352. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. [16] Fauvel M., Tarabalka Y., Benediktsson J., Chanusssot J., and Tilton J.. 2013. Advances in spectral-spatial classification of hyperspectral images. Proc. IEEE 101, 3 (2013), 652675.Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] Galliani S., Lanaras C., Marmanis D., Baltsavias E., and Schindler K.. 2017. Learned spectral super-resolution. Retrieved from https://arXiv:1703.09470.Google ScholarGoogle Scholar
  18. [18] Grohnfeldt C., Zhu X. X., and Bamler R.. 2013. Jointly sparse fusion of hyperspectral and multispectral imagery. In Proceedings of the IGARSS.Google ScholarGoogle Scholar
  19. [19] Han X.-H. and Chen Y.-W.. 2019. Deep residual network of spectral and spatial fusion for hyperspectral image super-resolutionn. In Proceedings of the BigMM. 266270.Google ScholarGoogle Scholar
  20. [20] Han X.-H., Shi B. X., and Zheng Y. Q.. 2018. Residual HSRCNN: Residual hyper-spectral reconstruction CNN from an RGB image. In Proceedings of the ICPR. 26642669.Google ScholarGoogle Scholar
  21. [21] Han X.-H., Shi B. X., and Zheng Y. Q.. 2018. Self-similarity constrained sparse representation for hyperspectral image super-resolution. IEEE Trans. Image Process. 27, 11 (2018), 56255637. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. [22] Han X.-H., Shi B. X., and Zheng Y. Q.. 2018. SSF-CNN: Spatial and spectral fusion with CNN for hyperspectral image super-resolution. In Proceedings of the ICIP. 25062510.Google ScholarGoogle Scholar
  23. [23] Han X.-H., Zheng Y. Q., and Chen Y.-W.. 2019. Multi-level and multi-scale spatial and spectral fusion CNN for hyperspectral image super-resolution. In Proceedings of the ICCV Workshop. 43304339.Google ScholarGoogle Scholar
  24. [24] Haris M., Widyanto M. R., and Nobuhara H.. 2017. First-order derivative-based super-resolution. Signal, Image Video Process. 11, 1 (2017), 19.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Haydn R., Dalke G. W., Henkel J., and Bare J. E.. 1982. Application of the IHS color transform to the processing of multisensor data and image enhancement. In Proceedings of the ISRSE.Google ScholarGoogle Scholar
  26. [26] Huang B., song H., Cui H., Peng J., and Xu Z.. 2014. Spatial and spectral image fusion using sparse matrix factorization. IEEE Trans Geosci. Remote Sens. 52, 3 (2014), 16931704.Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Jia Y. Q., Shelhamer E., Donahue J., Karayev S., Long J., Girshick R., Guadarrama S., and Darrell T.. 2014. Caffe: Convolutional architecture for fast feature embedding. Retrieved from https://arXiv:1408.5093. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. [28] Kawakami R., Wright J., Tai Y.-W., Matsushita Y., Ben-Ezra M., and Ikeuchi K.. 2011. High-resolution hyperspectral imaging via matrix factorization. In Proceedings of the CVPR. 23292336. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. [29] Kim J., Lee J. K., and Lee K. M.. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the CVPR.Google ScholarGoogle Scholar
  30. [30] Kruse F. A., Lefkoff A. B., Boardman J. W., Heidebrecht K. B., Shapiro A. T., Barloon P. J., and Goetz A. F. H.. 1993. The spectral image processing system (sips)—Interactive visualization and analysis of imaging spectrometer data. Remote Sens. Environ. 44, 2–3 (1993), 145163.Google ScholarGoogle ScholarCross RefCross Ref
  31. [31] Kwan C., Cjoi J. H., Chan S. H., Zhou J., and Budavari B.. 2017. Resolution enhancement for hyperspectral images: A super-resolution and fusion approach. In Proceedings of the ICASSP. 61806184.Google ScholarGoogle Scholar
  32. [32] Kwan C., Cjoi J. H., Chan S. H., Zhou J., and Budavari B.. 2018. A super-resolution and fusion approach to enhancing hyperspectral images. Remote Sens. 10 (2018), 1416.Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Lanaras C., Baltsavias E., and Schindler K.. 2015. Hyperspectral super-resolution by coupled spectral unmixing. In Proceedings of the ICCV. 35863595. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. [34] Ledig C., Theis L., Huszar F., Caballero J., Cunningham A., Acosta A., Aitken A., Tejani A., Totz J., Wang Z., and Shi W. Z.. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the CVPR.Google ScholarGoogle Scholar
  35. [35] Lee D. D. and Seung S. H.. 2001. Algorithms for non-negative matrix factorization. In Proceedings of the NIPS. 556562. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. [36] Li H. M.. 2014. Deep learning for image denoising. Int. J. Signal Process. Image Process. Pattern Recogn. 7, 3 (2014), 171180.Google ScholarGoogle Scholar
  37. [37] Li Y. S., Hua J., Zhao X., Xie W. Y., and Li J. J.. 2017. Hyperspectral image super-resolution using deep convolutional neural network. Neurocomputing 266 (2017), 2941.Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Mei S. H., Yuan X., Ji J. Y., and Zhang Y. F.. 2017. Hyperspectral image spatial super-resolution via 3D full convolutional neural network. Neurocomputing 9, 11 (2017).Google ScholarGoogle Scholar
  39. [39] Minghelli-Roman A., Polidori L., Mathieu-Blanc S., Loubersac L., and Cauneau F.. 2006. Spatial resolution improvement by merging MERIS-ETM images for coastal water monitoring. IEEE Geosci. Remote Sens. Lett. 3, 2 (2006), 227231.Google ScholarGoogle ScholarCross RefCross Ref
  40. [40] Nguyen H. V., Benerjee A., and Chellappa R.. 2010. Tracking via object reflectance using a hyperspectral video camera. In Proceedings of the CVPRW. 4451.Google ScholarGoogle Scholar
  41. [41] Nguyen R. M., Prasad D. K., and Brown M. S.. 2014. Training-based spectral reconstruction from a single RGB image. In Proceedings of the ECCV. 186201.Google ScholarGoogle Scholar
  42. [42] Ouyang W., Wang X., Zeng X., and Qiu S.. 2015. DeepID-Net: Deformable deep convolu- tional neural networks for object detection. In Proceedings of the CVPR. 24032412.Google ScholarGoogle Scholar
  43. [43] Qu Y., Qi H., and Kwan C.. 2018. Unsupervised sparse dirichlet-net for hyperspectral image super-resolution. In Proceedings of the CVPR. 28622869.Google ScholarGoogle Scholar
  44. [44] Simonyan K. and Zisserman A.. 2014. Very deep convolutional networks for large-scale image recognition. Retrieved from https://arxiv.org/abs/1409.1556.Google ScholarGoogle Scholar
  45. [45] Sun Y., Wang X., and Tang X.. 2014. Deep learning face representation by joint identification-verification. Adv. Neural Info. Process. Syst. 27 (2014), 19881996. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. [46] Szegedy C., Reed S., Erhan D., Anguelov D., and Ioffe S.. 2015. Scalable, high-quality object detection. Retrieved from https://arXiv:1412.1441v3. 110.Google ScholarGoogle Scholar
  47. [47] Tarabalka Y., Chanusssot J., and Benediktsson J.. 2010. Segmentation and classification of hyperspectral images using minimum spanning forest grown from automatically selected markers. IEEE Trans. Syst., Man, Cybern., Syst. 40, 5 (2010), 12671279. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. [48] Uzair M., Mahmood A., and Mian A.. 2013. Hyperspectral face recognition using 3D-DCT and partial least squares. In Proceedings of the BMVC. 57.1–57.10.Google ScholarGoogle Scholar
  49. [49] Wang Z., Bovik A., Sheikh H., and Simoncelli E.. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13, 4 (2004), 600612. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. [50] Wcoff E., Chan T. H., Jia K., Ma W. K., and Ma Y.. 2013. A non-negative sparse promoting algorithm for high-resolution hyperspectral imaging. In Proceedings of the ICASSP. 14091413.Google ScholarGoogle Scholar
  51. [51] Wei Q., Bioucas-Dias J., Dobigeon N., and Toureret J. Y.. 2015. Hyperspectral and multispectral image fusion based on a sparse representation. IEEE Trans Geosci. Remote Sens. 53, 7 (2015), 36583668.Google ScholarGoogle ScholarCross RefCross Ref
  52. [52] Xu Y., Wu Z., Chanussot J., and Wei Z.. 2019. Nonlocal patch tensor sparse representation for hyperspectral image super-resolution. IEEE Trans. Image Process. 28, 6 (2019), 30343047.Google ScholarGoogle ScholarCross RefCross Ref
  53. [53] Yokoya N., Yairi T., and Iwasaki A.. 2012. Coupled nonnegative matrix factorization for hyperspectral and multispectral data fusion. IEEE Trans Geosci. Remote Sens. 50, 2 (2012), 528537.Google ScholarGoogle ScholarCross RefCross Ref
  54. [54] Zhang D., Zuo W., and Yue F.. 2012. A comparative study of palmprint recognition algorithm. ACM Comput. Surv. 44, 1 (2012), 2:1–2:37. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. [55] Zhang L., Wei W., Bai C., Gao Y., and Zhang Y.. 2018. Exploiting clustering manifold structure for hyperspectral imagery super-resolutionn. IEEE Trans. Image Process. 27, 12 (2018), 59695982. Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. [56] Zhou J., Kwan X., and Budavari B.. 2016. Hyperspectral image super-resolution: A hybrid color mapping approach. J. Appl. Remote Sens. 10, 3 (2016), 035024.Google ScholarGoogle ScholarCross RefCross Ref
  57. [57] Zhou Y., Chang H., Barner K., Spellman P., and Parvin B.. 2014. Classification of histology sections via multispectral convolutional sparse coding. In Proceedings of the CVPR. 30813088. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. [58] Zurita-Milla R., Clevers J. G. P. W., and Schaepman M. E.. 2008. Unmixing-based landsat TM ane MERIS FR data fusion. IEEE Geosci. Remote Sens. Lett. 5, 3 (2008), 453457.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Hyperspectral Image Reconstruction Using Multi-scale Fusion Learning

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          • Published in

            cover image ACM Transactions on Multimedia Computing, Communications, and Applications
            ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 18, Issue 1
            January 2022
            517 pages
            ISSN:1551-6857
            EISSN:1551-6865
            DOI:10.1145/3505205
            Issue’s Table of Contents

            Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            • Published: 27 January 2022
            • Accepted: 1 July 2021
            • Revised: 1 April 2021
            • Received: 1 September 2020
            Published in tomm Volume 18, Issue 1

            Permissions

            Request permissions about this article.

            Request Permissions

            Check for updates

            Qualifiers

            • research-article
            • Refereed

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          Full Text

          View this article in Full Text.

          View Full Text

          HTML Format

          View this article in HTML Format .

          View HTML Format
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!