Abstract
Hyperspectral imaging is a promising imaging modality that simultaneously captures several images for the same scene on narrow spectral bands, and it has made considerable progress in different fields, such as agriculture, astronomy, and surveillance. However, the existing hyperspectral (HS) cameras sacrifice the spatial resolution for providing the detail spectral distribution of the imaged scene, which leads to low-resolution (LR) HS images compared with the common red-green-blue (RGB) images. Generating a high-resolution HS (HR-HS) image via fusing an observed LR-HS image with the corresponding HR-RGB image has been actively studied. Existing methods for this fusing task generally investigate hand-crafted priors to model the inherent structure of the latent HR-HS image, and they employ optimization approaches for solving it. However, proper priors for different scenes can possibly be diverse, and to figure it out for a specific scene is difficult. This study investigates a deep convolutional neural network (DCNN)-based method for automatic prior learning, and it proposes a novel fusion DCNN model with multi-scale spatial and spectral learning for effectively merging an HR-RGB and LR-HS images. Specifically, we construct an U-shape network architecture for gradually reducing the feature sizes of the HR-RGB image (Encoder-side) and increasing the feature sizes of the LR-HS image (Decoder-side), and we fuse the HR spatial structure and the detail spectral attribute in multiple scales for tackling the large resolution difference in spatial domain of the observed HR-RGB and LR-HS images. Then, we employ multi-level cost functions for the proposed multi-scale learning network to alleviate the gradient vanish problem in long-propagation procedure. In addition, for further improving the reconstruction performance of the HR-HS image, we refine the predicted HR-HS image using an alternating back-projection method for minimizing the reconstruction errors of the observed LR-HS and HR-RGB images. Experiments on three benchmark HS image datasets demonstrate the superiority of the proposed method in both quantitative values and visual qualities.
- [1] . 2009. A comparison between global and context-adaptive pansharpening of multispectral images. IEEE Geosci. Remote Sens. Lett. 6, 2 (2009), 302–306.Google Scholar
Cross Ref
- [2] . 2014. SUnGP: A greedy sparse approximation algorithm for hyperspectral unmixing. In Proceedings of the ICPR. 3726–3731. Google Scholar
Digital Library
- [3] . 2014. Sparse spatio-spectral representation for hyperspectral image super-resolution. In Proceedings of the ECCV. 63–78.Google Scholar
- [4] . 2015. Bayesian sparse representation for hyperspectral image super resolution. In Proceedings of the CVPR. 3631–3640.Google Scholar
- [5] . 2017. Adversarial networks for spatial context-aware spectral image reconstruction from RGB. In Proceedings of the ICCVW.Google Scholar
- [6] . 2016. Sparse recovery of hyperspectral signal from natural rgb images. In Proceedings of the ECCV. 19–34.Google Scholar
- [7] . 2013. Hyperspectral remote sensing data analysis and future challenges. IEEE Geosci. Remote Sens. Mag. 1, 2 (2013), 6–36.Google Scholar
Cross Ref
- [8] . 2009. Merfing hyperspectral and panchromatic image data: Qualitative and quantitative analysis. Int. J. Remote Sens. 30, 7 (2009), 1779–1804. Google Scholar
Digital Library
- [9] . 2011. Statistics of real-world hyperspectral images. In Proceedings of the CVPR. 193–200. Google Scholar
Digital Library
- [10] . 1991. Comparison of three different methods to merge multiresolution and multispectral data: Landsat TM and SPOT panchromatic. Photogramm. Eng. Rem. S. 30, 7 (1991), 1779–1804.Google Scholar
- [11] . 2019. Hyperspectral image super-resolution via subspace based low tensor multi-rank regularization. IEEE Trans. Image Process. 28, 10 (2019), 5135–5146.Google Scholar
Digital Library
- [12] . 2018. Deep hyperspectral image sharpening. IEEE Trans. Neural Netw. Learn. Syst. (2018), 1–11.Google Scholar
- [13] . 2015. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38, 2 (2015), 295–307. Google Scholar
Digital Library
- [14] . 2016. Accelerating the super-resolution convolutional neural network. In Proceedings of the ECCV.Google Scholar
- [15] . 2016. Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Trans. Image Process. 25, 3 (2016), 2337–2352. Google Scholar
Digital Library
- [16] . 2013. Advances in spectral-spatial classification of hyperspectral images. Proc. IEEE 101, 3 (2013), 652–675.Google Scholar
Cross Ref
- [17] . 2017. Learned spectral super-resolution. Retrieved from https://arXiv:1703.09470.Google Scholar
- [18] . 2013. Jointly sparse fusion of hyperspectral and multispectral imagery. In Proceedings of the IGARSS.Google Scholar
- [19] . 2019. Deep residual network of spectral and spatial fusion for hyperspectral image super-resolutionn. In Proceedings of the BigMM. 266–270.Google Scholar
- [20] . 2018. Residual HSRCNN: Residual hyper-spectral reconstruction CNN from an RGB image. In Proceedings of the ICPR. 2664–2669.Google Scholar
- [21] . 2018. Self-similarity constrained sparse representation for hyperspectral image super-resolution. IEEE Trans. Image Process. 27, 11 (2018), 5625–5637. Google Scholar
Digital Library
- [22] . 2018. SSF-CNN: Spatial and spectral fusion with CNN for hyperspectral image super-resolution. In Proceedings of the ICIP. 2506–2510.Google Scholar
- [23] . 2019. Multi-level and multi-scale spatial and spectral fusion CNN for hyperspectral image super-resolution. In Proceedings of the ICCV Workshop. 4330–4339.Google Scholar
- [24] . 2017. First-order derivative-based super-resolution. Signal, Image Video Process. 11, 1 (2017), 1–9.Google Scholar
Cross Ref
- [25] . 1982. Application of the IHS color transform to the processing of multisensor data and image enhancement. In Proceedings of the ISRSE.Google Scholar
- [26] . 2014. Spatial and spectral image fusion using sparse matrix factorization. IEEE Trans Geosci. Remote Sens. 52, 3 (2014), 1693–1704.Google Scholar
Cross Ref
- [27] . 2014. Caffe: Convolutional architecture for fast feature embedding. Retrieved from https://arXiv:1408.5093. Google Scholar
Digital Library
- [28] . 2011. High-resolution hyperspectral imaging via matrix factorization. In Proceedings of the CVPR. 2329–2336. Google Scholar
Digital Library
- [29] . 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the CVPR.Google Scholar
- [30] . 1993. The spectral image processing system (sips)—Interactive visualization and analysis of imaging spectrometer data. Remote Sens. Environ. 44, 2–3 (1993), 145–163.Google Scholar
Cross Ref
- [31] . 2017. Resolution enhancement for hyperspectral images: A super-resolution and fusion approach. In Proceedings of the ICASSP. 6180–6184.Google Scholar
- [32] . 2018. A super-resolution and fusion approach to enhancing hyperspectral images. Remote Sens. 10 (2018), 1416.Google Scholar
Cross Ref
- [33] . 2015. Hyperspectral super-resolution by coupled spectral unmixing. In Proceedings of the ICCV. 3586–3595. Google Scholar
Digital Library
- [34] . 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the CVPR.Google Scholar
- [35] . 2001. Algorithms for non-negative matrix factorization. In Proceedings of the NIPS. 556–562. Google Scholar
Digital Library
- [36] . 2014. Deep learning for image denoising. Int. J. Signal Process. Image Process. Pattern Recogn. 7, 3 (2014), 171–180.Google Scholar
- [37] . 2017. Hyperspectral image super-resolution using deep convolutional neural network. Neurocomputing 266 (2017), 29–41.Google Scholar
Cross Ref
- [38] . 2017. Hyperspectral image spatial super-resolution via 3D full convolutional neural network. Neurocomputing 9, 11 (2017).Google Scholar
- [39] . 2006. Spatial resolution improvement by merging MERIS-ETM images for coastal water monitoring. IEEE Geosci. Remote Sens. Lett. 3, 2 (2006), 227–231.Google Scholar
Cross Ref
- [40] . 2010. Tracking via object reflectance using a hyperspectral video camera. In Proceedings of the CVPRW. 44–51.Google Scholar
- [41] . 2014. Training-based spectral reconstruction from a single RGB image. In Proceedings of the ECCV. 186–201.Google Scholar
- [42] . 2015. DeepID-Net: Deformable deep convolu- tional neural networks for object detection. In Proceedings of the CVPR. 2403–2412.Google Scholar
- [43] . 2018. Unsupervised sparse dirichlet-net for hyperspectral image super-resolution. In Proceedings of the CVPR. 2862–2869.Google Scholar
- [44] . 2014. Very deep convolutional networks for large-scale image recognition. Retrieved from https://arxiv.org/abs/1409.1556.Google Scholar
- [45] . 2014. Deep learning face representation by joint identification-verification. Adv. Neural Info. Process. Syst. 27 (2014), 1988–1996. Google Scholar
Digital Library
- [46] . 2015. Scalable, high-quality object detection. Retrieved from https://arXiv:1412.1441v3. 1–10.Google Scholar
- [47] . 2010. Segmentation and classification of hyperspectral images using minimum spanning forest grown from automatically selected markers. IEEE Trans. Syst., Man, Cybern., Syst. 40, 5 (2010), 1267–1279. Google Scholar
Digital Library
- [48] . 2013. Hyperspectral face recognition using 3D-DCT and partial least squares. In Proceedings of the BMVC. 57.1–57.10.Google Scholar
- [49] . 2004. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13, 4 (2004), 600–612. Google Scholar
Digital Library
- [50] . 2013. A non-negative sparse promoting algorithm for high-resolution hyperspectral imaging. In Proceedings of the ICASSP. 1409–1413.Google Scholar
- [51] . 2015. Hyperspectral and multispectral image fusion based on a sparse representation. IEEE Trans Geosci. Remote Sens. 53, 7 (2015), 3658–3668.Google Scholar
Cross Ref
- [52] . 2019. Nonlocal patch tensor sparse representation for hyperspectral image super-resolution. IEEE Trans. Image Process. 28, 6 (2019), 3034–3047.Google Scholar
Cross Ref
- [53] . 2012. Coupled nonnegative matrix factorization for hyperspectral and multispectral data fusion. IEEE Trans Geosci. Remote Sens. 50, 2 (2012), 528–537.Google Scholar
Cross Ref
- [54] . 2012. A comparative study of palmprint recognition algorithm. ACM Comput. Surv. 44, 1 (2012), 2:1–2:37. Google Scholar
Digital Library
- [55] . 2018. Exploiting clustering manifold structure for hyperspectral imagery super-resolutionn. IEEE Trans. Image Process. 27, 12 (2018), 5969–5982. Google Scholar
Digital Library
- [56] . 2016. Hyperspectral image super-resolution: A hybrid color mapping approach. J. Appl. Remote Sens. 10, 3 (2016), 035024.Google Scholar
Cross Ref
- [57] . 2014. Classification of histology sections via multispectral convolutional sparse coding. In Proceedings of the CVPR. 3081–3088. Google Scholar
Digital Library
- [58] . 2008. Unmixing-based landsat TM ane MERIS FR data fusion. IEEE Geosci. Remote Sens. Lett. 5, 3 (2008), 453–457.Google Scholar
Cross Ref
Index Terms
Hyperspectral Image Reconstruction Using Multi-scale Fusion Learning
Recommendations
High-quality hyperspectral reconstruction using a spectral prior
We present a novel hyperspectral image reconstruction algorithm, which overcomes the long-standing tradeoff between spectral accuracy and spatial resolution in existing compressive imaging approaches. Our method consists of two steps: First, we learn ...
Multi-scale spatial-spectral fusion based on multi-input fusion calculation and coordinate attention for hyperspectral image classification
Highlights- Multi-scale spectral features and spatial features are acquired and fused.
- The ...
AbstractRecently, the deep learning method that integrates image features has gradually become a hot development trend in hyperspectral image classification. However, these studies did not fully consider the fusion of image features, and did ...
Unregistered Hyperspectral and Multispectral Image Fusion with Synchronous Nonnegative Matrix Factorization
Pattern Recognition and Computer VisionAbstractRecently, many methods have been proposed to generate a high spatial resolution (HR) hyperspectral image (HSI) by fusing HSI and multispectral image (MSI). Most methods need a precondition that HSI and MSI are well registered. However, in practice,...






Comments