skip to main content
research-article

Pansharpening Scheme Using Bi-dimensional Empirical Mode Decomposition and Neural Network

Authors Info & Claims
Published:04 March 2022Publication History
Skip Abstract Section

Abstract

The pansharpening is a combination of multispectral (MS) and panchromatic (PAN) images that produce a high-spatial-spectral-resolution MS images. In multiresolution analysis–based pansharpening schemes, some spatial and spectral distortions are found. It can be reduced by adding spatial detail images of the PAN image into MS images. In the convolution neural network– (CNN) based method, the lowpass filter image extracted by the CNN model when MS and PAN images are directly applied into the input. The feature values are very high and reduce the conversion efficiency. In the proposed scheme, bi-dimensional empirical mode decomposition is used to extract the spatial detail information of the PAN image to reduce the feature values of the input. This extracted PAN image information is applied to the CNN to produce the non-linear changes in the image pixels and transformed into the perfect spatial detail image. It identifies the spatial and spectral detail quantity for the proposed scheme and it also varies with the different datasets automatically of the same satellite images. Simulation results in the context of qualitative and quantitative analysis demonstrate the effectiveness of proposed scheme applied on datasets collected by different satellites.

REFERENCES

  1. [1] Aiazzi B., Alparone L., Baronti S., Garzelli A., and Selva M.. 2003. An MTF-based spectral distortion minimizing model for pan-sharpening of very high resolution multispectral images of urban areas. In Proceedings of the 2nd GRSS/ISPRS Joint Workshop on Remote Sensing and Data Fusion over Urban Areas. IEEE, 9094.Google ScholarGoogle ScholarCross RefCross Ref
  2. [2] Aiazzi B., Alparone L., Baronti S., Garzelli A., and Selva M.. 2006. MTF-tailored multiscale fusion of high-resolution MS and pan imagery. Photogram. Eng. Remote Sens. 72, 5 (2006), 591596.Google ScholarGoogle ScholarCross RefCross Ref
  3. [3] Aiazzi Bruno, Baronti Stefano, Selva Massimo, and Alparone Luciano. 2013. Bi-cubic interpolation for shift-free pan-sharpening. ISPRS J. Photogram. Remote Sensing 86 (2013), 6576.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Alparone Luciano, Aiazzi Bruno, Baronti Stefano, Garzelli Andrea, Nencini Filippo, and Selva Massimo. 2008. Multispectral and panchromatic data fusion assessment without reference. Photogram. Eng. Remote Sens. 74, 2 (2008), 193200.Google ScholarGoogle ScholarCross RefCross Ref
  5. [5] Alparone Luciano, Wald Lucien, Chanussot Jocelyn, Thomas Claire, Gamba Paolo, and Bruce Lori Mann. 2007. Comparison of pansharpening algorithms: Outcome of the 2006 GRS-S data-fusion contest. IEEE Trans. Geosci. Remote Sens. 45, 10 (2007), 30123021.Google ScholarGoogle ScholarCross RefCross Ref
  6. [6] Azarang Arian, Manoochehri Hafez E, and Kehtarnavaz Nasser. 2019. Convolutional autoencoder-based multispectral image fusion. IEEE Access 7 (2019), 3567335683.Google ScholarGoogle ScholarCross RefCross Ref
  7. [7] Chavez Pats, Sides Stuart C., Anderson Jeffrey A., et al. 1991. Comparison of three different methods to merge multiresolution and multispectral data- landsat TM and SPOT panchromatic. Photogram. Eng. Remote Sens. 57, 3 (1991), 295303.Google ScholarGoogle Scholar
  8. [8] Czaja Wojciech, Doster Timothy, and Murphy James M.. 2014. Wavelet packet mixing for image fusion and pan-sharpening. In Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XX, Vol. 9088. International Society for Optics and Photonics, 908803.Google ScholarGoogle Scholar
  9. [9] Mura Mauro Dalla, Prasad Saurabh, Pacifici Fabio, Gamba Paulo, Chanussot Jocelyn, and Benediktsson Jón Atli. 2015. Challenges and opportunities of multimodality and data fusion in remote sensing. Proc. IEEE 103, 9 (2015), 15851601.Google ScholarGoogle ScholarCross RefCross Ref
  10. [10] Dong Weihua, Li Xian’en, Lin Xiangguo, Li Zhilin, et al. 2014. A bidimensional empirical mode decomposition method for fusion of multispectral and panchromatic remote sensing images. Remote Sens. 6, 9 (2014), 84468467.Google ScholarGoogle ScholarCross RefCross Ref
  11. [11] Flandrin Patrick, Rilling Gabriel, and Goncalves Paulo. 2004. Empirical mode decomposition as a filter bank. IEEE Sign. Process. Lett. 11, 2 (2004), 112114.Google ScholarGoogle ScholarCross RefCross Ref
  12. [12] Gillespie Alan R., Kahle Anne B., and Walker Richard E.. 1987. Color enhancement of highly correlated images. II. Channel ratio and “chromaticity” transformation techniques. Remote Sens’ Environ’ 22, 3 (1987), 343365.Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] González-Audícana María, Saleta José Luis, Catalán Raquel García, and García Rafael. 2004. Fusion of multispectral and panchromatic images using improved IHS and PCA mergers based on wavelet decomposition. IEEE Trans’ Geosci. Remote Sens. 42, 6 (2004), 12911299.Google ScholarGoogle ScholarCross RefCross Ref
  14. [14] He Lin, Rao Yizhou, Li Jun, Chanussot Jocelyn, Plaza Antonio, Zhu Jiawei, and Li Bo. 2019. Pansharpening via detail injection based convolutional neural networks. IEEE J. Select. Top. Appl. Earth Observ. Remote Sens. (2019).Google ScholarGoogle Scholar
  15. [15] Helmy A. K. and El-Tawel Gh S.. 2015. An integrated scheme to improve pan-sharpening visual quality of satellite images. Egypt. Inf. J. 16, 1 (2015), 121131.Google ScholarGoogle Scholar
  16. [16] Huang Norden E., Shen Zheng, Long Steven R., Wu Manli C., Shih Hsing H., Zheng Quanan, Yen Nai-Chyuan, Tung Chi Chao, and Liu Henry H.. 1998. The empirical mode decomposition and the hilbert spectrum for nonlinear and non-stationary time series analysis. In Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, Vol. 454. The Royal Society, 903995.Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] Joshi Manjunath and Jalobeanu André. 2009. MAP estimation for multiresolution fusion in remotely sensed images using an IGMRF prior model. IEEE Trans. Geosci. Remote Sens. 48, 3 (2009), 12451255.Google ScholarGoogle Scholar
  18. [18] Kang Xudong, Li Shutao, and Benediktsson Jón Atli. 2013. Pansharpening with matting model. IEEE Trans. Geosci. Remote Sens. 52, 8 (2013), 50885099.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Kasem Hossam M., Hung Kwok-Wai, and Jiang Jianmin. 2018. Revised spatial transformer network towards improved image super-resolutions. In Proceedings of the 24th International Conference on Pattern Recognition (ICPR’18). IEEE, 26882692.Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Kim Jiwon, Lee Jung Kwon, and Lee Kyoung Mu. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 16461654.Google ScholarGoogle ScholarCross RefCross Ref
  21. [21] Lee Jonghwa and Lee Chulhee. 2010. Fast and efficient panchromatic sharpening. IEEE Trans. Geosci. Remote Sens. 48, 1 (2010), 155163.Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Li Shutao and Yang Bin. 2010. A new pan-sharpening method using a compressed sensing technique. IEEE Trans. Geosci. Remote Sens. 49, 2 (2010), 738746.Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Li Shutao, Yin Haitao, and Fang Leyuan. 2013. Remote sensing image fusion via sparse representations over learned dictionaries. IEEE Trans. Geosci. Remote Sens. 51, 9 (2013), 47794789.Google ScholarGoogle ScholarCross RefCross Ref
  24. [24] Li Yanping and Jiang Shengming. 2020. Multi-focus image fusion using geometric algebra based discrete fourier transform. IEEE Access 8 (2020), 6001960028.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Masi Giuseppe, Cozzolino Davide, Verdoliva Luisa, and Scarpa Giuseppe. 2016. Pansharpening by convolutional neural networks. Remote Sens. 8, 7 (2016), 594.Google ScholarGoogle ScholarCross RefCross Ref
  26. [26] Nunes Jean Claude, Bouaoune Yasmina, Delechelle Eric, Niang Oumar, and Bunel Ph. 2003. Image analysis by bidimensional empirical mode decomposition. Image Vis. Comput. 21, 12 (2003), l019–1026.Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Otazu Xavier, González-Audícana María, Fors Octavi, and Núñez Jorge. 2005. Introduction of sensor spectral response into image fusion methods. application to wavelet-based methods. IEEE Trans. Geosci. Remote Sens. 43, 10 (2005), 23762385.Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] Restaino Rocco, Vivone Gemine, Mura Mauro Dalla, and Chanussot Jocelyn. 2016. Fusion of multispectral and panchromatic images based on morphological operators. IEEE Trans. Image Process. 25, 6 (2016), 28822895.Google ScholarGoogle Scholar
  29. [29] Saxena Nidhi and Balasubramanian Raman. 2021. A pansharpening scheme using spectral graph wavelet transforms and convolutional neural networks. Int. J. Remote Sens. 42, 8 (2021), 28982919.Google ScholarGoogle ScholarCross RefCross Ref
  30. [30] Saxena Nidhi and Saxena Gaurav. 2021. Pansharpening scheme using spatial detail injection based convolutional neural networks. IET Image Process. (2021).Google ScholarGoogle Scholar
  31. [31] Saxena Nidhi and Sharma K. K.. 2017. A hybrid approach for pansharpening using hilbert vibration decomposition. In Proceedings of the IEEE International Conference on Power, Control, Signals and Instrumentation Engineering (ICPCSI’17). IEEE, 28012805.Google ScholarGoogle ScholarCross RefCross Ref
  32. [32] Saxena Nidhi and Sharma K. K.. 2019. A two-dimensional discrete fractional fourier transform-based pansharpening scheme. Int. J. Remote Sens. 40, 16 (2019), 60986115.Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Saxena Nidhi and Sharma Kamalesh K.. 2017. Pansharpening approach using hilbert vibration decomposition. IET Image Process. 11, 12 (2017), 11521162.Google ScholarGoogle ScholarCross RefCross Ref
  34. [34] Saxena Nidhi and Sharma Kamalesh K.. 2018. Pansharpening scheme using filtering in two-dimensional discrete fractional fourier transform. IET Image Process. 12, 6 (2018), 10131019.Google ScholarGoogle ScholarCross RefCross Ref
  35. [35] Scarpa Giuseppe, Vitale Sergio, and Cozzolino Davide. 2018. Target-adaptive CNN-based pansharpening. IEEE Trans. Geosci. Remote Sens. 56, 9 (2018), 54435457.Google ScholarGoogle ScholarCross RefCross Ref
  36. [36] Shah Vijay P., Younan Nicolas H., and King Roger L.. 2008. An efficient pan-sharpening method via a combined adaptive PCA approach and contourlets. IEEE Trans. Geosci. Remote Sens. 46, 5 (2008), 13231335.Google ScholarGoogle ScholarCross RefCross Ref
  37. [37] Song Huihui, Huang Bo, Liu Qingshan, and Zhang Kaihua. 2015. Improving the spatial resolution of landsat TM/ETM+ through fusion with SPOT5 images via learning-based super-resolution. IEEE Trans. Geosci. Remote Sens. 53, 3 (2015), 11951204.Google ScholarGoogle Scholar
  38. [38] Stroppiana D., Tansey K., Gregoire J., and Pereira J. M. C.. 2003. An algorithm for mapping burnt areas in Australia using SPOT-VEGETATION data. IEEE Trans. Geosci. Remote Sens. 41, 4 (April 2003), 907909. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  39. [39] Tao Xin, Gao Hongyun, Shen Xiaoyong, Wang Jue, and Jia Jiaya. 2018. Scale-recurrent network for deep image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 81748182.Google ScholarGoogle ScholarCross RefCross Ref
  40. [40] Thomas Claire, Ranchin Thierry, Wald Lucien, and Chanussot Jocelyn. 2008. Synthesis of multispectral images to high spatial resolution: A critical review of fusion methods based on remote sensing physics. IEEE Trans. Geosci. Remote Sens. 46, 5 (2008), 13011312.Google ScholarGoogle ScholarCross RefCross Ref
  41. [41] Tian Chunwei, Xu Yong, Fei Lunke, and Yan Ke. 2018. Deep learning for image denoising: A survey. In International Conference on Genetic and Evolutionary Computing. Springer, 563–572.Google ScholarGoogle Scholar
  42. [42] Tu Te-Ming, Huang Ping Sheng, Hung Chung-Ling, and Chang Chien-Ping. 2004. A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery. IEEE Geosci. Remote Sens. Lett. 1, 4 (2004), 309312.Google ScholarGoogle ScholarCross RefCross Ref
  43. [43] Vitale Sergio and Scarpa Giuseppe. 2020. A detail-preserving cross-scale learning strategy for CNN-based pansharpening. Remote Sens. 12, 3 (2020), 348.Google ScholarGoogle ScholarCross RefCross Ref
  44. [44] Vivone Gemine, Alparone Luciano, Chanussot Jocelyn, Mura Mauro Dalla, Garzelli Andrea, Licciardi Giorgio A, Restaino Rocco, and Wald Lucien. 2015. A critical comparison among pansharpening algorithms. IEEE Trans. Geosci. Remote Sens. 53, 5 (2015), 25652586.Google ScholarGoogle ScholarCross RefCross Ref
  45. [45] Vivone Gemine, Restaino Rocco, and Chanussot Jocelyn. 2018. Full scale regression-based injection coefficients for panchromatic sharpening. IEEE Trans. Image Process. 27, 7 (2018), 34183431.Google ScholarGoogle Scholar
  46. [46] Vivone Gemine, Restaino Rocco, Mura Mauro Dalla, Licciardi Giorgio, and Chanussot Jocelyn. 2014. Contrast and error-based fusion schemes for multispectral image pansharpening. IEEE Geosci. Remote Sens. Lett. 11, 5 (2014), 930934.Google ScholarGoogle ScholarCross RefCross Ref
  47. [47] Wald Lucien. 2002. Data Fusion: Definitions and Architectures: Fusion of Images of Different Spatial Resolutions. Presses des MINES.Google ScholarGoogle Scholar
  48. [48] Wang Zhou and Bovik Alan C.. 2002. A universal image quality index. IEEE Sign. Process. Lett. 9, 3 (2002), 8184.Google ScholarGoogle ScholarCross RefCross Ref
  49. [49] Wei Yancong, Yuan Qiangqiang, Shen Huanfeng, and Zhang Liangpei. 2017. Boosting the accuracy of multispectral image pansharpening by learning a deep residual network. IEEE Geosci. Remote Sens. Lett. 14, 10 (2017), 17951799.Google ScholarGoogle ScholarCross RefCross Ref
  50. [50] Xie Junyuan, Xu Linli, and Chen Enhong. 2012. Image denoising and inpainting with deep neural networks. In Advances in Neural Information Processing Systems. 341349.Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. [51] Yuhas Roberta H., Goetz Alexander F. H., and Boardman Joe W.. 1992. Discrimination among semi-arid landscape endmembers using the spectral angle mapper (SAM) algorithm. In Proc. Summaries 3rd Annu. JPL Airborne Geosci. Workshop, vol. 1. 147–149.Google ScholarGoogle Scholar
  52. [52] Sun Zhan-Li, Huang De-Shuang, Cheung Yiu-Ming, Liu Jiming, and Huang Guang-Bin. 2005. Using FCMC, FVS, and PCA techniques for feature extraction of multispectral images. IEEE Geosci. Remote Sens. Lett. 2, 2 (April 2005), 108112. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  53. [53] Zhang Liangpei, Shen Huanfeng, Gong Wei, and Zhang Hongyan. 2012. Adjustable model-based fusion method for multispectral and panchromatic images. IEEE Trans. Syst. Man Cybernet. B 42, 6 (2012), 16931704.Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. [54] Zhou W.. 2013. An object-based approach for urban land cover classification: Integrating LiDAR height and intensity data. IEEE Geosci. Remote Sens. Lett. 10, 4 (July 2013), 928931. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  55. [55] Zhu Xiao Xiang and Bamler Richard. 2012. A sparse image fusion algorithm with application to pan-sharpening. IEEE Trans. Geosci. Remote Sens. 51, 5 (2012), 28272836.Google ScholarGoogle ScholarCross RefCross Ref
  56. [56] Zhuang H., Deng K., Fan H., and Yu M.. 2016. Strategies combining spectral angle mapper and change vector analysis to unsupervised change detection in multispectral images. IEEE Geosci. Remote Sens. Lett. 13, 5 (May 2016), 681685. DOI:Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Pansharpening Scheme Using Bi-dimensional Empirical Mode Decomposition and Neural Network

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Multimedia Computing, Communications, and Applications
      ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 18, Issue 4
      November 2022
      497 pages
      ISSN:1551-6857
      EISSN:1551-6865
      DOI:10.1145/3514185
      • Editor:
      • Abdulmotaleb El Saddik
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 4 March 2022
      • Revised: 1 December 2021
      • Accepted: 1 December 2021
      • Received: 1 June 2021
      Published in tomm Volume 18, Issue 4

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Refereed
    • Article Metrics

      • Downloads (Last 12 months)89
      • Downloads (Last 6 weeks)5

      Other Metrics

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!