skip to main content
research-article

DDIFN: A Dual-discriminator Multi-modal Medical Image Fusion Network

Authors Info & Claims
Published:27 February 2023Publication History
Skip Abstract Section

Abstract

Multi-modal medical image fusion is a long-standing important research topic that can obtain informative medical images and assist doctors diagnose and treat diseases more efficiently. However, most fusion methods extract and fuse features by subjectively defining constraints, which easily distorts the unique information of source images. In this work, we present a novel end-to-end unsupervised network to fuse multi-modal medical images. It is composed of a generator and two symmetrical discriminators. The former aims to generate a ”real-like” fused image based on a specifically designed content and structure loss, while the latter are devoted to distinguishing the differences between the fused image and the source ones. They are trained alternately until discriminators cannot distinguish the fused image from the source ones. In addition, the symmetrical discriminator scheme is conducive to maintaining the feature consistency among different modalities. More importantly, to enhance the retention degree of texture details, U-Net is adopted as the generator heuristically, where the up-sampling method is modified to bilinear interpolation for avoiding checkerboard artifacts. As for the optimization, we define the content loss function, which preserves the gradient information and pixel activity of source images. Both visual analysis and quantitative evaluation of experimental results show the superiority of our method as compared to the cutting-edge baselines.

REFERENCES

  1. [1] Arjovsky Martin, Chintala Soumith, and Bottou Léon. 2017. Wasserstain GAN. arXiv preprint arXiv:1701.07875. Retrieved from https://arxiv.org/abs/1701.07875.Google ScholarGoogle Scholar
  2. [2] Bhatnagar Gaurav, Wu Qingming Jonathan, and Liu Zheng. 2013. Directive contrast based multimodal medical image fusion in NSCT domain. IEEE Trans. Multimedia 15, 5 (2013), 10141024.Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. [3] Chen Foji, Zhu Feng, Qingxiao Qu, Yinming Hao, Wan Ende, and Cui Yunge. 2021. Generative adversarial networks and their applications in image generation. Chin. J. Comput. 44, 2 (2021), 347369.Google ScholarGoogle Scholar
  4. [4] Eskicioglu Ahmet and Fisher Paul. 1995. Image quality measures and their performance. IEEE Trans. Commun. 43, 12 (1995), 29592965.Google ScholarGoogle ScholarCross RefCross Ref
  5. [5] Fan Fanda, Huang Yunyou, Wang Lei, Xiong Xingwang, Jiang Zihan, Zhang Zhifei, and Zhan Jianfeng. 2019. A semantic-based medical image fusion. arXiv preprint arXiv:1906.00225. Retrieved from https://arxiv.org/abs/1906.00225.Google ScholarGoogle Scholar
  6. [6] Goodfellow Ian, Pouget-Abadie Jean, Mirza Mehdi, Xu Bing, Warde-Farley David, Ozair Sherjil, Courville Aaron, and Bengio Yoshua. 2014. Generative adversarial networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems.26722680.Google ScholarGoogle Scholar
  7. [7] Han Yu, Cai Yunze, Cao Yin, and Xu Xiaoming. 2013. A new image fusion performance metric based on visual information fidelity. Inf. Fusion 14, 2 (2013), 127135.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. [8] Hermessi Haithem, Mourali Olfa, and Zagrouba Ezzeddine. 2021. Multimodal medical image fusion review: Theoretical background and recent advances. Sign. Process. 183 (2021), 108036.Google ScholarGoogle ScholarCross RefCross Ref
  9. [9] Karakaya Diclehan, Ulucan Oguzhan, and Turkan Mehmet. 2021. PAS-MEF: Multi-exposure image fusion based on principal component analysis, adaptive well-exposedness and saliency map. In Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing. 2345–2349.Google ScholarGoogle Scholar
  10. [10] Kim Minjae, Han David, and Hanseok Ko.. 2015. Joint patch clustering-based dictionary learning for multimodal image fusion. Inf. Fusion 27 (2015), 198214.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. [11] Kumar Aditya, Patel Atman, and Dwivedy Santosha. 1979. Development of a NAO humanoid based medical assistant. In Proceedings of the Advances in Robotics.16.Google ScholarGoogle Scholar
  12. [12] Lahoud Fayez and Süsstrunk Sabine. 2019. Zero-learning fast medical image fusion. In Proceedings of the 22th International Conference on Information Fusion.18.Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Li Xiaoxiao, Guo Xiaopeng, Han Pengfei, Wang Xiang, Li Huaguang, and Luo Tao. 2020. Laplacian redecomposition for multimodal medical image fusion. IEEE Trans. Instrum. Meas. 69, 9 (2020), 68806890.Google ScholarGoogle ScholarCross RefCross Ref
  14. [14] Liu Yu, Chen Xun, Ward Rabab, and Wang Jane. 2016. Image fusion with convolutional sparse representation. IEEE Sign. Process. Lett. 23, 12 (2016), 18821886.Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Ma Jiayi, WYu Wei, Liang Pengwei, Li Chang, and Jiang Junjun. 2019. FusionGAN: A generative adversarial network for infrared and visible image fusion. Inf. Fusion 48 (2019), 1126.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. [16] Ma Jiayi, Xu Han, Jiang Junjun, Mei Xiaoguang, and Zhang Xiaoping. 2020. DDcGAN: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Trans. Image Process. 29 (2020), 49804995.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. [17] Ma Jinlei, Zhou Zhiqiang, Wang Bo, and Zong Hua. 2017. Infrared and visible image fusion based on visual saliency map and weighted least square optimization. Infrar. Phys. Technol. 82 (2017), 817.Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Mao Xudong, Li Qing, Xie Haoran, Lau Raymond, Wang Zhen, and Smolley Stephen Paul. 2017. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision.27942802.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Perra Cristian, Massidda Francesco, and Giusto Daniele D.. 2005. Image blockiness evaluation based on sobel operator. In Proceedings of the IEEE International Conference on Image Processing.386389.Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Qu Guihong, Zhang Dali, and Yan Pingfan. 2002. Information measure for performance of image fusion. Electron. Lett. 38, 7 (2002), 313315.Google ScholarGoogle ScholarCross RefCross Ref
  21. [21] Radford Alecv, Metz Luke, and Chintala Soumith. 2016. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434. Retrieved from https://arxiv.org/abs/1511.06434.Google ScholarGoogle Scholar
  22. [22] Rao Yunjiang. 1997. In-fibre bragg grating sensors. Meas. Sci. Technol. 8, 4 (1997), 355375.Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Ratliff Lillian, Burden Samuel, and Sastry Shankar. 2013. Characterization and computation of local nash equilibria in continuous games. In Proceedings of the 51st Annual Allerton Conference on Communication, Control, and Computing.917924.Google ScholarGoogle ScholarCross RefCross Ref
  24. [24] Roberts Wesley, Aardt Jan Andreas Van, and Ahmed Fethi Babikker. 2008. Assessment of image fusion procedures using entropy, image quality, and multispectral classification. J. Appl. Remote Sens. 2, 1 (2008), 128.Google ScholarGoogle Scholar
  25. [25] Ronneberger Olaf, Fischer Philipp, and Brox Thomas. 2015. U-Net: Convolutional networks for biomedical image segmentation. In Proceedings of Medical Image Computing and Computer-Assisted Intervention.234241.Google ScholarGoogle ScholarCross RefCross Ref
  26. [26] Wang Xiaohan, Zhu Linchao, Wu Yu, and Yang Yi. 2020. Symbiotic attention with privileged information for egocentric action recognition. In Proceedings of the AAAI Conference on Artificial Intelligence. 12249–12256.Google ScholarGoogle Scholar
  27. [27] Wang Yang. 2021. Survey on deep multi-modal data analytics: Collaboration, rivalry, and fusion. ACM Trans. Multimedia Comput. Commun. Appl. 17, 1s (2021), 125.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. [28] Wang Zhou, Bovik Alan Conrad, Sheikh Hamid, and Simoncelli Eero. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13, 4 (2004), 600612.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. [29] Xydeas Costas and Petrovic Vladimir. 2000. Objective image fusion performance measure. Electron. Lett. 36 (2000), 308309.Google ScholarGoogle ScholarCross RefCross Ref
  30. [30] Yang Yi, Zhuang Yueting, and Pan Yunhe. 2021. Multiple knowledge representation for big data artificial intelligence: framework, applications, and case studies. Front. Inform. Tech. El. 22, 12 (2021), 15511558.Google ScholarGoogle ScholarCross RefCross Ref
  31. [31] Yi Li, Junli Zhao, Zhihan Lv, and Jinhua Li. 2021. Medical image fusion method by deep learning. Int. J. Cogn. Comput. Eng. 2 (2021), 2129.Google ScholarGoogle Scholar
  32. [32] Yin Ming, Liu Xiaoning, Liu Yu, and Chen Xun. 2018. Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain. IEEE Trans. Instrum. Meas. 68, 1 (2018), 4964.Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Yu Xiao, Liu Hui, Lin Yuxiu, Wu Yan, and Caiming Zhang. 2022. Auto-weighted sample-level fusion with anchors for incomplete multi-view clustering. Pattern Recogn. 130 (2022), 108772.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. [34] Yu Xiao, Liu Hui, Yan Wu, and Caiming Zhang. 2021. Fine-grained similarity fusion for multi-view spectral clustering. Inf. Sci. 568 (2021), 350368.Google ScholarGoogle Scholar
  35. [35] Zhang Hao, Xu Han, Tian Xin, Jiang Junjun, and Ma Jiayi. 2021. Image fusion meets deep learning: A survey and perspective. Inf. Fusion 76, 11 (2021), 323336.Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. [36] Zhang Hao, Xu Han, Yang Xiao, Guo Xiaojie, and Ma Jiayi. 2019. Rethinking the image fusion: A fast unified image fusion network based on proportional maintenance of gradient and intensity. In Proceedings of the AAAI Conference on Artificial Intelligence.1279712804.Google ScholarGoogle Scholar
  37. [37] Zhang Yu, Liu Yu, Sun Peng, Yan Han, Zhao Xiaolin, and Zhang Li. 2020. IFCNN: A general image fusion framework based on convolutional neural network. Inf. Fusion 54 (2020), 99118.Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Zhou Zhiqiang, Wang Bo, Sun Li, and Dong Mingjie. 2016. Perceptual fusion of infrared and visible images through a hybrid multi-scale decomposition with Gaussian and bilateral filters. Inf. Fusion 30 (2016), 1526.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. DDIFN: A Dual-discriminator Multi-modal Medical Image Fusion Network

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Multimedia Computing, Communications, and Applications
      ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 4
      July 2023
      263 pages
      ISSN:1551-6857
      EISSN:1551-6865
      DOI:10.1145/3582888
      • Editor:
      • Abdulmotaleb El Saddik
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 27 February 2023
      • Online AM: 17 January 2023
      • Accepted: 30 November 2022
      • Revised: 11 August 2022
      • Received: 13 April 2022
      Published in tomm Volume 19, Issue 4

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!