Abstract
Multi-modal medical image fusion is a long-standing important research topic that can obtain informative medical images and assist doctors diagnose and treat diseases more efficiently. However, most fusion methods extract and fuse features by subjectively defining constraints, which easily distorts the unique information of source images. In this work, we present a novel end-to-end unsupervised network to fuse multi-modal medical images. It is composed of a generator and two symmetrical discriminators. The former aims to generate a ”real-like” fused image based on a specifically designed content and structure loss, while the latter are devoted to distinguishing the differences between the fused image and the source ones. They are trained alternately until discriminators cannot distinguish the fused image from the source ones. In addition, the symmetrical discriminator scheme is conducive to maintaining the feature consistency among different modalities. More importantly, to enhance the retention degree of texture details, U-Net is adopted as the generator heuristically, where the up-sampling method is modified to bilinear interpolation for avoiding checkerboard artifacts. As for the optimization, we define the content loss function, which preserves the gradient information and pixel activity of source images. Both visual analysis and quantitative evaluation of experimental results show the superiority of our method as compared to the cutting-edge baselines.
- [1] . 2017. Wasserstain GAN. arXiv preprint arXiv:1701.07875. Retrieved from https://arxiv.org/abs/1701.07875.Google Scholar
- [2] . 2013. Directive contrast based multimodal medical image fusion in NSCT domain. IEEE Trans. Multimedia 15, 5 (2013), 1014–1024.Google Scholar
Digital Library
- [3] . 2021. Generative adversarial networks and their applications in image generation. Chin. J. Comput. 44, 2 (2021), 347–369.Google Scholar
- [4] . 1995. Image quality measures and their performance. IEEE Trans. Commun. 43, 12 (1995), 2959–2965.Google Scholar
Cross Ref
- [5] . 2019. A semantic-based medical image fusion.
arXiv preprint arXiv:1906.00225 . Retrieved from https://arxiv.org/abs/1906.00225.Google Scholar - [6] . 2014. Generative adversarial networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems.2672–2680.Google Scholar
- [7] . 2013. A new image fusion performance metric based on visual information fidelity. Inf. Fusion 14, 2 (2013), 127–135.Google Scholar
Digital Library
- [8] . 2021. Multimodal medical image fusion review: Theoretical background and recent advances. Sign. Process. 183 (2021), 108036.Google Scholar
Cross Ref
- [9] . 2021. PAS-MEF: Multi-exposure image fusion based on principal component analysis, adaptive well-exposedness and saliency map. In Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing. 2345–2349.Google Scholar
- [10] . 2015. Joint patch clustering-based dictionary learning for multimodal image fusion. Inf. Fusion 27 (2015), 198–214.Google Scholar
Digital Library
- [11] . 1979. Development of a NAO humanoid based medical assistant. In Proceedings of the Advances in Robotics.1–6.Google Scholar
- [12] . 2019. Zero-learning fast medical image fusion. In Proceedings of the 22th International Conference on Information Fusion.1–8.Google Scholar
Cross Ref
- [13] . 2020. Laplacian redecomposition for multimodal medical image fusion. IEEE Trans. Instrum. Meas. 69, 9 (2020), 6880–6890.Google Scholar
Cross Ref
- [14] . 2016. Image fusion with convolutional sparse representation. IEEE Sign. Process. Lett. 23, 12 (2016), 1882–1886.Google Scholar
Cross Ref
- [15] . 2019. FusionGAN: A generative adversarial network for infrared and visible image fusion. Inf. Fusion 48 (2019), 11–26.Google Scholar
Digital Library
- [16] . 2020. DDcGAN: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Trans. Image Process. 29 (2020), 4980–4995.Google Scholar
Digital Library
- [17] . 2017. Infrared and visible image fusion based on visual saliency map and weighted least square optimization. Infrar. Phys. Technol. 82 (2017), 8–17.Google Scholar
Cross Ref
- [18] . 2017. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision.2794–2802.Google Scholar
Cross Ref
- [19] . 2005. Image blockiness evaluation based on sobel operator. In Proceedings of the IEEE International Conference on Image Processing.386–389.Google Scholar
Cross Ref
- [20] . 2002. Information measure for performance of image fusion. Electron. Lett. 38, 7 (2002), 313–315.Google Scholar
Cross Ref
- [21] . 2016. Unsupervised representation learning with deep convolutional generative adversarial networks.
arXiv preprint arXiv:1511.06434 . Retrieved from https://arxiv.org/abs/1511.06434.Google Scholar - [22] . 1997. In-fibre bragg grating sensors. Meas. Sci. Technol. 8, 4 (1997), 355–375.Google Scholar
Cross Ref
- [23] . 2013. Characterization and computation of local nash equilibria in continuous games. In Proceedings of the 51st Annual Allerton Conference on Communication, Control, and Computing.917–924.Google Scholar
Cross Ref
- [24] . 2008. Assessment of image fusion procedures using entropy, image quality, and multispectral classification. J. Appl. Remote Sens. 2, 1 (2008), 1–28.Google Scholar
- [25] . 2015. U-Net: Convolutional networks for biomedical image segmentation. In Proceedings of Medical Image Computing and Computer-Assisted Intervention.234–241.Google Scholar
Cross Ref
- [26] . 2020. Symbiotic attention with privileged information for egocentric action recognition. In Proceedings of the AAAI Conference on Artificial Intelligence. 12249–12256.Google Scholar
- [27] . 2021. Survey on deep multi-modal data analytics: Collaboration, rivalry, and fusion. ACM Trans. Multimedia Comput. Commun. Appl. 17, 1s (2021), 1–25.Google Scholar
Digital Library
- [28] . 2004. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13, 4 (2004), 600–612.Google Scholar
Digital Library
- [29] . 2000. Objective image fusion performance measure. Electron. Lett. 36 (2000), 308–309.Google Scholar
Cross Ref
- [30] . 2021. Multiple knowledge representation for big data artificial intelligence: framework, applications, and case studies. Front. Inform. Tech. El. 22, 12 (2021), 1551–1558.Google Scholar
Cross Ref
- [31] . 2021. Medical image fusion method by deep learning. Int. J. Cogn. Comput. Eng. 2 (2021), 21–29.Google Scholar
- [32] . 2018. Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain. IEEE Trans. Instrum. Meas. 68, 1 (2018), 49–64.Google Scholar
Cross Ref
- [33] . 2022. Auto-weighted sample-level fusion with anchors for incomplete multi-view clustering. Pattern Recogn. 130 (2022), 108772.Google Scholar
Digital Library
- [34] . 2021. Fine-grained similarity fusion for multi-view spectral clustering. Inf. Sci. 568 (2021), 350–368.Google Scholar
- [35] . 2021. Image fusion meets deep learning: A survey and perspective. Inf. Fusion 76, 11 (2021), 323–336.Google Scholar
Digital Library
- [36] . 2019. Rethinking the image fusion: A fast unified image fusion network based on proportional maintenance of gradient and intensity. In Proceedings of the AAAI Conference on Artificial Intelligence.12797–12804.Google Scholar
- [37] . 2020. IFCNN: A general image fusion framework based on convolutional neural network. Inf. Fusion 54 (2020), 99–118.Google Scholar
Cross Ref
- [38] . 2016. Perceptual fusion of infrared and visible images through a hybrid multi-scale decomposition with Gaussian and bilateral filters. Inf. Fusion 30 (2016), 15–26.Google Scholar
Digital Library
Index Terms
DDIFN: A Dual-discriminator Multi-modal Medical Image Fusion Network
Recommendations
MMFGAN: A novel multimodal brain medical image fusion based on the improvement of generative adversarial network
AbstractIn recent years, the multimodal medical imaging assisted diagnosis and treatment technology has developed rapidly. In brain disease diagnosis, CT-SPECT, MRI-PET and MRI-SPECT fusion images are more favored by brain doctors because they contain ...
Medical Image Fusion in the NSST Domain with AR-Improved PA-PCNN
IMIP 2022: 2022 4th International Conference on Intelligent Medicine and Image ProcessingTo present a medical image fusion method based on the non-subsampled shearlet transform (NSST) and improved parametric adaptive pulse-coupled neural network (PA-PCNN) for CT and MRI, which makes the fused images clearer and contains more information. ...
Medical image fusion based on convolutional neural networks and non-subsampled contourlet transform
Highlights- Solved the problem that CNN cannot be directly used for medical image fusion.
- ...
AbstractAlthough many powerful convolutional neural networks (CNN) have been applied to various image processing fields, due to the lack of datasets for network training and the significant different intensities of diverse multi-modal source ...






Comments