skip to main content
research-article

MFGAN: Multi-modal Feature-fusion for CT Metal Artifact Reduction Using GANs

Published:23 January 2023Publication History
Skip Abstract Section

Abstract

Due to the existence of metallic implants in certain patients, the Computed Tomography (CT) images from these patients are often corrupted by undesirable metal artifacts, which causes severe problem of metal artifact. Although many methods have been proposed to reduce metal artifact, reduction is still challenging and inadequate. Some reduced results are suffering from symptom variance, second artifact, and poor subjective evaluation. To address these, we propose a novel method based on generative adversarial nets (GANs) to reduce metal artifacts. Specifically, we firstly encode interactive information (text) and imaging CT (image) to yield multi-modal feature-fusion representation, which overcomes representative ability limitation of single-modal CT images. The incorporation of interaction information constrains feature generation, which ensures symptom consistency between corrected and target CT. Then, we design an enhancement network to avoid second artifact and enhance edge as well as suppress noise. Besides, three radiology physicians are invited to evaluate the corrected CT image. Experiments show that our method gains significant improvement over other methods. Objectively, ours achieves an average increment of 7.44% PSNR and 6.12% SSIM on two medical image datasets. Subjectively, ours outperforms others in comparison in term of sharpness, resolution, invariance, and acceptability.

REFERENCES

  1. [1] Kalender Willi A., Hebel Robert, and Ebersberger Johannes. 1987. Reduction of CT artifacts caused by metallic implants. Radiology 164, 2 (1987), 576577.Google ScholarGoogle ScholarCross RefCross Ref
  2. [2] Arjovsky Martin, Chintala Soumith, and Bottou Léon. 2017. Wasserstein GAN. In International Conference on Machine Learning. 214223.Google ScholarGoogle Scholar
  3. [3] Bal Matthieu, Celik Hasan, Subramanyan Krishna, Kai Eck, and Spies Lothar. 2005. A radial adaptive filter for metal artifact reduction. In SPIE Medical Imaging. 20752082.Google ScholarGoogle Scholar
  4. [4] Desideri JeanAntoine. 2012. Multiple-gradient descent algorithm (MGDA) for multiobjective optimization. Comptes Rendus Mathematique 350, 5 (2012), 313318.Google ScholarGoogle ScholarCross RefCross Ref
  5. [5] Gjesteby Lars, Yang Qingsong, Yan Xi, Zhou Ye, and Wang Ge. 2017. Deep learning methods to guide CT image reconstruction and reduce metal artifacts. In SPIE Medical Imaging. 2127.Google ScholarGoogle Scholar
  6. [6] Goodfellow Ian J., Pouget-Abadie Jean, Mirza Mehdi, Xu Bing, Warde-Farley David, Ozair Sherjil, Courville Aaron, and Bengio Yoshua. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. 26722680.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. [7] Heusel Martin, Ramsauer Hubert, Unterthiner Thomas, and Nessler Bernhard. 2017. GANs trained by a two time-scale update rule converge to a Nash equilibrium. In Advances in Neural Information Processing Systems. 26722680.Google ScholarGoogle Scholar
  8. [8] Hoffer Elad, Hubara Itay, and Soudry Daniel. 2017. Train longer, generalize better: Closing the generalization gap in large batch training of neural networks. In Advances in Neural Information Processing Systems. 17311741.Google ScholarGoogle Scholar
  9. [9] Jiang Kui, Wang Zhongyuan, Yi Peng, Wang Guangcheng, Lu Tao, and Jiang Junjun. 2019. Edge-enhanced GAN for remote sensing image superresolution. IEEE Transactions on Geoscience and Remote Sensing 57, 8 (2019), 55795812.Google ScholarGoogle ScholarCross RefCross Ref
  10. [10] Kachelrie Marc, Watzke Oliver, and Kalender Willi A.. 2001. Generalized multi-dimensional adaptive filtering for conventional and spiral single-slice, multi-slice, and cone-beam CT. Medical Physics 28, 4 (2001), 475490.Google ScholarGoogle ScholarCross RefCross Ref
  11. [11] Karras Tero, Aila Timo, Laine Samuli, and Lehtinen Jaakko. 2017. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations. 126.Google ScholarGoogle Scholar
  12. [12] Ledig Christian, Theis Lucas, Huszar Ferenc, Caballero Jose, Cunningham Andrew, Acosta Alejandro, Aitken Andrew, Tejani Alykhan, Totz Johannes, Wang Zehan, and Shi Wenzhe. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In IEEE Conference on Computer Vision and Pattern Recognition. 46814690.Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Lemmens Catherine, Faul David, and Nuyts Johan. 2009. Suppression of metal artifacts in CT using a reconstruction procedure that combines MAP and projection completion. IEEE Transactions on Medical Imaging 28, 2 (2009), 250260.Google ScholarGoogle ScholarCross RefCross Ref
  14. [14] Levin Anat, Lischinski Dani, and Weiss Yair. 2007. A closed-form solution to natural image matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2 (2007), 228242.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. [15] Liao Haofu, Lin WeiAn, Zhou Kevin, and Luo Jiebo. 2020. ADN: Artifact disentanglement network for unsupervised metal artifact reduction. IEEE Transactions on Medical Imaging 39, 3 (2020), 634643.Google ScholarGoogle ScholarCross RefCross Ref
  16. [16] Lin Weian, Liao Haofu, Peng Cheng, Sun Xiaohang, Zhang Jingdan, Luo Jiebo, Chellappa Rama, and Zhou Shaohua. 2019. DuDoNet: Dual domain network for CT metal artifact reduction. In IEEE Conference on Computer Vision and Pattern Recognition. 1051210521.Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] Meyer Esther, Raupach Rainer, Lell Michael, Schmidt Bernhard, and Kachelrie Marc. 2010. Normalized metal artifact reduction (NMAR) in computed tomography. Medical Physics 37, 10 (2010), 54825493.Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Park Hyoung Suk, Hwang Dosik, and Seo Jin Keun. 2016. Metal artifact reduction for polychromatic X-ray CT based on a beam-hardening corrector. IEEE Transactions on Medical Imaging 35, 2 (2016), 480487.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Prell Daniel, Kyriakou Yiannis, Beister Marcel, and Kalender Willi A.. 2009. A novel forward projection-based metal artifact reduction method for flat-detector computed tomography. Physics in Medicine and Biology 54, 21 (2009), 65756591.Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Salimans Tim, Goodfellow Ian, Zaremba Wojciech, Cheung Vicki, Radford Alec, and Chen Xi. 2017. Improved techniques for training GANs. In Advances in Neural Information Processing Systems. 22342242.Google ScholarGoogle Scholar
  21. [21] Sheikh Hamid Rahim, Bovik Alan Conrad, and Veciana Gustavo de. 2005. An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Transactions on Image Processing 14, 12 (2005), 21172128.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. [22] Shmelkov Konstantin, Schmid Cordelia, and Alahari Karteek. 2018. How good is my GAN? In IEEE Conference on Computer Vision and Pattern Recognition. 36543668.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. [23] Wang Jianing, Zhao Yiyuan, Noble Jack, and Dawant Benoit. 2018. Conditional generative adversarial networks for metal artifact reduction in CT images of the ear. In International Conference on Medical Image Computing and Computer-Assisted Intervention. 311.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. [24] Wang Xin. 2007. Laplacian operator-based edge detectors. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 5 (2007), 886890.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. [25] Wang Zhou, Simoncelli Eero P., and Bovik Alan C.. 2003. Multiscale structural similarity for image quality assessment. In IEEE Asilomar Conference on Signals, Systems and Computers. 13981402.Google ScholarGoogle ScholarCross RefCross Ref
  26. [26] Xie Yutong, Zhang Jianpeng, Xia Yong, Fulham Michael, and Zhang Yanning. 2018. Fusing texture, shape and deep model-learned information at decision level for automated classification of lung nodules on chest CT. Information Fusion 42, 1 (2018), 102110.Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Xu Liming, Zeng Xianhua, Zhang He, Li Weisheng, Huang Zhiwei, and Lei Jianbo. 2020. BPGAN: Bidirectional CT-to-MRI prediction using multi-generative multi-adversarial nets with spectral normalization and localization. Neural Networks 128, 12 (2020), 8296.Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] Xu Shiyu and Dang Hao. 2018. Deep residual learning enabled metal artifact reduction in CT. In SPIE Medical Imaging. 950955.Google ScholarGoogle Scholar
  29. [29] Zhang Haimiao, Dong Bin, and Liu Baodong. 2018. A reweighted joint spatial-radon domain CT image reconstruction model for metal artifact reduction. Siam Journal on Imaging Sciences 11, 1 (2018), 707733.Google ScholarGoogle ScholarCross RefCross Ref
  30. [30] Zhang Lin, Zhang Lei, Mou Xuanqin, and Zhang David. 2011. FSIM: A feature similarity index for image quality assessment. IEEE Transactions on Image Processing 20, 8 (2011), 23782386.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. [31] Zhang Yanbo and Yu Hengyong. 2018. Convolutional neural network based metal artifact reduction in X-ray computed tomography. IEEE Transactions on Medical Imaging 37, 10 (2018), 13701381.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. MFGAN: Multi-modal Feature-fusion for CT Metal Artifact Reduction Using GANs

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Multimedia Computing, Communications, and Applications
        ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 1s
        February 2023
        504 pages
        ISSN:1551-6857
        EISSN:1551-6865
        DOI:10.1145/3572859
        • Editor:
        • Abdulmotaleb El Saddik
        Issue’s Table of Contents

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 23 January 2023
        • Online AM: 31 March 2022
        • Accepted: 21 March 2022
        • Revised: 26 February 2022
        • Received: 17 October 2021
        Published in tomm Volume 19, Issue 1s

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Refereed

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      View Full Text

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!