skip to main content
research-article

HIFGAN: A High-Frequency Information-Based Generative Adversarial Network for Image Super-Resolution

Published:11 May 2023Publication History
Skip Abstract Section

Abstract

Since the neural network was introduced into the super-resolution (SR) field, many SR deep models have been proposed and have achieved excellent results. However, there are two main drawbacks: one is that the methods based on the best peak-signal-to-noise ratio (PSNR) do not have enough comfortable visual quality; the other is that although the SR models based on generative adversarial network (GAN) have satisfactory visual quality, the structure of the reconstructed image has apparent defects. Therefore, according to the characteristics that human eyes are sensitive to high-frequency components in images, this article proposes an improved image SRGAN model based on high-frequency information fusion (HIFGAN). It builds a feature extraction network for high-frequency information fusion by designing a lightweight spatial attention module and improving the network architecture of enhanced super-resolution GAN (ESRGAN). It makes the generator in the GAN network have better feature recovery ability, reduces the dependence of the later training on the decider and loss function, and makes the generated image structure more consistent with the real situation. In addition, we build a high-frequency loss function to optimize the training of the generator network. Detailed experimental results show that HIFGAN performs excellently in both objective criterion evaluation and subjective visual effect. Compared with the state-of-the-art GAN-based SR networks, the reconstructed image by our model is more precise and complete in texture details.

REFERENCES

  1. [1] Dong Chao, Loy Chen Change, He Kaiming, and Tang Xiaoou. 2014. Learning a deep convolutional network for image super-resolution. In European Conference on Computer Vision. Springer, Cham. 184199. Retrieved Jan 10, 2020 from Google ScholarGoogle ScholarCross RefCross Ref
  2. [2] Yang Xin, Li Hengrui, and Li Xiaochuan.. 2022. Lightweight image super-resolution with feature cheap convolution and attention mechanism. Cluster Computing 25 (2022), 116. Retrieved Jan 25, 2022 from Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. [3] Lim Bee, Son Sanghyun, Kim Heewon, Nah Seungjun, and Lee Kyoung Mu. 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 136144. Retrieved Jan 10, 2013 from Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Yang Xin, Zhang Yifan, Guo Yingqing, and Zhou Dake. 2021. An image super-resolution deep learning network based on multi-level feature extraction module. Multimedia Tools and Applications 80, 5 (2021), 70637075. Retrieved Dec 22, 2021 from Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. [5] Ledig Christian, Theis Lucas, Huszár Ferenc, Caballero Jose, Cunningham Andrew, Acosta Alejandro, Aitken Andrew, and Tejani Alykhan. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 46814690. Retrieved Jan 10, 2020 from Google ScholarGoogle ScholarCross RefCross Ref
  6. [6] Goodfellow Ian, Pouget-Abadie Jean, Mirza Mehdi, Xu Bing, Warde-Farley David, and Ozair Sherjil. 2020. Generative adversarial networks. Communications of the ACM 63, 11 (2020), 139144. Retrieved Dec 22, 2020 from Google ScholarGoogle ScholarCross RefCross Ref
  7. [7] Wang Xintao, Yu Ke, Dong Chao, and Loy Chen Change. 2018. Recovering realistic texture in image super-resolution by deep spatial feature transform. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 606615. Retrieved Jan 10, 2020 from Google ScholarGoogle ScholarCross RefCross Ref
  8. [8] Simonyan K. and Zisserman A.. 2014. Very deep convolutional networks for large-scale image recognition. arXiv.1409.1556. Retrieved Jan 10, 2020 from Google ScholarGoogle ScholarCross RefCross Ref
  9. [9] He Kaiming, Zhang Xiangyu, Ren Shaoqing, and Sun Jian. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770778. Retrieved Dec 10, 2020 from Google ScholarGoogle ScholarCross RefCross Ref
  10. [10] Shi Wenzhe, Caballero Jose, Ferenc Huszár, and Totz Johannes. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 18741883. Retrieved July 15, 2020 from Google ScholarGoogle ScholarCross RefCross Ref
  11. [11] Wang Xintao, Yu Ke, Wu Shixiang, Gu Jinjin, and Liu Yiha. 2018. ESRGAN: 2018. Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV’18). Retrieved July 15, 2020 from Google ScholarGoogle ScholarCross RefCross Ref
  12. [12] Liu Jie, Zhang Wenjie, Tang Yuting, Tang Jie, and Wu Gangshan. 2020. Residual feature aggregation network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 23592368. Retrieved Dec 15, 2020 from Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Jaderberg M., Simonyan K., and Zisserman A.. 2015. Spatial transformer networks. Advances in Neural Information Processing Systems. Montreal: [s.n.], 20172025. Retrieved July 10, 2020 from Google ScholarGoogle ScholarCross RefCross Ref
  14. [14] Hwang J. W. and Lee H. S.. 2004. Adaptive image interpolation based on local gradient features. IEEE Signal Processing Letters 11, 3 (2004), 359362. Retrieved Jan 18, 2020 from Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Lai Wei-Sheng, Huang Jia-Bin, Ahuja Narendra, and Yang Ming-Hsuan. 2017. Deep Laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 624632. Retrieved July 10, 2020 from Google ScholarGoogle ScholarCross RefCross Ref
  16. [16] Kim J., Kwon Lee J., and Mu Lee K.. 2016. Deeply-recursive convolutional network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 16371645. Retrieved July 12, 2020 from Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] Tai Y., Yang J., and Liu X.. 2017. Image super-resolution via deep recursive residual net- work. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 31473155. Retrieved July 10, 2020 from https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8099781.Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Tai Ying, Yang Jian, Liu Xiaoming, and Xu Chunyan. 2017. MemNet: A persistent memory network for image restoration. In Proceedings of the IEEE International Conference on Computer Vision. 4539--4547. Retrieved July 12, 2020 from Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Hui Z., Wang X., and Gao X.. 2018. Two-stage convolutional network for image super-resolution. In ICPR. 26702675. Retrieved July 12, 2020 from Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Haris M., Shakhnarovich G., and Ukita N.. 2019. Deep back-projection networks for single image super-resolution. arXiv:1904.05677. Retrieved July 15, 2020 from Google ScholarGoogle ScholarCross RefCross Ref
  21. [21] Zhang Yulun, Tian Yapeng, Kong Yu, Zhong Bineng, and Fu Yun. 2018. Residual dense network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 24722481. Retrieved Dec 20, 2021 from Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Yang Xin, Li Xiaochuan, Li Zhiqiang, and Zhou Dake. 2021. Image super-resolution based on deep neural network of multiple attention mechanism. Journal of Visual Communication and Image Representation 75 (2021), 103019. Retrieved July 16, 2020 from Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Bevilacqua Marco, Roumy Aline, Guillemot Christine, and Marie-Line Alberi Morel. 2012. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. Retrieved July 16, 2020 from http://www.bmva.org/bmvc/2012/BMVC/paper135/abstract135.pdf.Google ScholarGoogle Scholar
  24. [24] Zeyde Roman, Elad Michael, and Protter Matan. 2010. On Single Image Scale-Up Using Sparse Representations. International Conference on Curves and Surfaces. Springer, Berlin, 711730. Retrieved July 16, 2020 from Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. [25] P. Arbelaáez, M. Maire, C. Fowlkes, and Malik J.. 2010. Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 5 (2010), 898916. Retrieved July 16, 2020 from Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. [26] Huang J. B., Singh A., and Ahuja N.. 2015. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 51975206. Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Matsui Yusuke, Ito Kota, Aramaki Yuji, Fujimoto Azuma, Ogawa Toru, Yamasaki Toshihiko, and Aizawa Kiyoharu. 2017. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications 76, 20 (2017), 2181121838. Retrieved July 16, 2020 from Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. [28] Zhang Richard, Isola Phillip, Efros Alexei A., Shechtman Eli, and Wang Oliver. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 586595. Retrieved July 18, 2020 from Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Blau Y. and T. Michaeli . 2018. The perception-distortion tradeoff. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 62286237. Retrieved July 18, 2020 from Google ScholarGoogle ScholarCross RefCross Ref
  30. [30] Muqeet Abdul, Hwang Jiwon, Yang Subin, Kang JungHeum, Kim Yongwoo, and Bae Sung-Ho. 2020. Multi-attention based ultra lightweight image super-resolution. ECCV 2020 Workshops. Retrieved Dec 15, 2020 from Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. [31] Wang Xintao, Xie Liangbin, Dong Chao, and Shan Ying. 2021. Real-ESRGAN: Training real-world blind super-resolution with pure synthetic data. Retrieved Jan 10, 2022 from Google ScholarGoogle ScholarCross RefCross Ref
  32. [32] Ma Cheng, Rao Yongming, Cheng Yean, Chen Ce, Lu Jiwen, and Zhou Jie. 2020. Structure-preserving super resolution with gradient guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 77697778. Retrieved Jan 10, 2022 from https://ieeexplore.ieee.org/document/7938379.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. HIFGAN: A High-Frequency Information-Based Generative Adversarial Network for Image Super-Resolution

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Multimedia Computing, Communications, and Applications
          ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 5
          September 2023
          262 pages
          ISSN:1551-6857
          EISSN:1551-6865
          DOI:10.1145/3585398
          • Editor:
          • Abdulmotaleb El Saddik
          Issue’s Table of Contents

          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 11 May 2023
          • Online AM: 4 January 2023
          • Accepted: 23 December 2022
          • Revised: 26 October 2022
          • Received: 28 December 2021
          Published in tomm Volume 19, Issue 5

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Full Text

        View this article in Full Text.

        View Full Text

        HTML Format

        View this article in HTML Format .

        View HTML Format
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!