skip to main content
research-article

Structure-aware Meta-fusion for Image Super-resolution

Published:16 February 2022Publication History
Skip Abstract Section

Abstract

There are two main categories of image super-resolution algorithms: distortion oriented and perception oriented. Recent evidence shows that reconstruction accuracy and perceptual quality are typically in disagreement with each other. In this article, we present a new image super-resolution framework that is capable of striking a balance between distortion and perception. The core of our framework is a deep fusion network capable of generating a final high-resolution image by fusing a pair of deterministic and stochastic images using spatially varying weights. To make a single fusion model produce images with varying degrees of stochasticity, we further incorporate meta-learning into our fusion network. Once equipped with the kernel produced by a kernel prediction module, our meta fusion network is able to produce final images at any desired level of stochasticity. Experimental results indicate that our meta fusion network outperforms existing state-of-the-art SISR algorithms on widely used datasets, including PIRM-val, DIV2K-val, Set5, Set14, Urban100, Manga109, and B100. In addition, it is capable of producing high-resolution images that achieve low distortion and high perceptual quality simultaneously.

REFERENCES

  1. [1] Agustsson Eirikur and Timofte Radu. 2017. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 126135.Google ScholarGoogle ScholarCross RefCross Ref
  2. [2] Andrychowicz Marcin, Denil Misha, Gomez Sergio, Hoffman Matthew W., Pfau David, Schaul Tom, Shillingford Brendan, and Freitas Nando De. 2016. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems. 39813989. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. [3] Bai Yancheng, Zhang Yongqiang, Ding Mingli, and Ghanem Bernard. 2018. Finding tiny faces in the wild with generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2130.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Bevilacqua Marco, Roumy Aline, Guillemot Christine, and Alberi-Morel Marie Line. 2012. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In Proceedings British Machine Vision Conference. 135.1–135.10.Google ScholarGoogle Scholar
  5. [5] Blau Yochai, Mechrez Roey, Timofte Radu, Michaeli Tomer, and Zelnik-Manor Lihi. 2018. The 2018 PIRM challenge on perceptual image super-resolution. In Proceedings of the European Conference on Computer Vision (ECCV’18). 00.Google ScholarGoogle Scholar
  6. [6] Blau Yochai and Michaeli Tomer. 2018. The perception-distortion tradeoff. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 62286237.Google ScholarGoogle ScholarCross RefCross Ref
  7. [7] Chang Hong, Yeung Dit-Yan, and Xiong Yimin. 2004. Super-resolution through neighbor embedding. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’04), Vol. 1. IEEE, I–I.Google ScholarGoogle ScholarCross RefCross Ref
  8. [8] Choi Jun-Ho, Kim Jun-Hyuk, Cheon Manri, and Lee Jong-Seok. 2020. Deep learning-based image super-resolution considering quantitative and perceptual quality. Neurocomputing 398 (2020), 347–359.Google ScholarGoogle ScholarCross RefCross Ref
  9. [9] Dong Chao, Loy Chen Change, He Kaiming, and Tang Xiaoou. 2015. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38, 2 (2015), 295307. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. [10] Egiazarian Karen, Astola Jaakko, Ponomarenko Nikolay, Lukin Vladimir, Battisti Federica, and Carli Marco. 2006. New full-reference quality metrics based on HVS. In Proceedings of the 2nd International Workshop on Video Processing and Quality Metrics, Vol. 4.Google ScholarGoogle Scholar
  11. [11] Freeman William T., Jones Thouis R., and Pasztor Egon C.. 2002. Example-based super-resolution. IEEE Comput. Graph. Appl. 22, 2 (2002), 5665. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. [12] Ge Weifeng, Gong Bingchen, and Yu Yizhou. 2018. Image super-resolution via deterministic-stochastic synthesis and local statistical rectification. ACM Trans. Graph. 37, 6 (2018), 114.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. [13] Gu Jinjin, Lu Hannan, Zuo Wangmeng, and Dong Chao. 2019. Blind super-resolution with iterative kernel correction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 16041613.Google ScholarGoogle ScholarCross RefCross Ref
  14. [14] Haris Muhammad, Shakhnarovich Gregory, and Ukita Norimichi. 2019. Recurrent back-projection network for video super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 38973906.Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Hu Xuecai, Mu Haoyuan, Zhang Xiangyu, Wang Zilei, Tan Tieniu, and Sun Jian. 2019. Meta-SR: A magnification-arbitrary network for super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 15751584.Google ScholarGoogle ScholarCross RefCross Ref
  16. [16] Huang Jia-Bin, Singh Abhishek, and Ahuja Narendra. 2015. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 51975206.Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] Ledig Christian, Theis Lucas, Huszár Ferenc, Caballero Jose, Cunningham Andrew, Acosta Alejandro, Aitken Andrew, Tejani Alykhan, Totz Johannes, Wang Zehan, et al. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 46814690.Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Li Sheng, He Fengxiang, Du Bo, Zhang Lefei, Xu Yonghao, and Tao Dacheng. 2019. Fast spatio-temporal residual network for video super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1052210531.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Lim Bee, Son Sanghyun, Kim Heewon, Nah Seungjun, and Lee Kyoung Mu. 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 136144.Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Liu Jie, Zhang Wenjie, Tang Yuting, Tang Jie, and Wu Gangshan. 2020. Residual feature aggregation network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 23592368.Google ScholarGoogle ScholarCross RefCross Ref
  21. [21] Lugmayr Andreas, Danelljan Martin, Gool Luc Van, and Timofte Radu. 2020. SRFlow: Learning the super-resolution space with normalizing flow. In Proceedings of the European Conference on Computer Vision (ECCV’20).Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. [22] Maninis Kevis-Kokitsi, Pont-Tuset Jordi, Arbeláez Pablo, and Gool Luc Van. 2016. Convolutional oriented boundaries. In Proceedings of the European Conference on Computer Vision. Springer, 580596.Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Martin David, Fowlkes Charless, Tal Doron, and Malik Jitendra. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the 8th IEEE International Conference on Computer Vision (ICCV’01), Vol. 2. IEEE, 416423.Google ScholarGoogle ScholarCross RefCross Ref
  24. [24] Matsui Yusuke, Ito Kota, Aramaki Yuji, Fujimoto Azuma, Ogawa Toru, Yamasaki Toshihiko, and Aizawa Kiyoharu. 2017. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools Appl. 76, 20 (2017), 2181121838. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. [25] Mittal Anish, Soundararajan Rajiv, and Bovik Alan C.. 2012. Making a “completely blind” image quality analyzer. IEEE Sign. Process. Lett. 20, 3 (2012), 209212.Google ScholarGoogle ScholarCross RefCross Ref
  26. [26] Michelini Pablo Navarrete, Zhu Dan, and Liu Hanwen. 2018. Multi–scale recursive and perception–distortion controllable image super–resolution. In Proceedings of the European Conference on Computer Vision (ECCV’18). 00.Google ScholarGoogle Scholar
  27. [27] Pătrăucean Viorica, Gurdjos Pierre, and Gioi Rafael Grompone Von. 2012. A parameterless line segment and elliptical arc detector with enhanced ellipse fitting. In Proceedings of the European Conference on Computer Vision. Springer, 572585. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. [28] Ponomarenko Nikolay, Silvestri Flavia, Egiazarian Karen, Carli Marco, Astola Jaakko, and Lukin Vladimir. 2007. On between-coefficient contrast masking of DCT basis functions. In Proceedings of the 3rd International Workshop on Video Processing and Quality Metrics, Vol. 4.Google ScholarGoogle Scholar
  29. [29] Ravi Sachin and Larochelle Hugo. 2016. Optimization as a model for few-shot learning. In International Conference on Learning Representations.Google ScholarGoogle Scholar
  30. [30] Soh Jae Woong, Park Gu Yong, Jo Junho, and Cho Nam Ik. 2019. Natural and realistic single image super-resolution with explicit natural manifold discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 81228131.Google ScholarGoogle ScholarCross RefCross Ref
  31. [31] Sun Jian, Xu Zongben, and Shum Heung-Yeung. 2008. Image super-resolution using gradient profile prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 18.Google ScholarGoogle Scholar
  32. [32] Tai Yu-Wing, Liu Shuaicheng, Brown Michael S., and Lin Stephen. 2010. Super resolution using edge prior and single image detail synthesis. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 24002407.Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Timofte Radu, Agustsson Eirikur, Gool Luc Van, Yang Ming-Hsuan, and Zhang Lei. 2017. Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 114125.Google ScholarGoogle ScholarCross RefCross Ref
  34. [34] Timofte Radu, Smet Vincent De, and Gool Luc Van. 2013. Anchored neighborhood regression for fast example-based super-resolution. In Proceedings of the IEEE International Conference on Computer Vision. 19201927. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. [35] Timofte Radu, Smet Vincent De, and Gool Luc Van. 2014. A+: Adjusted anchored neighborhood regression for fast super-resolution. In Proceedings of the Asian Conference on Computer Vision. Springer, 111126.Google ScholarGoogle Scholar
  36. [36] Vasu Subeesh, Madam Nimisha Thekke, and Rajagopalan A. N.. 2018. Analyzing perception-distortion tradeoff using enhanced perceptual super-resolution network. In Proceedings of the European Conference on Computer Vision (ECCV’18). 00.Google ScholarGoogle Scholar
  37. [37] Wang Longguang, Dong Xiaoyu, Wang Yingqian, Ying Xinyi, Lin Zaiping, An Wei, and Guo Yulan. 2021. Exploring sparsity in image super-resolution for efficient inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 49174926.Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Wang Longguang, Wang Yingqian, Dong Xiaoyu, Xu Qingyu, Yang Jungang, An Wei, and Guo Yulan. 2021. Unsupervised degradation representation learning for blind super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1058110590.Google ScholarGoogle ScholarCross RefCross Ref
  39. [39] Wang Longguang, Wang Yingqian, Lin Zaiping, Yang Jungang, An Wei, and Guo Yulan. 2021. Learning a single network for scale-arbitrary super-resolution. In Proceedings of the IEEE Conference on International Conference on Computer Vision. 1058110590.Google ScholarGoogle ScholarCross RefCross Ref
  40. [40] Wang Xintao, Yu Ke, Dong Chao, and Loy Chen Change. 2018. Recovering realistic texture in image super-resolution by deep spatial feature transform. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 606615.Google ScholarGoogle ScholarCross RefCross Ref
  41. [41] Wang Xintao, Yu Ke, Dong Chao, Tang Xiaoou, and Loy Chen Change. 2019. Deep network interpolation for continuous imagery effect transition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 16921701.Google ScholarGoogle ScholarCross RefCross Ref
  42. [42] Wang Xintao, Yu Ke, Wu Shixiang, Gu Jinjin, Liu Yihao, Dong Chao, Qiao Yu, and Loy Chen Change. 2018. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV’18).Google ScholarGoogle Scholar
  43. [43] Wang Yu-Xiong and Hebert Martial. 2016. Learning to learn: Model regression networks for easy small sample learning. In Proceedings of the European Conference on Computer Vision. Springer, 616634.Google ScholarGoogle ScholarCross RefCross Ref
  44. [44] Xie Saining and Tu Zhuowen. 2015. Holistically-nested edge detection. In Proceedings of the IEEE International Conference on Computer Vision. 13951403. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. [45] Xu Xuan and Li Xin. 2019. Scan: Spatial color attention networks for real single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 00.Google ScholarGoogle ScholarCross RefCross Ref
  46. [46] Yang Jianchao, Wright John, Huang Thomas, and Ma Yi. 2008. Image super-resolution as sparse representation of raw image patches. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 18.Google ScholarGoogle Scholar
  47. [47] Yıldırım Deniz and Güngör Oğuz. 2012. A novel image fusion method using IKONOS satellite images. J. Geodesy Geoinf. 1, 1 (2012), 7583.Google ScholarGoogle ScholarCross RefCross Ref
  48. [48] Zeyde Roman, Elad Michael, and Protter Matan. 2010. On single image scale-up using sparse-representations. In Proceedings of the International Conference on Curves and Surfaces. Springer, 711730. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. [49] Zhang Kai, Gool Luc Van, and Timofte Radu. 2020. Deep unfolding network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 32173226.Google ScholarGoogle ScholarCross RefCross Ref
  50. [50] Zhang Yulun, Li Kunpeng, Li Kai, Wang Lichen, Zhong Bineng, and Fu Yun. 2018. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (ECCV’18). 286301.Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. [51] Zhang Zhifei, Wang Zhaowen, Lin Zhe, and Qi Hairong. 2019. Image super-resolution by neural texture transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 79827991.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Structure-aware Meta-fusion for Image Super-resolution

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Multimedia Computing, Communications, and Applications
        ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 18, Issue 2
        May 2022
        494 pages
        ISSN:1551-6857
        EISSN:1551-6865
        DOI:10.1145/3505207
        Issue’s Table of Contents

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 16 February 2022
        • Revised: 1 July 2021
        • Accepted: 1 July 2021
        • Received: 1 December 2020
        Published in tomm Volume 18, Issue 2

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Refereed

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      View Full Text

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!