skip to main content
research-article

Efficient Single-image Super-resolution Using Dual path Connections with Multiple scale Learning

Authors Info & Claims
Published:25 February 2023Publication History
Skip Abstract Section

Abstract

Deep convolutional neural networks have been demonstrated to be effective for single-image super-resolution in recent years. On the one hand, residual connections and dense connections have been used widely to ease forward information and backward gradient flows to boost performance. However, current methods use residual connections and dense connections separately in most network layers in a sub-optimal way. On the other hand, although various networks and methods have been designed to improve computation efficiency, save parameters, or utilize training data of multiple scale factors for each other to boost performance, they either do super-resolution in high-resolution space to have a high computation cost or cannot share parameters between models of different scale factors to save parameters and inference time. To tackle these challenges, we propose an efficient single-image super-resolution network using dual path connections with multiple scale learning (EMSRDPN). By introducing dual path connections inspired by Dual path Networks into EMSRDPN, it uses residual connections and dense connections in an integrated way in most network layers. Dual path connections have the benefits of both reusing common features of residual connections and exploring new features of dense connections to learn a good representation for single-image super-resolution. To utilize the feature correlation of multiple scale factors, EMSRDPN shares all network units in low-resolution space between different scale factors to learn shared features and only uses a separate reconstruction unit for each scale factor, which can utilize training data of multiple scale factors to help each other to boost performance, meanwhile, which can save parameters and support shared inference for multiple scale factors to improve efficiency. Experiments show EMSRDPN achieves better performance and comparable or even better parameter and inference efficiency over state-of-the-art methods. Code will be available at https://github.com/yangbincheng/EMSRDPN.

REFERENCES

  1. [1] Agustsson Eirikur and Timofte Radu. 2017. NTIRE 2017 challenge on single-image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW’17). 11221131.Google ScholarGoogle Scholar
  2. [2] Allebach Jan P. and Wong Ping Wah. 1996. Edge-directed interpolation. In Proceedings of the International Conference on Image Processing. IEEE Computer Society, 707710. Google ScholarGoogle ScholarCross RefCross Ref
  3. [3] Anwar Saeed and Barnes Nick. 2022. Densely residual Laplacian super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. 44 (2022), 11921204.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Bevilacqua Marco, Roumy Aline, Guillemot Christine, and Morel Marie-Line Alberi. 2012. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In Proceedings of the British Machine Vision Conference. BMVA Press, 135.1–135.10. Google ScholarGoogle ScholarCross RefCross Ref
  5. [5] Chang Hong, Yeung Dit-Yan, and Xiong Yimin. 2004. Super-resolution through neighbor embedding. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’04), Vol. 1. IEEE, I–I.Google ScholarGoogle ScholarCross RefCross Ref
  6. [6] Chen Yunpeng, Li Jianan, Xiao Huaxin, Jin Xiaojie, Yan Shuicheng, and Feng Jiashi. 2017. Dual path networks. In Advances in Neural Information Processing Systems. 44674475.Google ScholarGoogle Scholar
  7. [7] Dai Tao, Cai Jianrui, Zhang Yongbing, Xia Shu-Tao, and Zhang Lei. 2019. Second-order attention network for single-image super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19). Computer Vision Foundation/IEEE, 1106511074. Google ScholarGoogle ScholarCross RefCross Ref
  8. [8] Deng Jia, Dong Wei, Socher Richard, Li Li-Jia, Li Kai, and Fei-Fei Li. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 248255.Google ScholarGoogle Scholar
  9. [9] Dong Chao, Loy Chen Change, He Kaiming, and Tang Xiaoou. 2016. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38, 2 (2016), 295307.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. [10] Dong Chao, Loy Chen Change, and Tang Xiaoou. 2016. Accelerating the super-resolution convolutional neural network. In Proceedings of the European Conference on Computer Vision. Springer, 391407.Google ScholarGoogle ScholarCross RefCross Ref
  11. [11] Dong Weisheng, Zhang Lei, Shi Guangming, and Wu Xiaolin. 2011. Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization. IEEE Trans. Image Process. 20, 7 (2011), 18381857. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. [12] Freeman William T., Jones Thouis R., and Pasztor Egon C.. 2002. Example-based super-resolution. IEEE Comput. Graph. Appl. 22, 2 (2002), 5665.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. [13] Freeman William T. and Pasztor Egon C.. 1999. Learning low-level vision. In Proceedings of the International Conference on Computer Vision. IEEE Computer Society, 11821189. Google ScholarGoogle ScholarCross RefCross Ref
  14. [14] Fujimoto Azuma, Ogawa Toru, Yamamoto Kazuyoshi, Matsui Yusuke, Yamasaki Toshihiko, and Aizawa Kiyoharu. 2016. Manga109 dataset and creation of metadata. In Proceedings of the 1st International Workshop on coMics ANalysis, Processing and Understanding ([email protected]’16), Ogier Jean-Marc, Aizawa Kiyoharu, Kise Koichi, Burie Jean-Christophe, Yamasaki Toshihiko, and Iwata Motoi (Eds.). ACM, 2:1–2:5. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. [15] Glasner Daniel, Bagon Shai, and Irani Michal. 2009. Super-resolution from a single image. In Proceedings of the IEEE 12th International Conference on Computer Vision. IEEE, 349356.Google ScholarGoogle ScholarCross RefCross Ref
  16. [16] Haris Muhammad, Shakhnarovich Gregory, and Ukita Norimichi. 2018. Deep back-projection networks for super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 16641673.Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] He Kaiming, Zhang Xiangyu, Ren Shaoqing, and Sun Jian. 2016. Identity mappings in deep residual networks. In Proceedings of the European Conference on Computer Vision. Springer, 630645.Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Huang Jia-Bin, Singh Abhishek, and Ahuja Narendra. 2015. single-image super-resolution from transformed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 51975206.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Hui Zheng, Wang Xiumei, and Gao Xinbo. 2018. Fast and accurate single-image super-resolution via information distillation network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 723731.Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Ioffe Sergey and Szegedy Christian. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 37), Bach Francis and Blei David (Eds.). PMLR, Lille, France, 448456. Retrieved from http://proceedings.mlr.press/v37/ioffe15.html.Google ScholarGoogle Scholar
  21. [21] Irani Michal and Peleg Shmuel. 1991. Improving resolution by image registration. CVGIP: Graph. Model Image Process. 53 (1991), 231239.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. [22] Johnson Justin, Alahi Alexandre, and Fei-Fei Li. 2016. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of the European Conference on Computer Vision. Springer, 694711.Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Kim Jiwon, Lee Jung Kwon, and Lee Kyoung Mu. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 16461654.Google ScholarGoogle ScholarCross RefCross Ref
  24. [24] Kim Jiwon, Lee Jung Kwon, and Lee Kyoung Mu. 2016. Deeply recursive convolutional network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). IEEE Computer Society, 16371645. Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Kingma Diederik P. and Ba Jimmy. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR’15), Bengio Yoshua and LeCun Yann (Eds.). Retrieved from http://arxiv.org/abs/1412.6980Google ScholarGoogle Scholar
  26. [26] Lai Wei-Sheng, Huang Jia-Bin, Ahuja Narendra, and Yang Ming-Hsuan. 2017. Deep Laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 624632.Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Lai Wei-Sheng, Huang Jia-Bin, Ahuja Narendra, and Yang Ming-Hsuan. 2018. Fast and accurate image super-resolution with deep Laplacian pyramid networks. IEEE Trans. Pattern Anal. Mach. Intell. 41, 11 (2018), 25992613.Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] Ledig Christian, Theis Lucas, Huszár Ferenc, Caballero Jose, Cunningham Andrew, Acosta Alejandro, Aitken Andrew, Tejani Alykhan, Totz Johannes, Wang Zehan et al. 2017. Photo-realistic single-image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 46814690.Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Li Juncheng, Fang Faming, Mei Kangfu, and Zhang Guixu. 2018. Multi-scale residual network for image super-resolution. In Proceedings of the European Conference on Computer Vision (ECCV’18). 517532.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. [30] Li Xin and Orchard Michael T.. 2001. New edge-directed interpolation. IEEE Trans. Image Process. 10, 10 (2001), 15211527. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. [31] Li Yanchun, Cao Jianglian, Li Zhetao, Oh Sangyoon, and Komuro Nobuyoshi. 2021. Lightweight single-image super-resolution with dense connection distillation network. ACM Trans. Multimedia Comput., Commun. Appl. 17, 1s (2021), 117.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. [32] Liebel Lukas and Körner Marco. 2016. Single-image super resolution for multispectral remote sensing data using convolutional neural networks. ISPRS-Int. Arch. Photogram., Remote Sens. Spatial Info. Sci. 41 (2016), 883890.Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Lim Bee, Son Sanghyun, Kim Heewon, Nah Seungjun, and Lee Kyoung Mu. 2017. Enhanced deep residual networks for single-image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17).Google ScholarGoogle ScholarCross RefCross Ref
  34. [34] Lim Bee, Son Sanghyun, Kim Heewon, Nah Seungjun, and Lee Kyoung Mu. 2017. Enhanced deep residual networks for single-image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 136144.Google ScholarGoogle ScholarCross RefCross Ref
  35. [35] Liu Ding, Wen Bihan, Fan Yuchen, Loy Chen Change, and Huang Thomas S.. 2018. Non-local recurrent network for image restoration. In Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NeurIPS’18), Bengio Samy, Wallach Hanna M., Larochelle Hugo, Grauman Kristen, Cesa-Bianchi Nicolò, and Garnett Roman (Eds.). 16801689. Retrieved from https://proceedings.neurips.cc/paper/2018/hash/fc49306d97602c8ed1be1dfbf0835ead-Abstract.html.Google ScholarGoogle Scholar
  36. [36] Mao Xiaojiao, Shen Chunhua, and Yang Yu-Bin. 2016. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In Advances in Neural Information Processing Systems. 28022810.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. [37] Martin David R., Fowlkes Charless C., Tal Doron, and Malik Jitendra. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the 8th International Conference On Computer Vision (ICCV’01). IEEE Computer Society, 416425. Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Mei Yiqun, Fan Yuchen, and Zhou Yuqian. 2021. Image super-resolution with non-local sparse attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 35173526.Google ScholarGoogle ScholarCross RefCross Ref
  39. [39] Niu Ben, Wen Weilei, Ren Wenqi, Zhang Xiangde, Yang Lianping, Wang Shuzhen, Zhang Kaihao, Cao Xiaochun, and Shen Haifeng. 2020. Single-image super-resolution via a holistic attention network. In Proceedings of the European Conference on Computer Vision. Springer, 191207.Google ScholarGoogle Scholar
  40. [40] Qiu Yajun, Wang Ruxin, Tao Dapeng, and Cheng Jun. 2019. Embedded block residual network: A recursive restoration model for single-image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV’19). IEEE, 41794188. Google ScholarGoogle ScholarCross RefCross Ref
  41. [41] Sajjadi Msm, Scholkopf B., and Hirsch M.. 2017. EnhanceNet: Single-image super-resolution through automated texture synthesis. In Proceedings of the IEEE International Conference on Computer Vision.Google ScholarGoogle ScholarCross RefCross Ref
  42. [42] Shi Wenzhe, Caballero Jose, Huszár Ferenc, Totz Johannes, Aitken Andrew P., Bishop Rob, Rueckert Daniel, and Wang Zehan. 2016. Real-time single-image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 18741883.Google ScholarGoogle ScholarCross RefCross Ref
  43. [43] Shi Wenzhe, Caballero Jose, Ledig Christian, Zhuang Xiahai, Bai Wenjia, Bhatia Kanwal K., Marvao Antonio M. Simoes Monteiro de, Dawes Tim, O’Regan Declan P., and Rueckert Daniel. 2013. Cardiac image super-resolution with global correspondence using multi-atlas patchmatch. In Proceedings of the 16th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI’13)(Lecture Notes in Computer Science, Vol. 8151), Mori Kensaku, Sakuma Ichiro, Sato Yoshinobu, Barillot Christian, and Navab Nassir (Eds.). Springer, 916. Google ScholarGoogle ScholarCross RefCross Ref
  44. [44] Sun Jian, Xu Zongben, and Shum Heung-Yeung. 2008. Image super-resolution using gradient profile prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 18.Google ScholarGoogle Scholar
  45. [45] Tai Ying, Yang Jian, and Liu Xiaoming. 2017. Image super-resolution via deep recursive residual network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 31473155.Google ScholarGoogle ScholarCross RefCross Ref
  46. [46] Tai Ying, Yang Jian, Liu Xiaoming, and Xu Chunyan. 2017. Memnet: A persistent memory network for image restoration. In Proceedings of the IEEE International Conference on Computer Vision. 45394547.Google ScholarGoogle ScholarCross RefCross Ref
  47. [47] Tai Yu-Wing, Liu Shuaicheng, Brown Michael S., and Lin Stephen. 2010. Super resolution using edge prior and single-image detail synthesis. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 24002407.Google ScholarGoogle ScholarCross RefCross Ref
  48. [48] Timofte Radu, Smet Vincent De, and Gool Luc Van. 2014. A+: Adjusted anchored neighborhood regression for fast super-resolution. In Proceedings of the Asian Conference on Computer Vision. Springer, 111126.Google ScholarGoogle Scholar
  49. [49] Tong Tong, Li Gen, Liu Xiejie, and Gao Qinquan. 2017. Image super-resolution using dense skip connections. In Proceedings of the IEEE International Conference on Computer Vision. 47994807.Google ScholarGoogle ScholarCross RefCross Ref
  50. [50] Wang Zhaowen, Liu Ding, Yang Jianchao, Han Wei, and Huang Thomas. 2015. Deep networks for image super-resolution with sparse prior. In Proceedings of the IEEE International Conference on Computer Vision. 370378.Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. [51] Xu Lie, Choy Chiu-sing, and Li Yi-Wen. 2016. Deep sparse rectifier neural networks for speech denoising. In Proceedings of the IEEE International Workshop on Acoustic Signal Enhancement (IWAENC’16). IEEE, 15. Google ScholarGoogle ScholarCross RefCross Ref
  52. [52] Yang Bin-Cheng. 2019. Super resolution using dual path connections. In Proceedings of the 27th ACM International Conference on Multimedia (MM’19). Association for Computing Machinery, New York, NY, 15521560. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. [53] Yang Chih-Yuan and Yang Ming-Hsuan. 2013. Fast direct super-resolution by simple functions. In Proceedings of the IEEE International Conference on Computer Vision. 561568.Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. [54] Yang Jianchao, Wright John, Huang Thomas S., and Ma Yi. 2010. Image super-resolution via sparse representation. IEEE Trans. Image Process. 19, 11 (2010), 28612873.Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. [55] Zeyde Roman, Elad Michael, and Protter Matan. 2010. On single-image scale-up using sparse-representations. In Proceedings of the International Conference on Curves and Surfaces. Springer, 711730.Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. [56] Zhang Dongyang, Shao Jie, and Shen Heng Tao. 2020. Kernel attention network for single-image super-resolution. ACM Trans. Multimedia Comput., Commun., Appl. 16, 3 (2020), 115.Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. [57] Zhang Kaibing, Gao Xinbo, Tao Dacheng, and Li Xuelong. 2012. Single-image super-resolution with non-local means and steering kernel regression. IEEE Trans. Image Process. 21 (2012), 45444556.Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. [58] Zhang Kai, Zuo Wangmeng, and Zhang Lei. 2018. Learning a single convolutional super-resolution network for multiple degradations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). Computer Vision Foundation/IEEE Computer Society, 32623271. Google ScholarGoogle ScholarCross RefCross Ref
  59. [59] Zhang Lei and Wu Xiaolin. 2006. An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans. Image Process. 15 (2006), 22262238.Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. [60] Zhang Yulun, Li Kunpeng, Li Kai, Wang Lichen, Zhong Bineng, and Fu Yun. 2018. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (ECCV’18). 286301.Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. [61] Zhang Yulun, Li Kunpeng, Li Kai, Zhong Bineng, and Fu Yun. 2019. Residual non-local attention networks for image restoration. Retrieved from https://arXiv:1903.10082.Google ScholarGoogle Scholar
  62. [62] Zhang Yulun, Tian Yapeng, Kong Yu, Zhong Bineng, and Fu Yun. 2018. Residual dense network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 24722481.Google ScholarGoogle ScholarCross RefCross Ref
  63. [63] Zhang Yulun, Tian Yapeng, Kong Yu, Zhong Bineng, and Fu Yun. 2020. Residual dense network for image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 43, 7 (2020), 24802495.Google ScholarGoogle ScholarCross RefCross Ref
  64. [64] Zou Wilman W. W. and Yuen Pong Chi. 2010. Very low resolution face recognition problem. In Proceedings of the 4th IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS’10). 16.Google ScholarGoogle Scholar

Index Terms

  1. Efficient Single-image Super-resolution Using Dual path Connections with Multiple scale Learning

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Multimedia Computing, Communications, and Applications
        ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 3
        May 2023
        514 pages
        ISSN:1551-6857
        EISSN:1551-6865
        DOI:10.1145/3582886
        • Editor:
        • Abdulmotaleb El Saddik
        Issue’s Table of Contents

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 25 February 2023
        • Online AM: 3 November 2022
        • Accepted: 25 October 2022
        • Revised: 13 June 2022
        • Received: 26 December 2021
        Published in tomm Volume 19, Issue 3

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
      • Article Metrics

        • Downloads (Last 12 months)86
        • Downloads (Last 6 weeks)6

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      View Full Text

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!