Abstract
Deep convolutional neural networks have been demonstrated to be effective for single-image super-resolution in recent years. On the one hand, residual connections and dense connections have been used widely to ease forward information and backward gradient flows to boost performance. However, current methods use residual connections and dense connections separately in most network layers in a sub-optimal way. On the other hand, although various networks and methods have been designed to improve computation efficiency, save parameters, or utilize training data of multiple scale factors for each other to boost performance, they either do super-resolution in high-resolution space to have a high computation cost or cannot share parameters between models of different scale factors to save parameters and inference time. To tackle these challenges, we propose an efficient single-image super-resolution network using dual path connections with multiple scale learning (EMSRDPN). By introducing dual path connections inspired by Dual path Networks into EMSRDPN, it uses residual connections and dense connections in an integrated way in most network layers. Dual path connections have the benefits of both reusing common features of residual connections and exploring new features of dense connections to learn a good representation for single-image super-resolution. To utilize the feature correlation of multiple scale factors, EMSRDPN shares all network units in low-resolution space between different scale factors to learn shared features and only uses a separate reconstruction unit for each scale factor, which can utilize training data of multiple scale factors to help each other to boost performance, meanwhile, which can save parameters and support shared inference for multiple scale factors to improve efficiency. Experiments show EMSRDPN achieves better performance and comparable or even better parameter and inference efficiency over state-of-the-art methods. Code will be available at https://github.com/yangbincheng/EMSRDPN.
- [1] . 2017. NTIRE 2017 challenge on single-image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW’17). 1122–1131.Google Scholar
- [2] . 1996. Edge-directed interpolation. In Proceedings of the International Conference on Image Processing. IEEE Computer Society, 707–710. Google Scholar
Cross Ref
- [3] . 2022. Densely residual Laplacian super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. 44 (2022), 1192–1204.Google Scholar
Cross Ref
- [4] . 2012. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In Proceedings of the British Machine Vision Conference. BMVA Press, 135.1–135.10. Google Scholar
Cross Ref
- [5] . 2004. Super-resolution through neighbor embedding. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’04), Vol. 1. IEEE, I–I.Google Scholar
Cross Ref
- [6] . 2017. Dual path networks. In Advances in Neural Information Processing Systems. 4467–4475.Google Scholar
- [7] . 2019. Second-order attention network for single-image super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19). Computer Vision Foundation/IEEE, 11065–11074. Google Scholar
Cross Ref
- [8] . 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 248–255.Google Scholar
- [9] . 2016. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38, 2 (2016), 295–307.Google Scholar
Digital Library
- [10] . 2016. Accelerating the super-resolution convolutional neural network. In Proceedings of the European Conference on Computer Vision. Springer, 391–407.Google Scholar
Cross Ref
- [11] . 2011. Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization. IEEE Trans. Image Process. 20, 7 (2011), 1838–1857. Google Scholar
Digital Library
- [12] . 2002. Example-based super-resolution. IEEE Comput. Graph. Appl. 22, 2 (2002), 56–65.Google Scholar
Digital Library
- [13] . 1999. Learning low-level vision. In Proceedings of the International Conference on Computer Vision. IEEE Computer Society, 1182–1189. Google Scholar
Cross Ref
- [14] . 2016. Manga109 dataset and creation of metadata. In Proceedings of the 1st International Workshop on coMics ANalysis, Processing and Understanding ([email protected]’16), , , , , , and (Eds.). ACM, 2:1–2:5. Google Scholar
Digital Library
- [15] . 2009. Super-resolution from a single image. In Proceedings of the IEEE 12th International Conference on Computer Vision. IEEE, 349–356.Google Scholar
Cross Ref
- [16] . 2018. Deep back-projection networks for super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1664–1673.Google Scholar
Cross Ref
- [17] . 2016. Identity mappings in deep residual networks. In Proceedings of the European Conference on Computer Vision. Springer, 630–645.Google Scholar
Cross Ref
- [18] . 2015. single-image super-resolution from transformed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5197–5206.Google Scholar
Cross Ref
- [19] . 2018. Fast and accurate single-image super-resolution via information distillation network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 723–731.Google Scholar
Cross Ref
- [20] . 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning(
Proceedings of Machine Learning Research , Vol. 37), and (Eds.). PMLR, Lille, France, 448–456. Retrieved from http://proceedings.mlr.press/v37/ioffe15.html.Google Scholar - [21] . 1991. Improving resolution by image registration. CVGIP: Graph. Model Image Process. 53 (1991), 231–239.Google Scholar
Digital Library
- [22] . 2016. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of the European Conference on Computer Vision. Springer, 694–711.Google Scholar
Cross Ref
- [23] . 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1646–1654.Google Scholar
Cross Ref
- [24] . 2016. Deeply recursive convolutional network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). IEEE Computer Society, 1637–1645. Google Scholar
Cross Ref
- [25] . 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR’15), and (Eds.). Retrieved from http://arxiv.org/abs/1412.6980Google Scholar
- [26] . 2017. Deep Laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 624–632.Google Scholar
Cross Ref
- [27] . 2018. Fast and accurate image super-resolution with deep Laplacian pyramid networks. IEEE Trans. Pattern Anal. Mach. Intell. 41, 11 (2018), 2599–2613.Google Scholar
Cross Ref
- [28] . 2017. Photo-realistic single-image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4681–4690.Google Scholar
Cross Ref
- [29] . 2018. Multi-scale residual network for image super-resolution. In Proceedings of the European Conference on Computer Vision (ECCV’18). 517–532.Google Scholar
Digital Library
- [30] . 2001. New edge-directed interpolation. IEEE Trans. Image Process. 10, 10 (2001), 1521–1527. Google Scholar
Digital Library
- [31] . 2021. Lightweight single-image super-resolution with dense connection distillation network. ACM Trans. Multimedia Comput., Commun. Appl. 17, 1s (2021), 1–17.Google Scholar
Digital Library
- [32] . 2016. Single-image super resolution for multispectral remote sensing data using convolutional neural networks. ISPRS-Int. Arch. Photogram., Remote Sens. Spatial Info. Sci. 41 (2016), 883–890.Google Scholar
Cross Ref
- [33] . 2017. Enhanced deep residual networks for single-image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17).Google Scholar
Cross Ref
- [34] . 2017. Enhanced deep residual networks for single-image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 136–144.Google Scholar
Cross Ref
- [35] . 2018. Non-local recurrent network for image restoration. In Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NeurIPS’18), , , , , , and (Eds.). 1680–1689. Retrieved from https://proceedings.neurips.cc/paper/2018/hash/fc49306d97602c8ed1be1dfbf0835ead-Abstract.html.Google Scholar
- [36] . 2016. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In Advances in Neural Information Processing Systems. 2802–2810.Google Scholar
Digital Library
- [37] . 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the 8th International Conference On Computer Vision (ICCV’01). IEEE Computer Society, 416–425. Google Scholar
Cross Ref
- [38] . 2021. Image super-resolution with non-local sparse attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3517–3526.Google Scholar
Cross Ref
- [39] . 2020. Single-image super-resolution via a holistic attention network. In Proceedings of the European Conference on Computer Vision. Springer, 191–207.Google Scholar
- [40] . 2019. Embedded block residual network: A recursive restoration model for single-image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV’19). IEEE, 4179–4188. Google Scholar
Cross Ref
- [41] . 2017. EnhanceNet: Single-image super-resolution through automated texture synthesis. In Proceedings of the IEEE International Conference on Computer Vision.Google Scholar
Cross Ref
- [42] . 2016. Real-time single-image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1874–1883.Google Scholar
Cross Ref
- [43] . 2013. Cardiac image super-resolution with global correspondence using multi-atlas patchmatch. In Proceedings of the 16th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI’13)(
Lecture Notes in Computer Science , Vol. 8151), , , , , and (Eds.). Springer, 9–16. Google ScholarCross Ref
- [44] . 2008. Image super-resolution using gradient profile prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1–8.Google Scholar
- [45] . 2017. Image super-resolution via deep recursive residual network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3147–3155.Google Scholar
Cross Ref
- [46] . 2017. Memnet: A persistent memory network for image restoration. In Proceedings of the IEEE International Conference on Computer Vision. 4539–4547.Google Scholar
Cross Ref
- [47] . 2010. Super resolution using edge prior and single-image detail synthesis. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 2400–2407.Google Scholar
Cross Ref
- [48] . 2014. A+: Adjusted anchored neighborhood regression for fast super-resolution. In Proceedings of the Asian Conference on Computer Vision. Springer, 111–126.Google Scholar
- [49] . 2017. Image super-resolution using dense skip connections. In Proceedings of the IEEE International Conference on Computer Vision. 4799–4807.Google Scholar
Cross Ref
- [50] . 2015. Deep networks for image super-resolution with sparse prior. In Proceedings of the IEEE International Conference on Computer Vision. 370–378.Google Scholar
Digital Library
- [51] . 2016. Deep sparse rectifier neural networks for speech denoising. In Proceedings of the IEEE International Workshop on Acoustic Signal Enhancement (IWAENC’16). IEEE, 1–5. Google Scholar
Cross Ref
- [52] . 2019. Super resolution using dual path connections. In Proceedings of the 27th ACM International Conference on Multimedia (MM’19). Association for Computing Machinery, New York, NY, 1552–1560. Google Scholar
Digital Library
- [53] . 2013. Fast direct super-resolution by simple functions. In Proceedings of the IEEE International Conference on Computer Vision. 561–568.Google Scholar
Digital Library
- [54] . 2010. Image super-resolution via sparse representation. IEEE Trans. Image Process. 19, 11 (2010), 2861–2873.Google Scholar
Digital Library
- [55] . 2010. On single-image scale-up using sparse-representations. In Proceedings of the International Conference on Curves and Surfaces. Springer, 711–730.Google Scholar
Digital Library
- [56] . 2020. Kernel attention network for single-image super-resolution. ACM Trans. Multimedia Comput., Commun., Appl. 16, 3 (2020), 1–15.Google Scholar
Digital Library
- [57] . 2012. Single-image super-resolution with non-local means and steering kernel regression. IEEE Trans. Image Process. 21 (2012), 4544–4556.Google Scholar
Digital Library
- [58] . 2018. Learning a single convolutional super-resolution network for multiple degradations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). Computer Vision Foundation/IEEE Computer Society, 3262–3271. Google Scholar
Cross Ref
- [59] . 2006. An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans. Image Process. 15 (2006), 2226–2238.Google Scholar
Digital Library
- [60] . 2018. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (ECCV’18). 286–301.Google Scholar
Digital Library
- [61] . 2019. Residual non-local attention networks for image restoration. Retrieved from https://arXiv:1903.10082.Google Scholar
- [62] . 2018. Residual dense network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2472–2481.Google Scholar
Cross Ref
- [63] . 2020. Residual dense network for image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 43, 7 (2020), 2480–2495.Google Scholar
Cross Ref
- [64] . 2010. Very low resolution face recognition problem. In Proceedings of the 4th IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS’10). 1–6.Google Scholar
Index Terms
Efficient Single-image Super-resolution Using Dual path Connections with Multiple scale Learning
Recommendations
Single image super-resolution via self-similarity and low-rank matrix recovery
We propose a novel single-image super resolution (SISR) approach using self-similarity of image and the low-rank matrix recovery (LRMR). The method performs multiple upsampling steps with relatively small magnification factors to recover a desired high ...
Single Image Super-Resolution Based on Local Self-Similarity
ACPR '13: Proceedings of the 2013 2nd IAPR Asian Conference on Pattern RecognitionSingle image super-resolution, namely increasing the resolution from only one coarse-resolution image, is a fundamental problem in computer vision. Although it has been extensively studied for decades, super-resolving a real-world image still remains ...
Texture enhancement for improving single-image super-resolution performance
Although various single-image super-resolution algorithms have been developed to increase image resolution, they still do not provide adequate performance in the texture region due to the lack of fine textures in the processed image. In this paper, we ...






Comments