Abstract
There are two main categories of image super-resolution algorithms: distortion oriented and perception oriented. Recent evidence shows that reconstruction accuracy and perceptual quality are typically in disagreement with each other. In this article, we present a new image super-resolution framework that is capable of striking a balance between distortion and perception. The core of our framework is a deep fusion network capable of generating a final high-resolution image by fusing a pair of deterministic and stochastic images using spatially varying weights. To make a single fusion model produce images with varying degrees of stochasticity, we further incorporate meta-learning into our fusion network. Once equipped with the kernel produced by a kernel prediction module, our meta fusion network is able to produce final images at any desired level of stochasticity. Experimental results indicate that our meta fusion network outperforms existing state-of-the-art SISR algorithms on widely used datasets, including PIRM-val, DIV2K-val, Set5, Set14, Urban100, Manga109, and B100. In addition, it is capable of producing high-resolution images that achieve low distortion and high perceptual quality simultaneously.
- [1] . 2017. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 126–135.Google Scholar
Cross Ref
- [2] . 2016. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems. 3981–3989. Google Scholar
Digital Library
- [3] . 2018. Finding tiny faces in the wild with generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 21–30.Google Scholar
Cross Ref
- [4] . 2012. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In Proceedings British Machine Vision Conference. 135.1–135.10.Google Scholar
- [5] . 2018. The 2018 PIRM challenge on perceptual image super-resolution. In Proceedings of the European Conference on Computer Vision (ECCV’18). 0–0.Google Scholar
- [6] . 2018. The perception-distortion tradeoff. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6228–6237.Google Scholar
Cross Ref
- [7] . 2004. Super-resolution through neighbor embedding. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’04), Vol. 1. IEEE, I–I.Google Scholar
Cross Ref
- [8] . 2020. Deep learning-based image super-resolution considering quantitative and perceptual quality. Neurocomputing 398 (2020), 347–359.Google Scholar
Cross Ref
- [9] . 2015. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38, 2 (2015), 295–307. Google Scholar
Digital Library
- [10] . 2006. New full-reference quality metrics based on HVS. In Proceedings of the 2nd International Workshop on Video Processing and Quality Metrics, Vol. 4.Google Scholar
- [11] . 2002. Example-based super-resolution. IEEE Comput. Graph. Appl. 22, 2 (2002), 56–65. Google Scholar
Digital Library
- [12] . 2018. Image super-resolution via deterministic-stochastic synthesis and local statistical rectification. ACM Trans. Graph. 37, 6 (2018), 1–14.Google Scholar
Digital Library
- [13] . 2019. Blind super-resolution with iterative kernel correction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1604–1613.Google Scholar
Cross Ref
- [14] . 2019. Recurrent back-projection network for video super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3897–3906.Google Scholar
Cross Ref
- [15] . 2019. Meta-SR: A magnification-arbitrary network for super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1575–1584.Google Scholar
Cross Ref
- [16] . 2015. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5197–5206.Google Scholar
Cross Ref
- [17] . 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4681–4690.Google Scholar
Cross Ref
- [18] . 2019. Fast spatio-temporal residual network for video super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10522–10531.Google Scholar
Cross Ref
- [19] . 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 136–144.Google Scholar
Cross Ref
- [20] . 2020. Residual feature aggregation network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2359–2368.Google Scholar
Cross Ref
- [21] . 2020. SRFlow: Learning the super-resolution space with normalizing flow. In Proceedings of the European Conference on Computer Vision (ECCV’20).Google Scholar
Digital Library
- [22] . 2016. Convolutional oriented boundaries. In Proceedings of the European Conference on Computer Vision. Springer, 580–596.Google Scholar
Cross Ref
- [23] . 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the 8th IEEE International Conference on Computer Vision (ICCV’01), Vol. 2. IEEE, 416–423.Google Scholar
Cross Ref
- [24] . 2017. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools Appl. 76, 20 (2017), 21811–21838. Google Scholar
Digital Library
- [25] . 2012. Making a “completely blind” image quality analyzer. IEEE Sign. Process. Lett. 20, 3 (2012), 209–212.Google Scholar
Cross Ref
- [26] . 2018. Multi–scale recursive and perception–distortion controllable image super–resolution. In Proceedings of the European Conference on Computer Vision (ECCV’18). 0–0.Google Scholar
- [27] . 2012. A parameterless line segment and elliptical arc detector with enhanced ellipse fitting. In Proceedings of the European Conference on Computer Vision. Springer, 572–585. Google Scholar
Digital Library
- [28] . 2007. On between-coefficient contrast masking of DCT basis functions. In Proceedings of the 3rd International Workshop on Video Processing and Quality Metrics, Vol. 4.Google Scholar
- [29] . 2016. Optimization as a model for few-shot learning. In International Conference on Learning Representations.Google Scholar
- [30] . 2019. Natural and realistic single image super-resolution with explicit natural manifold discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8122–8131.Google Scholar
Cross Ref
- [31] . 2008. Image super-resolution using gradient profile prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1–8.Google Scholar
- [32] . 2010. Super resolution using edge prior and single image detail synthesis. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 2400–2407.Google Scholar
Cross Ref
- [33] . 2017. Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 114–125.Google Scholar
Cross Ref
- [34] . 2013. Anchored neighborhood regression for fast example-based super-resolution. In Proceedings of the IEEE International Conference on Computer Vision. 1920–1927. Google Scholar
Digital Library
- [35] . 2014. A+: Adjusted anchored neighborhood regression for fast super-resolution. In Proceedings of the Asian Conference on Computer Vision. Springer, 111–126.Google Scholar
- [36] . 2018. Analyzing perception-distortion tradeoff using enhanced perceptual super-resolution network. In Proceedings of the European Conference on Computer Vision (ECCV’18). 0–0.Google Scholar
- [37] . 2021. Exploring sparsity in image super-resolution for efficient inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4917–4926.Google Scholar
Cross Ref
- [38] . 2021. Unsupervised degradation representation learning for blind super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10581–10590.Google Scholar
Cross Ref
- [39] . 2021. Learning a single network for scale-arbitrary super-resolution. In Proceedings of the IEEE Conference on International Conference on Computer Vision. 10581–10590.Google Scholar
Cross Ref
- [40] . 2018. Recovering realistic texture in image super-resolution by deep spatial feature transform. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 606–615.Google Scholar
Cross Ref
- [41] . 2019. Deep network interpolation for continuous imagery effect transition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1692–1701.Google Scholar
Cross Ref
- [42] . 2018. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV’18).Google Scholar
- [43] . 2016. Learning to learn: Model regression networks for easy small sample learning. In Proceedings of the European Conference on Computer Vision. Springer, 616–634.Google Scholar
Cross Ref
- [44] . 2015. Holistically-nested edge detection. In Proceedings of the IEEE International Conference on Computer Vision. 1395–1403. Google Scholar
Digital Library
- [45] . 2019. Scan: Spatial color attention networks for real single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 0–0.Google Scholar
Cross Ref
- [46] . 2008. Image super-resolution as sparse representation of raw image patches. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1–8.Google Scholar
- [47] . 2012. A novel image fusion method using IKONOS satellite images. J. Geodesy Geoinf. 1, 1 (2012), 75–83.Google Scholar
Cross Ref
- [48] . 2010. On single image scale-up using sparse-representations. In Proceedings of the International Conference on Curves and Surfaces. Springer, 711–730. Google Scholar
Digital Library
- [49] . 2020. Deep unfolding network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3217–3226.Google Scholar
Cross Ref
- [50] . 2018. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (ECCV’18). 286–301.Google Scholar
Digital Library
- [51] . 2019. Image super-resolution by neural texture transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7982–7991.Google Scholar
Cross Ref
Index Terms
Structure-aware Meta-fusion for Image Super-resolution
Recommendations
Singular value decomposition based fusion for super-resolution image reconstruction
In this paper, we address a super-resolution problem of generating a high-resolution image from low-resolution images. The proposed super-resolution method consists of three steps: image registration, singular value decomposition (SVD)-based image ...
Simultaneous image fusion and super-resolution using sparse representation
Given multiple source images of the same scene, image fusion integrates the inherent complementary information into one single image, and thus provides a more complete and accurate description. However, when the source images are of low-resolution, the ...
Regional Varying Image Super-Resolution
CSO '09: Proceedings of the 2009 International Joint Conference on Computational Sciences and Optimization - Volume 01In this paper a new method for single image Super-Resolution using some high resolution images is proposed. It is assumed that each high resolution image shows a region of the low resolution image, with some differences about illumination or view point. ...






Comments