Abstract
Multi-scale design has been considered in recent image super-resolution (SR) works to explore the hierarchical feature information. Existing multi-scale networks aim at building elaborate blocks or progressive architecture for restoration. In general, larger scale features concentrate more on structural and high-level information, while smaller scale features contain plentiful details and textured information. In this point of view, information from larger scale features can be derived from smaller ones. Based on the observation, in this article, we build a sequential hierarchical learning super-resolution network (SHSR) for effective image SR. Specially, we consider the inter-scale correlations of features, and devise a sequential multi-scale block (SMB) to progressively explore the hierarchical information. SMB is designed in a recursive way based on the linearity of convolution with restricted parameters. Besides the sequential hierarchical learning, we also investigate the correlations among the feature maps and devise a distribution transformation block (DTB). Different from attention-based methods, DTB regards the transformation in a normalization manner, and jointly considers the spatial and channel-wise correlations with scaling and bias factors. Experiment results show SHSR achieves superior quantitative performance and visual quality to state-of-the-art methods with near 34% parameters and 50% MACs off when scaling factor is × 4. To boost the performance without further training, the extension model SHSR+ with self-ensemble achieves competitive performance than larger networks with near 92% parameters and 42% MACs off with scaling factor ×4.
- [1] . 2018. Full 3D reconstruction of non-rigidly deforming objects. ACM Transactions on Multimedia Computing, Communications, and Applications 14, 1s, Article
24 (2018), 23 pages.Google ScholarDigital Library
- [2] . 2017. NTIRE 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1122–1131.
DOI: DOI: Google ScholarCross Ref
- [3] . 2018. Fast, accurate, and lightweight super-resolution with cascading residual network. In Proceedings of the European Conference on Computer Vision. 256–272.Google Scholar
Digital Library
- [4] . 2020. Enforcing affinity feature learning through self-attention for person re-identification. ACM Transactions on Multimedia Computing, Communications, and Applications 16, 1, Article
16 (2020), 22 pages.Google ScholarDigital Library
- [5] . 2020. Densely residual laplacian super-resolution. In Proceedings of the IEEE Transactions on Pattern Analysis and Machine Intelligence. 1–1.Google Scholar
- [6] . 2020. Hierarchical residual attention network for single image super-resolution. (2020). arXiv:eess.IV/2012.04578. Retrieved from https://arxiv.org/abs/2012.04578.Google Scholar
- [7] . 2012. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In British Machine Vision Conference. 135.1–135.10.Google Scholar
- [8] . 2021. Learning spatial attention for face super-resolution. IEEE Transactions on Image Processing 30 (2021), 1219–1231.Google Scholar
Digital Library
- [9] . 2017. A deep convolutional neural network with selection units for super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1150–1156.Google Scholar
Cross Ref
- [10] . 2019. Second-order attention network for single image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11057–11066.Google Scholar
Cross Ref
- [11] . 2016. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 38, 2 (2016), 295–307.Google Scholar
Digital Library
- [12] . 2016. Accelerating the super-resolution convolutional neural network. In Proceedings of the European Conference on Computer Vision. 391–407.Google Scholar
Cross Ref
- [13] . 2020. Remote sensing image super-resolution using second-order multi-scale networks. IEEE Transactions on Geoscience and Remote Sensing (TGRS’20), 59, 4 (2020), 3473–3485.Google Scholar
- [14] . 2017. Balanced two-stage residual networks for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1157–1164.Google Scholar
Cross Ref
- [15] . 2020. Closed-loop matters: Dual regression networks for single image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5406–5415.Google Scholar
Cross Ref
- [16] . 2020. Deep back-projection networks for single image super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI’20), 43, 2 (2020), 4323–4337.Google Scholar
- [17] . 2019. ODE-inspired network design for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1732–1741.Google Scholar
Cross Ref
- [18] . 2020. Squeeze-and-excitation networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 8 (2020), 2011–2023.Google Scholar
Digital Library
- [19] . 2021. Single image super-resolution with multi-scale information cross-fusion network. Signal Process 179 (2021), 107831.Google Scholar
Cross Ref
- [20] . 2020. Channel-wise and spatial feature modulation network for single image super-resolution. IEEE Transactions on Circuits and Systems for Video Technology 30, 11 (2020), 3911–3927.Google Scholar
Cross Ref
- [21] . 2015. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5197–5206.Google Scholar
Cross Ref
- [22] . 2019. Lightweight image super-resolution with information multi-distillation network. In Proceedings of the ACM International Conference on Multimedia. 2024–2032.Google Scholar
Digital Library
- [23] . 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the International Conference on Machine Learning (ICML) (JMLR Workshop and Conference Proceedings). 448–456.Google Scholar
- [24] . 2020. Hierarchical dense recursive network for image super-resolution. Pattern Recognition 107 (2020), 107475.Google Scholar
Cross Ref
- [25] . 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1646–1654.Google Scholar
Cross Ref
- [26] . 2016. Deeply-recursive convolutional network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1637–1645.Google Scholar
Cross Ref
- [27] . 2014. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015.Google Scholar
- [28] . 2017. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5835–5843.Google Scholar
Cross Ref
- [29] . 2019. Fast and accurate image super-resolution with deep laplacian pyramid networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 41, 11 (2019), 2599–2613.Google Scholar
Cross Ref
- [30] . 2020. MADNet: A fast and lightweight network for single-image super resolution. IEEE Transactions on Cybernetics 51, 3 (2020), 1443–1453.Google Scholar
- [31] . 2020. s-LWSR: Super lightweight super-resolution network. IEEE Transactions on Image Processing 29 (2020), 8368–8380.Google Scholar
Cross Ref
- [32] . 2020. MDCN: Multi-scale dense cross network for image super-resolution. IEEE Transactions on Circuits and Systems for Video Technology 31, 7 (2020), 2547–2561.Google Scholar
- [33] . 2018. Multi-scale residual network for image super-resolution. In Proceedings of the European Conference on Computer Vision. 527–542.Google Scholar
Digital Library
- [34] . 2020. A deep learning approach for face hallucination guided by facial boundary responses. ACM Transactions on Multimedia Computing, Communications, and Applications 16, 1, Article
17 (2020), 23 pages.Google ScholarDigital Library
- [35] . 2019. Feedback network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3862–3871.Google Scholar
Cross Ref
- [36] . 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1132–1140.Google Scholar
Cross Ref
- [37] . 2020. A fast and accurate super-resolution network using progressive residual learning. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. 1818–1822.Google Scholar
Cross Ref
- [38] . 2020. Residual feature aggregation network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2356–2365.Google Scholar
Cross Ref
- [39] . 2014. No-reference image quality assessment based on spatial and spectral entropies. Signal Processing: Image Communication 29, 8 (2014), 856–863.Google Scholar
Cross Ref
- [40] . 2020. Unfolding the alternating optimization for blind super resolution. In Proceedings of the 34th International Conference on Neural Information Processing Systems.Google Scholar
- [41] . 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the IEEE International Conference on Computer Vision.416–423.Google Scholar
Cross Ref
- [42] . 2017. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications 76, 20 (2017), 21811–21838.Google Scholar
Digital Library
- [43] . 2013. Making a “Completely Blind” image quality analyzer. IEEE Signal Process. Letters 20, 3 (2013), 209–212.Google Scholar
Cross Ref
- [44] . 2019. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of the Neural Information Processing Systems. 8024–8035.Google Scholar
- [45] . 2019. Embedded block residual network: A recursive restoration model for single-image super-resolution. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. 4179–4188.
DOI: DOI: Google ScholarCross Ref
- [46] . 2017. Image super resolution based on fusing multiple convolution neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1050–1057.Google Scholar
Cross Ref
- [47] . 2016. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Transactions on Medical Imaging 35, 5 (2016), 1285–1298.Google Scholar
Cross Ref
- [48] . 2015. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1–9.Google Scholar
Cross Ref
- [49] . 2017. Image super-resolution via deep recursive residual network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2790–2798.Google Scholar
Cross Ref
- [50] . 2017. MemNet: A persistent memory network for image restoration. In Proceedings of the IEEE International Conference on Computer Vision. 4549–4557.Google Scholar
Cross Ref
- [51] . 2020. Deep inception-residual laplacian pyramid networks for accurate single-image super-resolution. IEEE Transactions on Neural Networks and Learning Systems 31, 5 (2020), 1514–1528.Google Scholar
Cross Ref
- [52] . 2020. Lightweight image super-resolution by multi-scale aggregation. IEEE Transactions on Broadcasting 67, 2 (2020), 375–382.Google Scholar
- [53] . 2018. ESRGAN: Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision Workshop. 63–79.Google Scholar
- [54] . 2021. Deep learning for image super-resolution: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 10 (2021), 3365–3387.Google Scholar
Cross Ref
- [55] . 2020. Multi-grained attention networks for single image super-resolution. IEEE Transactions on Circuits and Systems for Video Technology 31, 2 (2020), 512–522.Google Scholar
- [56] . 2017. Sketch-a-Net: A deep neural network that beats humans. International Journal on Computer Vision 122, 3 (2017), 411–425.Google Scholar
Digital Library
- [57] . 2014. Visualizing and understanding convolutional networks. In Proceedings of the European Conference on Computer Vision. 818–833.Google Scholar
Cross Ref
- [58] . 2011. Adaptive deconvolutional networks for mid and high level feature learning. In Proceedings of the IEEE International Conference on Computer Vision. 2018–2025.Google Scholar
Digital Library
- [59] . 2010. On single image scale-up using sparse-representations. In Proceedings of the International Conference on Curves and Surfaces. Springer, 711–730.Google Scholar
Digital Library
- [60] . 2020. Kernel attention network for single image super-resolution. ACM Transactions on Multimedia Computing, Communications, and Applications 16, 3 (2020), 90:1–90:15.Google Scholar
Digital Library
- [61] . 2020. Multi-scale image super-resolution via a single extendable deep network. IEEE Journal of Selected Topics in Signal Processing 15, 2 (2020), 253–263.Google Scholar
- [62] . 2020. Deep unfolding network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3214–3223.Google Scholar
Cross Ref
- [63] . 2017. Learning deep CNN denoiser prior for image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2808–2817.Google Scholar
Cross Ref
- [64] . 2019. Deep plug-and-play super-resolution for arbitrary blur kernels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1671–1681.Google Scholar
Cross Ref
- [65] . 2018. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision. 294–310.Google Scholar
Digital Library
- [66] . 2020. Residual dense network for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 7 (2020), 2480–2495.Google Scholar
Index Terms
Sequential Hierarchical Learning with Distribution Transformation for Image Super-Resolution
Recommendations
Adaptive large scale artifact reduction in edge-based image super-resolution
SIP '07: Proceedings of the Ninth IASTED International Conference on Signal and Image ProcessingThe goal of multi-frame image super-resolution is to use information from low-resolution images to construct highresolution images. Existing construction techniques are highly sensitive to prominent large scale artifacts in the low-resolution images. ...
Image super-resolution by estimating the enhancement weight of self example and external missing patches
Image super-resolution (SR) is the process of generating a high-resolution (HR) image using one or more low-resolution (LR) inputs. Many SR methods have been proposed, but generating the small-scale structure of an SR image remains a challenging task. ...
Image super-resolution via 2D tensor regression learning
Presenting a novel framework on 2D tensor regression learning model.Proposing three types of regularization to assist the model learning.Implementing the optimization algorithms for each specific model.Conducting comprehensive experiments on testing ...






Comments