skip to main content
research-article

Sequential Hierarchical Learning with Distribution Transformation for Image Super-Resolution

Authors Info & Claims
Published:23 January 2023Publication History
Skip Abstract Section

Abstract

Multi-scale design has been considered in recent image super-resolution (SR) works to explore the hierarchical feature information. Existing multi-scale networks aim at building elaborate blocks or progressive architecture for restoration. In general, larger scale features concentrate more on structural and high-level information, while smaller scale features contain plentiful details and textured information. In this point of view, information from larger scale features can be derived from smaller ones. Based on the observation, in this article, we build a sequential hierarchical learning super-resolution network (SHSR) for effective image SR. Specially, we consider the inter-scale correlations of features, and devise a sequential multi-scale block (SMB) to progressively explore the hierarchical information. SMB is designed in a recursive way based on the linearity of convolution with restricted parameters. Besides the sequential hierarchical learning, we also investigate the correlations among the feature maps and devise a distribution transformation block (DTB). Different from attention-based methods, DTB regards the transformation in a normalization manner, and jointly considers the spatial and channel-wise correlations with scaling and bias factors. Experiment results show SHSR achieves superior quantitative performance and visual quality to state-of-the-art methods with near 34% parameters and 50% MACs off when scaling factor is × 4. To boost the performance without further training, the extension model SHSR+ with self-ensemble achieves competitive performance than larger networks with near 92% parameters and 42% MACs off with scaling factor ×4.

REFERENCES

  1. [1] Afzal Hassan, Aouada Djamila, Mirbach Bruno, and Ottersten Björn. 2018. Full 3D reconstruction of non-rigidly deforming objects. ACM Transactions on Multimedia Computing, Communications, and Applications 14, 1s, Article 24 (2018), 23 pages.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. [2] Agustsson E. and Timofte R.. 2017. NTIRE 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 11221131. DOI:DOI:Google ScholarGoogle ScholarCross RefCross Ref
  3. [3] Ahn Namhyuk, Kang Byungkon, and Sohn Kyung-Ah. 2018. Fast, accurate, and lightweight super-resolution with cascading residual network. In Proceedings of the European Conference on Computer Vision. 256272.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. [4] Ainam Jean-Paul, Qin Ke, Liu Guisong, Luo Guangchun, and Agyemang Brighter. 2020. Enforcing affinity feature learning through self-attention for person re-identification. ACM Transactions on Multimedia Computing, Communications, and Applications 16, 1, Article 16 (2020), 22 pages.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. [5] Anwar S. and Barnes N.. 2020. Densely residual laplacian super-resolution. In Proceedings of the IEEE Transactions on Pattern Analysis and Machine Intelligence. 11.Google ScholarGoogle Scholar
  6. [6] Behjati Parichehr, Rodriguez Pau, Mehri Armin, Hupont Isabelle, Tena Carles Fernández, and Gonzalez Jordi. 2020. Hierarchical residual attention network for single image super-resolution. (2020). arXiv:eess.IV/2012.04578. Retrieved from https://arxiv.org/abs/2012.04578.Google ScholarGoogle Scholar
  7. [7] Bevilacqua Marco, Roumy Aline, Guillemot Christine, and Morel Marie line Alberi. 2012. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In British Machine Vision Conference. 135.1–135.10.Google ScholarGoogle Scholar
  8. [8] Chen C., Gong D., Wang H., Li Z., and Wong K. Y. K.. 2021. Learning spatial attention for face super-resolution. IEEE Transactions on Image Processing 30 (2021), 12191231.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Choi J. and Kim M.. 2017. A deep convolutional neural network with selection units for super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 11501156.Google ScholarGoogle ScholarCross RefCross Ref
  10. [10] Dai T., Cai J., Zhang Y., Xia S., and Zhang L.. 2019. Second-order attention network for single image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1105711066.Google ScholarGoogle ScholarCross RefCross Ref
  11. [11] Dong C., Loy C. C., He K., and Tang X.. 2016. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 38, 2 (2016), 295307.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. [12] Dong Chao, Loy Chen Change, and Tang Xiaoou. 2016. Accelerating the super-resolution convolutional neural network. In Proceedings of the European Conference on Computer Vision. 391407.Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Dong X., Wang L., Sun X., Jia X., Gao L., and Zhang B.. 2020. Remote sensing image super-resolution using second-order multi-scale networks. IEEE Transactions on Geoscience and Remote Sensing (TGRS’20), 59, 4 (2020), 3473–3485.Google ScholarGoogle Scholar
  14. [14] Fan Y., Shi H., Yu J., Liu D., Han W., Yu H., Wang Z., Wang X., and Huang T. S.. 2017. Balanced two-stage residual networks for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 11571164.Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Guo Y., Chen J., Wang J., Chen Q., Cao J., Deng Z., Xu Y., and Tan M.. 2020. Closed-loop matters: Dual regression networks for single image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 54065415.Google ScholarGoogle ScholarCross RefCross Ref
  16. [16] Haris M., Shakhnarovich G., and Ukita N.. 2020. Deep back-projection networks for single image super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI’20), 43, 2 (2020), 4323–4337.Google ScholarGoogle Scholar
  17. [17] He Xiangyu, Mo Zitao, Wang Peisong, Liu Yang, Yang Mingyuan, and Cheng Jian. 2019. ODE-inspired network design for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 17321741.Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Hu J., Shen L., Albanie S., Sun G., and Wu E.. 2020. Squeeze-and-excitation networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 8 (2020), 20112023.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. [19] Hu Yanting, Gao Xinbo, Li Jie, Huang Yuanfei, and Wang Hanzi. 2021. Single image super-resolution with multi-scale information cross-fusion network. Signal Process 179 (2021), 107831.Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Hu Y., Li J., Huang Y., and Gao X.. 2020. Channel-wise and spatial feature modulation network for single image super-resolution. IEEE Transactions on Circuits and Systems for Video Technology 30, 11 (2020), 39113927.Google ScholarGoogle ScholarCross RefCross Ref
  21. [21] Huang J., Singh A., and Ahuja N.. 2015. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 51975206.Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Hui Zheng, Gao Xinbo, Yang Yunchu, and Wang Xiumei. 2019. Lightweight image super-resolution with information multi-distillation network. In Proceedings of the ACM International Conference on Multimedia. 20242032.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. [23] Ioffe Sergey and Szegedy Christian. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the International Conference on Machine Learning (ICML) (JMLR Workshop and Conference Proceedings). 448456.Google ScholarGoogle Scholar
  24. [24] Jiang Kui, Wang Zhongyuan, Yi Peng, and Jiang Junjun. 2020. Hierarchical dense recursive network for image super-resolution. Pattern Recognition 107 (2020), 107475.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Kim Jiwon, Lee Jung Kwon, and Lee Kyoung Mu. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 16461654.Google ScholarGoogle ScholarCross RefCross Ref
  26. [26] Kim Jiwon, Lee Jung Kwon, and Lee Kyoung Mu. 2016. Deeply-recursive convolutional network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 16371645.Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Kingma Diederik P. and Ba Jimmy. 2014. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015.Google ScholarGoogle Scholar
  28. [28] Lai W., Huang J., Ahuja N., and Yang M.. 2017. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 58355843.Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Lai W., Huang J., Ahuja N., and Yang M.. 2019. Fast and accurate image super-resolution with deep laplacian pyramid networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 41, 11 (2019), 25992613.Google ScholarGoogle ScholarCross RefCross Ref
  30. [30] Lan R., Sun L., Liu Z., Lu H., Pang C., and Luo X.. 2020. MADNet: A fast and lightweight network for single-image super resolution. IEEE Transactions on Cybernetics 51, 3 (2020), 14431453.Google ScholarGoogle Scholar
  31. [31] Li B., Wang B., Liu J., Qi Z., and Shi Y.. 2020. s-LWSR: Super lightweight super-resolution network. IEEE Transactions on Image Processing 29 (2020), 83688380.Google ScholarGoogle ScholarCross RefCross Ref
  32. [32] Li J., Fang F., Li J., Mei K., and Zhang G.. 2020. MDCN: Multi-scale dense cross network for image super-resolution. IEEE Transactions on Circuits and Systems for Video Technology 31, 7 (2020), 25472561.Google ScholarGoogle Scholar
  33. [33] Li Juncheng, Fang Faming, Mei Kangfu, and Zhang Guixu. 2018. Multi-scale residual network for image super-resolution. In Proceedings of the European Conference on Computer Vision. 527542.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. [34] Li Mengyan, Zhang Zhaoyu, Xie Guochen, and Yu Jun. 2020. A deep learning approach for face hallucination guided by facial boundary responses. ACM Transactions on Multimedia Computing, Communications, and Applications 16, 1, Article 17 (2020), 23 pages.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. [35] Li Z., Yang J., Liu Z., Yang X., Jeon G., and Wu W.. 2019. Feedback network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 38623871.Google ScholarGoogle ScholarCross RefCross Ref
  36. [36] Lim Bee, Son Sanghyun, Kim Heewon, Nah Seungjun, and Lee Kyoung Mu. 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 11321140.Google ScholarGoogle ScholarCross RefCross Ref
  37. [37] Liu H., Lu Z., Shi W., and Tu J.. 2020. A fast and accurate super-resolution network using progressive residual learning. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. 18181822.Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] Liu J., Zhang W., Tang Y., Tang J., and Wu G.. 2020. Residual feature aggregation network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 23562365.Google ScholarGoogle ScholarCross RefCross Ref
  39. [39] Liu Lixiong, Liu Bao, Huang Hua, and Bovik Alan Conrad. 2014. No-reference image quality assessment based on spatial and spectral entropies. Signal Processing: Image Communication 29, 8 (2014), 856863.Google ScholarGoogle ScholarCross RefCross Ref
  40. [40] Luo Zhengxiong, Huang Yan, Li Shang, Wang Liang, and Tan Tieniu. 2020. Unfolding the alternating optimization for blind super resolution. In Proceedings of the 34th International Conference on Neural Information Processing Systems.Google ScholarGoogle Scholar
  41. [41] Martin D., Fowlkes C., Tal D., and Malik J.. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the IEEE International Conference on Computer Vision.416423.Google ScholarGoogle ScholarCross RefCross Ref
  42. [42] Matsui Yusuke, Ito Kota, Aramaki Yuji, Fujimoto Azuma, Ogawa Toru, Yamasaki Toshihiko, and Aizawa Kiyoharu. 2017. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications 76, 20 (2017), 2181121838.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. [43] Mittal Anish, Soundararajan Rajiv, and Bovik Alan C.. 2013. Making a “Completely Blind” image quality analyzer. IEEE Signal Process. Letters 20, 3 (2013), 209212.Google ScholarGoogle ScholarCross RefCross Ref
  44. [44] Paszke Adam, Gross Sam, Massa Francisco, Lerer Adam, Bradbury James, Chanan Gregory, Killeen Trevor, Lin Zeming, Gimelshein Natalia, Antiga Luca, et al. 2019. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of the Neural Information Processing Systems. 80248035.Google ScholarGoogle Scholar
  45. [45] Qiu Yajun, Wang Ruxin, Tao Dapeng, and Cheng Jun. 2019. Embedded block residual network: A recursive restoration model for single-image super-resolution. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. 41794188. DOI:DOI:Google ScholarGoogle ScholarCross RefCross Ref
  46. [46] Ren H., El-Khamy M., and Lee J.. 2017. Image super resolution based on fusing multiple convolution neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 10501057.Google ScholarGoogle ScholarCross RefCross Ref
  47. [47] Shin Hoo-Chang, Roth Holger R., Gao Mingchen, Lu Le, Xu Ziyue, Nogues Isabella, Yao Jianhua, Mollura Daniel J., and Summers Ronald M.. 2016. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Transactions on Medical Imaging 35, 5 (2016), 12851298.Google ScholarGoogle ScholarCross RefCross Ref
  48. [48] Szegedy Christian, Liu Wei, Jia Yangqing, Sermanet Pierre, Reed Scott E., Anguelov Dragomir, Erhan Dumitru, Vanhoucke Vincent, and Rabinovich Andrew. 2015. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 19.Google ScholarGoogle ScholarCross RefCross Ref
  49. [49] Tai Y., Yang J., and Liu X.. 2017. Image super-resolution via deep recursive residual network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 27902798.Google ScholarGoogle ScholarCross RefCross Ref
  50. [50] Tai Ying, Yang Jian, Liu Xiaoming, and Xu Chunyan. 2017. MemNet: A persistent memory network for image restoration. In Proceedings of the IEEE International Conference on Computer Vision. 45494557.Google ScholarGoogle ScholarCross RefCross Ref
  51. [51] Tang Y., Gong W., Chen X., and Li W.. 2020. Deep inception-residual laplacian pyramid networks for accurate single-image super-resolution. IEEE Transactions on Neural Networks and Learning Systems 31, 5 (2020), 15141528.Google ScholarGoogle ScholarCross RefCross Ref
  52. [52] Wan J., Yin H., Liu Z., Chong A., and Liu Y.. 2020. Lightweight image super-resolution by multi-scale aggregation. IEEE Transactions on Broadcasting 67, 2 (2020), 375382.Google ScholarGoogle Scholar
  53. [53] Wang Xintao, Yu Ke, Wu Shixiang, Gu Jinjin, Liu Yihao, Dong Chao, Qiao Yu, and Loy Chen Change. 2018. ESRGAN: Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision Workshop. 6379.Google ScholarGoogle Scholar
  54. [54] Wang Zhihao, Chen Jian, and Hoi Steven C. H.. 2021. Deep learning for image super-resolution: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 10 (2021), 33653387.Google ScholarGoogle ScholarCross RefCross Ref
  55. [55] Wu H., Zou Z., Gui J., Zeng W., Ye J., Zhang J., Liu H., and Wei Z.. 2020. Multi-grained attention networks for single image super-resolution. IEEE Transactions on Circuits and Systems for Video Technology 31, 2 (2020), 512522.Google ScholarGoogle Scholar
  56. [56] Yu Qian, Yang Yongxin, Liu Feng, Song Yi-Zhe, Xiang Tao, and Hospedales Timothy M.. 2017. Sketch-a-Net: A deep neural network that beats humans. International Journal on Computer Vision 122, 3 (2017), 411425.Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. [57] Zeiler Matthew D. and Fergus Rob. 2014. Visualizing and understanding convolutional networks. In Proceedings of the European Conference on Computer Vision. 818833.Google ScholarGoogle ScholarCross RefCross Ref
  58. [58] Zeiler Matthew D., Taylor Graham W., and Fergus Rob. 2011. Adaptive deconvolutional networks for mid and high level feature learning. In Proceedings of the IEEE International Conference on Computer Vision. 20182025.Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. [59] Zeyde Roman, Elad Michael, and Protter Matan. 2010. On single image scale-up using sparse-representations. In Proceedings of the International Conference on Curves and Surfaces. Springer, 711730.Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. [60] Zhang Dongyang, Shao Jie, and Shen Heng Tao. 2020. Kernel attention network for single image super-resolution. ACM Transactions on Multimedia Computing, Communications, and Applications 16, 3 (2020), 90:1–90:15.Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. [61] Zhang H., Xiao J., and Jin Z.. 2020. Multi-scale image super-resolution via a single extendable deep network. IEEE Journal of Selected Topics in Signal Processing 15, 2 (2020), 253263.Google ScholarGoogle Scholar
  62. [62] Zhang Kai, Gool Luc Van, and Timofte Radu. 2020. Deep unfolding network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 32143223.Google ScholarGoogle ScholarCross RefCross Ref
  63. [63] Zhang Kai, Zuo Wangmeng, Gu Shuhang, and Zhang Lei. 2017. Learning deep CNN denoiser prior for image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 28082817.Google ScholarGoogle ScholarCross RefCross Ref
  64. [64] Zhang Kai, Zuo Wangmeng, and Zhang Lei. 2019. Deep plug-and-play super-resolution for arbitrary blur kernels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 16711681.Google ScholarGoogle ScholarCross RefCross Ref
  65. [65] Zhang Yulun, Li Kunpeng, Li Kai, Wang Lichen, Zhong Bineng, and Fu Yun. 2018. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision. 294310.Google ScholarGoogle ScholarDigital LibraryDigital Library
  66. [66] Zhang Y., Tian Y., Kong Y., Zhong B., and Fu Y.. 2020. Residual dense network for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 7 (2020), 24802495.Google ScholarGoogle Scholar

Index Terms

  1. Sequential Hierarchical Learning with Distribution Transformation for Image Super-Resolution

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Multimedia Computing, Communications, and Applications
        ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 1s
        February 2023
        504 pages
        ISSN:1551-6857
        EISSN:1551-6865
        DOI:10.1145/3572859
        • Editor:
        • Abdulmotaleb El Saddik
        Issue’s Table of Contents

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 23 January 2023
        • Online AM: 6 May 2022
        • Accepted: 19 April 2022
        • Revised: 17 October 2021
        • Received: 5 April 2021
        Published in tomm Volume 19, Issue 1s

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Refereed

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      View Full Text

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!