skip to main content
research-article

Lightweight Single Image Super-resolution with Dense Connection Distillation Network

Authors Info & Claims
Published:31 March 2021Publication History
Skip Abstract Section

Abstract

Single image super-resolution attempts to reconstruct a high-resolution (HR) image from its corresponding low-resolution (LR) image, which has been a research hotspot in computer vision and image processing for decades. To improve the accuracy of super-resolution images, many works adopt very deep networks to model the translation from LR to HR, resulting in memory and computation consumption. In this article, we design a lightweight dense connection distillation network by combining the feature fusion units and dense connection distillation blocks (DCDB) that include selective cascading and dense distillation components. The dense connections are used between and within the distillation block, which can provide rich information for image reconstruction by fusing shallow and deep features. In each DCDB, the dense distillation module concatenates the remaining feature maps of all previous layers to extract useful information, the selected features are then assessed by the proposed layer contrast-aware channel attention mechanism, and finally the cascade module aggregates the features. The distillation mechanism helps to reduce training parameters and improve training efficiency, and the layer contrast-aware channel attention further improves the performance of model. The quality and quantity experimental results on several benchmark datasets show the proposed method performs better tradeoff in term of accuracy and efficiency.

References

  1. Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn. 2018. Fast, accurate, and lightweight super-resolution with cascading residual network. In Proceedings of the European Conference on Computer Vision (ECCV’18).Google ScholarGoogle ScholarCross RefCross Ref
  2. Wong P. W. Allebach J. 1996. Edge-directed interpolation. In Proceedings of the International Conference on Image Processing.Google ScholarGoogle Scholar
  3. Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie-Line Alberi-Morel. 2012. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In Proceedings of the British Machine Vision Conference (BMVC’12).Google ScholarGoogle ScholarCross RefCross Ref
  4. C. E. Duchon. 1979. Lanczos filtering in one and two dimensions. J. Appl. Meteorol. 18, 8 (1979), 1016–1022.Google ScholarGoogle ScholarCross RefCross Ref
  5. Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang. 2019. Second-order attention network for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 11065–11074.Google ScholarGoogle ScholarCross RefCross Ref
  6. Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2016. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38, 2 (2016), 295–307. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Chao Dong, Chen Change Loy, and Xiaoou Tang. 2016. Accelerating the super-resolution convolutional neural network. In Proceedings of the European Conference on Computer Vision (ECCV’16).Google ScholarGoogle ScholarCross RefCross Ref
  8. Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. 2017. A learned representation for artistic style. CoRR abs/1610.07629 (2017).Google ScholarGoogle Scholar
  9. Dan Guo, Shuo Wang, Qi Tian, and Meng Wang. 2019. Dense temporal convolution network for sign language translation. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI’19). Google ScholarGoogle ScholarCross RefCross Ref
  10. Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. 2018. Deep back-projection networks for super-resolution. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1664–1673.Google ScholarGoogle ScholarCross RefCross Ref
  11. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16), 770–778.Google ScholarGoogle ScholarCross RefCross Ref
  12. J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu. 2019. Squeeze-and-excitation networks. IEEE Trans. Pattern Anal. Mach. Intell. 142, 8 (2019). Google ScholarGoogle Scholar
  13. Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17).Google ScholarGoogle Scholar
  14. Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. 2015. Single image super-resolution from transformed self-exemplars. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). 5197–5206.Google ScholarGoogle ScholarCross RefCross Ref
  15. Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang. 2019. Lightweight image super-resolution with information multi-distillation network. In Proceedings of the 27th ACM International Conference on Multimedia (ACM MM’19). 2024–2032. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Zheng Hui, Xiumei Wang, and Xinbo Gao. 2018. Fast and accurate single image super-resolution via information distillation network. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 723–731.Google ScholarGoogle ScholarCross RefCross Ref
  17. Jithin Saji Isaac and Ramesh Kulkarni. 2015. Super resolution techniques for medical image processing. In Proceedings of the International Conference on Technologies for Sustainable Development. 1–6.Google ScholarGoogle ScholarCross RefCross Ref
  18. L. Itti, C. Koch, and E. Niebur. 1998. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20, 11 (1998), 1254–1259. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16), 1646–1654.Google ScholarGoogle ScholarCross RefCross Ref
  20. Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Deeply-recursive convolutional network for image super-resolution. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16), 1637–1645.Google ScholarGoogle ScholarCross RefCross Ref
  21. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. CoRR abs/1412.6980 (2015).Google ScholarGoogle Scholar
  22. Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. 2017. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17), 5835–5843.Google ScholarGoogle ScholarCross RefCross Ref
  23. Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the The IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 105–114.Google ScholarGoogle ScholarCross RefCross Ref
  24. Zhang Lei and Wu Xiaolin. 2006. An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans. Image Process. 15, 8 (2006), 2226–2238. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW’17), 1132–1140.Google ScholarGoogle ScholarCross RefCross Ref
  26. B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee. 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW’17). 1132–1140.Google ScholarGoogle Scholar
  27. Xiaojiao Mao, Chunhua Shen, and Yu-Bin Yang. 2016. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In Advances in Neural Information Processing Systems 29, D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Eds.). 2802–2810. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. David R. Martin, Charless C. Fowlkes, Doron Tal, and Jitendra Malik. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the International Conference on Computer Vision (ICCV’01).Google ScholarGoogle ScholarCross RefCross Ref
  29. Yusuke Matsui, Kota Ito, Yuji Aramaki, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2016. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools Appl. 76 (2016), 21811–21838. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, and Alexander Ku. 2018. Image transformer. In Proceedings of the International Conference on Machine Learning (ICML).Google ScholarGoogle Scholar
  31. Adam Paszke, Sam Gross, and Adam Lerer. 2017. Automatic differentiation in pytorch.Google ScholarGoogle Scholar
  32. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. CoRR abs/1705.04304 (2018).Google ScholarGoogle Scholar
  33. Mehdi S. M. Sajjadi, Bernhard Scholkopf, and Michael Hirsch. 2017. EnhanceNet: Single image super-resolution through automated texture synthesis. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17).Google ScholarGoogle ScholarCross RefCross Ref
  34. Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16), 1874–1883.Google ScholarGoogle ScholarCross RefCross Ref
  35. W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). 1874–1883. Google ScholarGoogle Scholar
  36. Jian Sun, Zongben Xu, and Heung-Yeung Shum. 2008. Image super-resolution using gradient profile prior. In Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’08). 1–8.Google ScholarGoogle Scholar
  37. Ying Tai, Jian Yang, and Xiaoming Liu. 2017. Image super-resolution via deep recursive residual network. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17), 2790–2798.Google ScholarGoogle ScholarCross RefCross Ref
  38. Ying Tai, Jian Xi Yang, Xiaoming Liu, and Chunyan Xu. 2017. MemNet: A persistent memory network for image restoration. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV’17), 4549–4557.Google ScholarGoogle ScholarCross RefCross Ref
  39. Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, and Lei Zhang. 2017. NTIRE 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17) Workshops. 1110–1121.Google ScholarGoogle Scholar
  40. Radu Timofte, Vincent De Smet, and Luc Van Gool. 2013. Anchored neighborhood regression for fast example-based super-resolution. In Proceedings of the 2013 IEEE International Conference on Computer Vision, 1920–1927. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Tong Tong, Gen Li, Xiejie Liu, and Qinquan Gao. 2017. Image super-resolution using dense skip connections. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV’17), 4809–4817.Google ScholarGoogle ScholarCross RefCross Ref
  42. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Conference and Workshop on Neural Information Processing Systems (NIPS’17). 2472–2481. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Jiheng Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 4 (2004), 600–612. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. M. Wang. 2012. Movie2Comics: Towards a lively video content presentation. IEEE Trans. Multimedia 14, 3 (2012), 858–870. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Meng Wang, Richang Hong, Guangda Li, Zheng-Jun Zha, Shuicheng Yan, and Tat-Seng Chua. [n.d.]. Event driven web video summarization by tag localization and key-shot identification. IEEE Trans. Multimedia 14, 4 ([n. d.]), 975–985. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, and Xiaoou Tang. 2019. ESRGAN: Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision Workshops (ECCV’19). 63–79.Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Yang Wang. 2020. Survey on deep multi-modal data analytics: Collaboration, rivalry and fusion. arXiv abs/2006.08159. Retrieved from https://arxiv.org/abs/2006.08159.Google ScholarGoogle Scholar
  48. Y. Wang, X. Lin, L. Wu, and W. Zhang. 2017. Effective multi-query expansions: Collaborative deep networks for robust landmark retrieval. IEEE Trans. Image Process. 26, 3 (2017), 1393–1404. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Zhaowen Wang, Ding Liu, Jianchao Yang, Wei Han, and Thomas S. Huang. 2015. Deep networks for image super-resolution with sparse prior. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV’15), 370–378. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Bingzhe Wu, Haodong Duan, Zhichao Liu, and Guangyu Sun. 2017. SRPGAN: Perceptual generative adversarial network for single image super resolution. CoRR abs/1712.05927 (2017).Google ScholarGoogle Scholar
  51. Lihua Wu, Yang Wang, Junbin Gao, and Xue Li. 2019. Where-and-when to look: Deep siamese attention networks for video-based person re-identification. IEEE Trans. Multimedia 21, 6 (2019), 1412–1424.Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. L. Wu, Y. Wang, J. Gao, M. Wang, Z. Zha, and D. Tao. 2020. Deep coattention-based comparator for relative representation learning in person re-identification. IEEE Trans. Neural Netw. Learn. Syst. (2020), 1–14.Google ScholarGoogle Scholar
  53. L. Wu, Y. Wang, X. Li, and J. Gao. 2019. Deep attention-based spatially recursive networks for fine-grained visual recognition. IEEE Trans. Cybernet. 49, 5 (2019), 1791–1802.Google ScholarGoogle ScholarCross RefCross Ref
  54. L. Wu, Y. Wang, and L. Shao. 2019. Cycle-consistent deep generative hashing for cross-modal retrieval. IEEE Trans. Image Process. 28, 4 (2019), 1602–1612.Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. L. Wu, Y. Wang, H. Yin, M. Wang, and L. Shao. 2020. Few-shot deep adversarial learning for video-based person re-identification. IEEE Trans. Image Process. 29 (2020), 1233–1245.Google ScholarGoogle ScholarCross RefCross Ref
  56. Roman Zeyde, Michael Elad, and Matan Protter. 2010. On single image scale-up using sparse-representations. In Curves and Surfaces. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Han Zhang, Ian J. Goodfellow, Dimitris N. Metaxas, and Augustus Odena. 2018. Self-attention generative adversarial networks. CoRR abs/1805.08318 (2018).Google ScholarGoogle Scholar
  58. Kaibing Zhang, Xinbo Gao, Dacheng Tao, and Xuelong Li. 2012. Single image super-resolution with non-local means and steering kernel regression. IEEE Trans. Image Process. 21, 11 (2012), 4544–4556. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Kai Zhang, Wangmeng Zuo, and Lei Zhang. 2017. Learning a single convolutional super-resolution network for multiple degradations. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3262–3271.Google ScholarGoogle Scholar
  60. Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. 2018. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (ECCV’18).Google ScholarGoogle ScholarCross RefCross Ref
  61. Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2018. Residual dense network for image super-resolution. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2472–2481.Google ScholarGoogle ScholarCross RefCross Ref
  62. Wilman W. W. Zou and Pong C. Yuen. 2012. Very low resolution face recognition problem. IEEE Trans. Image Process. 21, 1 (2012), 327–40. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Lightweight Single Image Super-resolution with Dense Connection Distillation Network

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Multimedia Computing, Communications, and Applications
        ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 17, Issue 1s
        January 2021
        353 pages
        ISSN:1551-6857
        EISSN:1551-6865
        DOI:10.1145/3453990
        Issue’s Table of Contents

        Copyright © 2021 Association for Computing Machinery.

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 31 March 2021
        • Revised: 1 July 2020
        • Accepted: 1 July 2020
        • Received: 1 April 2020
        Published in tomm Volume 17, Issue 1s

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Refereed
      • Article Metrics

        • Downloads (Last 12 months)48
        • Downloads (Last 6 weeks)3

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!