skip to main content
research-article

Multi-task Learning-based All-in-one Collaboration Framework for Degraded Image Super-resolution

Authors Info & Claims
Published:16 April 2021Publication History
Skip Abstract Section

Abstract

In this article, we address the degraded image super-resolution problem in a multi-task learning (MTL) manner. To better share representations between multiple tasks, we propose an all-in-one collaboration framework (ACF) with a learnable “junction” unit to handle two major problems that exist in MTL—“How to share” and “How much to share.” Specifically, ACF consists of a sharing phase and a reconstruction phase. Considering the intrinsic characteristic of multiple image degradations, we propose to first deal with the compression artifact, motion blur, and spatial structure information of the input image in parallel under a three-branch architecture in the sharing phase. Subsequently, in the reconstruction phase, we up-sample the previous features for high-resolution image reconstruction with a channel-wise and spatial attention mechanism. To coordinate two phases, we introduce a learnable “junction” unit with a dual-voting mechanism to selectively filter or preserve shared feature representations that come from sharing phase, learning an optimal combination for the following reconstruction phase. Finally, a curriculum learning-based training scheme is further proposed to improve the convergence of the whole framework. Extensive experimental results on synthetic and real-world low-resolution images show that the proposed all-in-one collaboration framework not only produces favorable high-resolution results while removing serious degradation, but also has high computational efficiency, outperforming state-of-the-art methods. We also have applied ACF to some image-quality sensitive practical task, such as pose estimation, to improve estimation accuracy of low-resolution images.

References

  1. Abrar H. Abdulnabi, Gang Wang, Jiwen Lu, and Kui Jia. 2015. Multi-task CNN model for attribute prediction. IEEE Trans. Multimedia 17, 11 (2015), 1949–1959.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. 2011. Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33, 5 (2011), 898–916.Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. 2012. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In Proceedings of the 23rd British Machine Vision Conference (BMVC'12). BMVA Press, 135.1--135.10.Google ScholarGoogle ScholarCross RefCross Ref
  4. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 (2014).Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. J. Mach. Learn. Res. 12, Aug. (2011), 2493–2537.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Chao Dong, Yubin Deng, Chen Change Loy, and Xiaoou Tang. 2015. Compression artifacts reduction by a deep convolutional network. In ICCV. 576–584.Google ScholarGoogle Scholar
  7. Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2014. Learning a deep convolutional network for image super-resolution. In ECCV. Springer, 184–199.Google ScholarGoogle Scholar
  8. Chao Dong, Chen Change Loy, and Xiaoou Tang. 2016. Accelerating the super-resolution convolutional neural network. In ECCV. Springer, 391–407.Google ScholarGoogle Scholar
  9. Ross Girshick. 2015. Fast R-CNN. In ICCV. 1440–1448.Google ScholarGoogle Scholar
  10. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In NIPS. 2672–2680.Google ScholarGoogle Scholar
  11. Yong Guo, Jian Chen, Jingdong Wang, Qi Chen, Jiezhang Cao, Zeshuai Deng, Yanwu Xu, and Mingkui Tan. 2020. Closed-loop matters: Dual regression networks for single image super-resolution. In CVPR. 5407–5416.Google ScholarGoogle Scholar
  12. Wei Han, Shiyu Chang, Ding Liu, Mo Yu, Michael Witbrock, and Thomas S. Huang. 2018. Image super-resolution via dual-state recurrent networks. In CVPR.Google ScholarGoogle Scholar
  13. Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita. 2018. Deep backprojection networks for super-resolution. In CVPR.Google ScholarGoogle Scholar
  14. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. 770–778.Google ScholarGoogle Scholar
  15. Zewei He, Siliang Tang, Jiangxin Yang, Yanlong Cao, Michael Ying Yang, and Yanpeng Cao. 2018. Cascaded deep networks with multiple receptive fields for infrared image super-resolution. IEEE Trans. Circ. Syst. Vid. Technol. 29, 8 (2018).Google ScholarGoogle Scholar
  16. Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-excitation networks. In CVPR.Google ScholarGoogle Scholar
  17. Zhe Hu, Li Xu, and Ming-Hsuan Yang. 2014. Joint depth estimation and camera shake removal from single blurry image. In CVPR. 2893–2900.Google ScholarGoogle Scholar
  18. Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. 2017. Densely connected convolutional networks. In CVPR, Vol. 1. 3.Google ScholarGoogle Scholar
  19. Jizhou Huang, Wei Zhang, Yaming Sun, Haifeng Wang, and Ting Liu. 2018. Improving entity recommendation with search log and multi-task learning. In IJCAI. 4107–4114.Google ScholarGoogle Scholar
  20. Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. 2015. Single image super-resolution from transformed self-exemplars. In CVPR. 5197–5206.Google ScholarGoogle Scholar
  21. Jun-Jie Huang and Wan-Chi Siu. 2017. Learning hierarchical decision trees for single-image super-resolution. IEEE Trans. Circ. Syst. Video Technol. 27, 5 (2017), 937–950.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Jeremy Jancsary, Sebastian Nowozin, and Carsten Rother. 2012. Loss-specific training of non-parametric image restoration models: A new state of the art. In ECCV. Springer, 112–125.Google ScholarGoogle Scholar
  23. Alex Kendall, Yarin Gal, and Roberto Cipolla. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR. 3 (2018).Google ScholarGoogle Scholar
  24. Faisal Khan, Bilge Mutlu, and Xiaojin Zhu. 2011. How do humans teach: On curriculum learning and teaching dimension. In NIPS. 1449–1457.Google ScholarGoogle Scholar
  25. Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Deeply-recursive convolutional network for image super-resolution. In CVPR. 1637–1645.Google ScholarGoogle Scholar
  26. Neeraj Kumar and Amit Sethi. 2016. Fast learning-based single image super-resolution. IEEE Trans. Multimedia 18, 8 (2016), 1504–1515.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiri Matas. 2017. DeblurGAN: Blind motion deblurring using conditional adversarial networks. arXiv preprint arXiv:1711.07064 (2017).Google ScholarGoogle Scholar
  28. Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiri Matas. 2018. DeblurGAN: Blind motion deblurring using conditional adversarial networks. In CVPR.Google ScholarGoogle Scholar
  29. Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. 2017. Deep Laplacian pyramid networks for fast and accurate superresolution. In CVPR, Vol. 2. 5.Google ScholarGoogle Scholar
  30. Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR, Vol. 2. 4.Google ScholarGoogle Scholar
  31. Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced deep residual networks for single image super-resolution. In CVPR Workshops, Vol. 1. 4.Google ScholarGoogle ScholarCross RefCross Ref
  32. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In ECCV. Springer, 740–755.Google ScholarGoogle Scholar
  33. Xiaojiao Mao, Chunhua Shen, and Yu-Bin Yang. 2016. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In NIPS. 2802–2810.Google ScholarGoogle Scholar
  34. Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. 2016. Cross-stitch networks for multi-task learning. In CVPR. 3994–4003.Google ScholarGoogle Scholar
  35. Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR, Vol. 1. 3.Google ScholarGoogle Scholar
  36. Mehdi Noroozi, Paramanand Chandramouli, and Paolo Favaro. 2017. Motion deblurring in the wild. In DAGM-GCPR. Springer, 65–77.Google ScholarGoogle Scholar
  37. Guillaume Obozinski, Ben Taskar, and Michael I. Jordan. 2010. Joint covariate selection and joint subspace selection for multiple classification problems. Stat. Comput. 20, 2 (2010), 231–252.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Jinshan Pan, Deqing Sun, Hanspeter Pfister, and Ming-Hsuan Yang. 2016. Blind image deblurring using dark channel prior. In CVPR. 1628–1636.Google ScholarGoogle Scholar
  39. Dongwon Park, Kwanyoung Kim, and Se Young Chun. 2018. Efficient module based single image super resolution for multiple problems. In CVPR Workshops.Google ScholarGoogle ScholarCross RefCross Ref
  40. Güngör Polatkan, Mingyuan Zhou, Lawrence Carin, David Blei, and Ingrid Daubechies. 2015. A Bayesian nonparametric approach to image super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. 37, 2 (2015), 346–358.Google ScholarGoogle ScholarCross RefCross Ref
  41. Mehdi S. M. Sajjadi, Bernhard Schölkopf, and Michael Hirsch. 2017. Enhancenet: Single image super-resolution through automated texture synthesis. In ICCV. IEEE, 4501–4510.Google ScholarGoogle Scholar
  42. Samuel Schulter, Christian Leistner, and Horst Bischof. 2015. Fast and accurate image upscaling with super-resolution forests. In CVPR. 3791–3799.Google ScholarGoogle Scholar
  43. Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR. 1874–1883.Google ScholarGoogle Scholar
  44. Yukai Shi, Keze Wang, Chongyu Chen, Li Xu, and Liang Lin. 2017. Structure-preserving image super-resolution via contextualized multitask learning. IEEE Trans. Multimedia 19, 12 (2017), 2804–2815.Google ScholarGoogle ScholarCross RefCross Ref
  45. Assaf Shocher, Nadav Cohen, and Michal Irani. 2018. “Zero-Shot” super-resolution using deep internal learning. In CVPR. 3118–3126.Google ScholarGoogle Scholar
  46. Pavel Svoboda, Michal Hradis, David Barina, and Pavel Zemcik. 2016. Compression artifacts removal using convolutional neural networks. arXiv preprint arXiv:1605.00366 (2016).Google ScholarGoogle Scholar
  47. Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. 2017. MemNet: A persistent memory network for image restoration. In CVPR. 4539–4547.Google ScholarGoogle Scholar
  48. Xin Tao, Hongyun Gao, Yi Wang, Xiaoyong Shen, Jue Wang, and Jiaya Jia. 2018. Scale-recurrent network for deep image deblurring. In CVPR.Google ScholarGoogle Scholar
  49. Piotr Teterwak and Lorenzo Torresani. 2014. Shared Roots: Regularizing Deep Neural Networks through Multitask Learning. Dartmouth College Undergraduate Theses. 92. https://digitalcommons.dartmouth.edu/senior_theses/92.Google ScholarGoogle Scholar
  50. Tong Tong, Gen Li, Xiejie Liu, and Qinquan Gao. 2017. Image super-resolution using dense skip connections. In ICCV. IEEE, 4809–4817.Google ScholarGoogle Scholar
  51. Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. 2018. Recovering realistic texture in image super-resolution by deep spatial feature transform. In CVPR.Google ScholarGoogle Scholar
  52. Yifan Wang, Federico Perazzi, Brian McWilliams, Alexander Sorkine-Hornung, Olga Sorkine-Hornung, and Christopher Schroers. 2018. A fully progressive approach to single-image super-resolution. arXiv preprint arXiv:1804.02900 (2018).Google ScholarGoogle Scholar
  53. Ou Wu, Haiqiang Zuo, Weiming Hu, and Bing Li. 2016. Multimodal web aesthetics assessment based on structural SVM and multitask fusion learning. IEEE Trans. Multimedia 18, 6 (2016), 1062–1076.Google ScholarGoogle ScholarCross RefCross Ref
  54. Li Xu, Shicheng Zheng, and Jiaya Jia. 2013. Unnatural L0 sparse representation for natural image deblurring. In CVPR. 1107–1114.Google ScholarGoogle Scholar
  55. Wenming Yang, Yapeng Tian, Fei Zhou, Qingmin Liao, Hai Chen, and Chenglin Zheng. 2016. Consistent coding scheme for single-image super-resolution via independent dictionaries. IEEE Trans. Multimedia 18, 3 (2016), 313–325.Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. Xin Yang, Haiyang Mei, Jiqing Zhang, Ke Xu, Baocai Yin, Qiang Zhang, and Xiaopeng Wei. 2019. DRFN: Deep recurrent fusion network for single-image super-resolution with large factors. IEEE Trans. Multimedia 21, 2 (2019), 328–337.Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Yi Yang, Zhigang Ma, Alexander G. Hauptmann, and Nicu Sebe. 2013. Feature selection for multimedia analysis by sharing information among multiple tasks. IEEE Trans. Multimedia 15, 3 (2013), 661–669.Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Junho Yim, Heechul Jung, ByungIn Yoo, Changkyu Choi, Dusik Park, and Junmo Kim. 2015. Rotating your face using multi-task deep neural network. In CVPR. 676–684.Google ScholarGoogle Scholar
  59. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In NIPS. 3320–3328.Google ScholarGoogle Scholar
  60. Ke Yu, Chao Dong, Chen Change Loy, and Xiaoou Tang. 2016. Deep convolution networks for compression artifacts reduction. arXiv preprint arXiv:1608.02778 (2016).Google ScholarGoogle Scholar
  61. Yuan Yuan, Siyuan Liu, Jiawei Zhang, Yongbing Zhang, Chao Dong, and Liang Lin. 2018. Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. In CVPR. 701–710.Google ScholarGoogle Scholar
  62. Roman Zeyde, Michael Elad, and Matan Protter. 2010. On single image scale-up using sparse-representations. In ICCS. Springer, 711–730.Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. Kai Zhang, Wangmeng Zuo, and Lei Zhang. 2018. Learning a single convolutional super-resolution network for multiple degradations. In CVPR, Vol. 6.Google ScholarGoogle ScholarCross RefCross Ref
  64. Kai Zhang, Wangmeng Zuo, and Lei Zhang. 2019. Deep plug-and-play super-resolution for arbitrary blur kernels. In CVPR. 1671–1681.Google ScholarGoogle Scholar
  65. Xinyi Zhang, Hang Dong, Zhe Hu, Wei-Sheng Lai, Fei Wang, and Ming-Hsuan Yang. 2018. Gated fusion network for joint image deblurring and super-resolution. In BMVC.Google ScholarGoogle Scholar
  66. Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. 2018. Image super-resolution using very deep residual channel attention networks. In ECCV.Google ScholarGoogle Scholar
  67. Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. 2018. Image super-resolution using very deep residual channel attention networks. In ECCV.Google ScholarGoogle Scholar
  68. Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2018. Residual dense network for image super-resolution. In CVPR.Google ScholarGoogle Scholar
  69. Wei Zhao, Benyou Wang, Jianbo Ye, Min Yang, Zhou Zhao, Ruotian Luo, and Yu Qiao. 2018. A multi-task learning approach for image captioning. In IJCAI.Google ScholarGoogle Scholar
  70. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV.Google ScholarGoogle Scholar
  71. Zhiliang Zhu, Fangda Guo, Hai Yu, and Chen Chen. 2014. Fast single image super-resolution via self-example learning and sparse representation. IEEE Trans. Multimedia 16, 8 (2014), 2178–2190.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Multi-task Learning-based All-in-one Collaboration Framework for Degraded Image Super-resolution

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!