skip to main content
research-article

Local Bidirection Recurrent Network for Efficient Video Deblurring with the Fused Temporal Merge Module

Published:07 June 2023Publication History
Skip Abstract Section

Abstract

Video deblurring methods exploit the correlation between consecutive blurry inputs to generate sharp frames. However, designing an effective and efficient method is a challenging problem for video deblurring. To guarantee the effectiveness and further improve the deblurring performance, we adopt the recurrent-based method as the baseline and reconsider the recurrent mechanism as well as the temporal feature alignment in the state-of-the-art methods. For the recurrent mechanism, we add the local backward connection to the global forward recurrent backbone to effectively exploit accurate future information. For the temporal alignment, we adopt a fused temporal merge module that exploits the superiority of flow-based and kernel-based methods with progressive correlation volumes estimation. In addition, we evaluate our method with both synthetic datasets (GoPro, DVD) and a realistic dataset (BSD). The experimental results demonstrate that our method achieves significant performance improvement with a slight computational cost increase against the state-of-the-art video deblurring methods. The extended ablation studies verify the effectiveness of our model.

Skip Supplemental Material Section

Supplemental Material

REFERENCES

  1. [1] Bao Wenbo, Lai Wei-Sheng, Ma Chao, Zhang Xiaoyun, Gao Zhiyong, and Yang Ming-Hsuan. 2019. Depth-aware video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 37033712.Google ScholarGoogle ScholarCross RefCross Ref
  2. [2] Chan Kelvin C. K., Wang Xintao, Yu Ke, Dong Chao, and Loy Chen Change. 2021. BasicVSR: The search for essential components in video super-resolution and beyond. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 49474956.Google ScholarGoogle ScholarCross RefCross Ref
  3. [3] Chan Kelvin C. K., Zhou Shangchen, Xu Xiangyu, and Loy Chen Change. 2022. BasicVSR++: Improving video super-resolution with enhanced propagation and alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 59725981.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Cho Sunghyun, Wang Jue, and Lee Seungyong. 2012. Video deblurring for hand-held cameras using patch-based synthesis. ACM Transactions on Graphics 31, 4 (2012), 19.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. [5] Delbracio Mauricio and Sapiro Guillermo. 2015. Hand-held video deblurring via efficient fourier aggregation. IEEE Transactions on Computational Imaging 1, 4 (2015), 270283.Google ScholarGoogle ScholarCross RefCross Ref
  6. [6] He Jiale, Yang Gaobo, Liu Xin, and Ding Xiangling. 2020. Spatio-temporal saliency-based motion vector refinement for frame rate up-conversion. ACM Transactions on Multimedia Computing, Communications, and Applications 16, 2 (2020), 118.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. [7] Hu Mengshun, Jiang Kui, Nie Zhixiang, and Wang Zheng. 2022. You only align once: Bidirectional interaction for spatial-temporal video super-resolution. In Proceedings of the 30th ACM International Conference on Multimedia. 847855.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. [8] Kim Tae Hyun, Ahn Byeongjoo, and Lee Kyoung Mu. 2013. Dynamic scene deblurring. In Proceedings of the IEEE International Conference on Computer Vision. 31603167.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Kim Tae Hyun and Lee Kyoung Mu. 2014. Segmentation-free dynamic scene deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 27662773.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. [10] Kim Tae Hyun and Lee Kyoung Mu. 2015. Generalized video deblurring for dynamic scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 54265434.Google ScholarGoogle ScholarCross RefCross Ref
  11. [11] Kim Tae Hyun, Lee Kyoung Mu, Scholkopf Bernhard, and Hirsch Michael. 2017. Online video deblurring via dynamic temporal blending network. In Proceedings of the IEEE International Conference on Computer Vision. 40384047.Google ScholarGoogle ScholarCross RefCross Ref
  12. [12] Kim Tae Hyun, Nah Seungjun, and Lee Kyoung Mu. 2017. Dynamic video deblurring using a locally adaptive blur model. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 10 (2017), 23742387.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. [13] Kingma Diederik P. and Ba Jimmy. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google ScholarGoogle Scholar
  14. [14] Li Chen, Song Li, Xie Rong, and Zhang Wenjun. 2022. L0 structure-prior assisted blur-intensity aware efficient video deblurring. Neurocomputing 483 (2022), 195209.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. [15] Li Dongxu, Xu Chenchen, Zhang Kaihao, Yu Xin, Zhong Yiran, Ren Wenqi, Suominen Hanna, and Li Hongdong. 2021. Arvo: Learning all-range volumetric correspondence for video deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 77217731.Google ScholarGoogle ScholarCross RefCross Ref
  16. [16] Li Xianguo, Sun Yemei, Yang Yanli, and Miao Changyun. 2019. Symmetrical residual connections for single image super-resolution. ACM Transactions on Multimedia Computing, Communications, and Applications 15, 1 (2019), 110.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. [17] Li Yunpeng, Kang Sing Bing, Joshi Neel, Seitz Steve M., and Huttenlocher Daniel P.. 2010. Generating sharp panoramas from motion-blurred videos. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 24242431.Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Lin Jing, Cai Yuanhao, Hu Xiaowan, Wang Haoqian, Yan Youliang, Zou Xueyi, Ding Henghui, Zhang Yulun, Timofte Radu, and Gool Luc Van. 2022. Flow-guided sparse transformer for video deblurring. In Proceedings of the International Conference on Machine Learning. 1333413343.Google ScholarGoogle Scholar
  19. [19] Matsushita Yasuyuki, Ofek Eyal, Ge Weina, Tang Xiaoou, and Shum Heung-Yeung. 2006. Full-frame video stabilization with motion inpainting. IEEE Transactions on Pattern Analysis and Machine Intelligence7 (2006), 11501163.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. [20] Nah Seungjun, Baik Sungyong, Hong Seokil, Moon Gyeongsik, Son Sanghyun, Timofte Radu, and Lee Kyoung Mu. 2019. NTIRE 2019 challenge on video deblurring and super-resolution: Dataset and study. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.Google ScholarGoogle ScholarCross RefCross Ref
  21. [21] Nah Seungjun, Kim Tae Hyun, and Lee Kyoung Mu. 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 38833891.Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Nah Seungjun, Son Sanghyun, and Lee Kyoung Mu. 2019. Recurrent neural networks with intra-frame iterations for video deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 81028111.Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Pan Jinshan, Bai Haoran, and Tang Jinhui. 2020. Cascaded deep video deblurring using temporal sharpness prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 30433051.Google ScholarGoogle ScholarCross RefCross Ref
  24. [24] Ren Wenqi, Pan Jinshan, Cao Xiaochun, and Yang Ming-Hsuan. 2017. Video deblurring via semantic segmentation and pixel-wise non-linear kernel. In Proceedings of the IEEE International Conference on Computer Vision. 10771085.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Son Hyeongseok, Lee Junyong, Lee Jonghyeop, Cho Sunghyun, and Lee Seungyong. 2021. Recurrent video deblurring with blur-invariant motion estimation and pixel volumes. ACM Transactions on Graphics 40, 5 (2021), 118.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. [26] Su Shuochen, Delbracio Mauricio, Wang Jue, Sapiro Guillermo, Heidrich Wolfgang, and Wang Oliver. 2017. Deep video deblurring for hand-held cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 12791288.Google ScholarGoogle ScholarCross RefCross Ref
  27. [27] Sun Deqing, Yang Xiaodong, Liu Ming-Yu, and Kautz Jan. 2018. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 89348943.Google ScholarGoogle ScholarCross RefCross Ref
  28. [28] Sun Jian, Cao Wenfei, Xu Zongben, and Ponce Jean. 2015. Learning a convolutional neural network for non-uniform motion blur removal. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 769777.Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Wang Xintao, Chan Kelvin C. K., Yu Ke, Dong Chao, and Loy Chen Change. 2019. EDVR: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.Google ScholarGoogle ScholarCross RefCross Ref
  30. [30] Wieschollek Patrick, Hirsch Michael, Scholkopf Bernhard, and Lensch Hendrik. 2017. Learning blind motion deblurring. In Proceedings of the IEEE International Conference on Computer Vision. 231240.Google ScholarGoogle ScholarCross RefCross Ref
  31. [31] Wulff Jonas and Black Michael Julian. 2014. Modeling blurred video with layers. In Proceedings of the European Conference on Computer Vision. 236252.Google ScholarGoogle ScholarCross RefCross Ref
  32. [32] Yi Peng, Wang Zhongyuan, Jiang Kui, Jiang Junjun, Lu Tao, Tian Xin, and Ma Jiayi. 2021. Omniscient video super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 44294438.Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Zhang Haichao, Wipf David, and Zhang Yanning. 2014. Multi-observation blind deconvolution with an adaptive sparse prior. IEEE Transactions on Pattern Analysis and Machine Intelligence 36, 8 (2014), 16281643.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. [34] Zhang Huicong, Xie Haozhe, and Yao Hongxun. 2022. Spatio-temporal deformable attention network for video deblurring. In Computer Vision—ECCV 2022. Lecture Notes in Computer Science, Vol. 13676. Springer, 581596.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. [35] Zhong Zhihang, Gao Ye, Zheng Yinqiang, and Zheng Bo. 2020. Efficient spatio-temporal recurrent neural network for video deblurring. In Proceedings of the European Conference on Computer Vision. 191207.Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. [36] Zhou Shangchen, Zhang Jiawei, Pan Jinshan, Xie Haozhe, Zuo Wangmeng, and Ren Jimmy. 2019. Spatio-temporal filter adaptive network for video deblurring. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 24822491.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Local Bidirection Recurrent Network for Efficient Video Deblurring with the Fused Temporal Merge Module

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Multimedia Computing, Communications, and Applications
        ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 5s
        October 2023
        280 pages
        ISSN:1551-6857
        EISSN:1551-6865
        DOI:10.1145/3599694
        • Editor:
        • Abdulmotaleb El Saddik
        Issue’s Table of Contents

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 7 June 2023
        • Online AM: 13 March 2023
        • Accepted: 7 March 2023
        • Revised: 6 February 2023
        • Received: 8 October 2022
        Published in tomm Volume 19, Issue 5s

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      View Full Text
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!