skip to main content
research-article

Robust Video Stabilization based on Motion Decomposition

Published:16 March 2023Publication History
Skip Abstract Section

Abstract

Video stabilization aims to eliminate camera jitter and improve the visual experience of shaky videos. Video stabilization methods often ignore the active movement of the foreground objects and the camera, and may result in distortion and over-smoothing problems. To resolve these issues, this paper proposes a novel video stabilization method based on motion decomposition. Since the inter-frame movement of foreground objects is different from that of the background, we separate foreground feature points from background feature points by modifying the classic density based spatial clustering method of applications with noise (DBSCAN). The movement of background feature points is consistent with the movement of the camera, which can be decomposed into the camera jitter and the active movement of the camera. And the movement of foreground feature points can be decomposed into the movement of the camera and the active movement of foreground objects. Based on motion decomposition, we design first-order and second-order trajectory smoothing constraints to eliminate the high-frequency and low-frequency components of the camera jitter. To reduce content distortion, shape-preserving constraints, and regularization constraints are taken to generate stabilized views of all feature points. Experimental results demonstrate the effectiveness and robustness of the proposed video stabilization method on a variety of challenging videos.

REFERENCES

  1. [1] Buehler Chris, Bosse Michael, and McMillan Leonard. 2001. Non-metric image-based rendering for video stabilization. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Google ScholarGoogle ScholarCross RefCross Ref
  2. [2] Chang Hung-Chang, Lai Shang-Hong, and Lu Kuang-Rong. 2004. A robust and efficient video stabilization algorithm. In Proceedings of the 2004 IEEE International Conference on Multimedia and Expo.2932.Google ScholarGoogle Scholar
  3. [3] Chang Hung-Chang, Lai Shang-Hong, and Lu Kuang-Rong. 2006. A robust real-time video stabilization algorithm. In Proceedings of the Journal of Visual Communication and Image Representation. 659673.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Choi Jinsoo and Kweon In So. 2020. Deep iterative frame interpolation for full-frame video stabilization. In Proceedings of the ACM Transactions on Graphics.19.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. [5] Chu Chung-Hua. 2015. Visual comfort for stereoscopic 3D by using motion sensors on 3D mobile devices. ACM Transactions on Multimedia Computing, Communications, and Applications 12, 1s (2015), 120.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. [6] Dong Jing and Liu Haibo. 2016. Video stabilization for strict real-time applications. In Proceedings of the IEEE Transactions on Circuits and Systems for Video Technology. 716724.Google ScholarGoogle Scholar
  7. [7] Ester Martin, Kriegel Hans-Peter, Sander Jörg, and Xu Xiaowei. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the kdd. 226231.Google ScholarGoogle Scholar
  8. [8] Fischler Martin A. and Bolles Robert C.. 1981. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. In Proceedings of the Association for Computing Machinery. 381395.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Gleicher Michael L. and Liu Feng. 2008. Re-cinematography: Improving the camerawork of casual video. ACM Transactions on Multimedia Computing, Communications, and Applications 5, 1 (2008), 128.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. [10] Goldstein Amit and Fattal Raanan. 2012. Video stabilization using epipolar geometry. In Proceedings of the ACM Transactions on Graphics.110.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. [11] Grundmann Matthias, Kwatra Vivek, Castro Daniel, and Essa Irfan. 2012. Calibration-free rolling shutter removal. In Proceedings of the 2012 IEEE International Conference on Computational Photography. 18.Google ScholarGoogle ScholarCross RefCross Ref
  12. [12] Grundmann M., Kwatra V., Castro D., and Essa I.. 2012. Calibration-free rolling shutter removal. In Proceedings of the 2012 IEEE International Conference on Computational Photography. 18.Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Grundmann M., Kwatra V., and Essa I.. 2011. Auto-directed video stabilization with robust L1 optimal camera paths. In Proceedings of the CVPR 2011. 225232.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. [14] Igarashi Takeo, Moscovich Tomer, and Hughes John F.. 2005. As-rigid-as-possible shape manipulation. In Proceedings of the ACM Transactions on Graphics. 11341141.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. [15] Kalal Zdenek, Mikolajczyk Krystian, and Matas Jiri. 2010. Forward-backward error: Automatic detection of tracking failures. In Proceedings of the 2010 20th International Conference on Pattern Recognition. 27562759.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. [16] Koh Y. J., Lee C., and Kim C.. 2015. Video stabilization based on feature trajectory augmentation and selection and robust mesh grid warping. In Proceedings of the IEEE Transactions on Image Processing. 52605273.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. [17] Lee Der-Tsai and Schachter Bruce J.. 1980. Two algorithms for constructing a delaunay triangulation. In Proceedings of the International Journal of Computer Information Sciences. 219242.Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Lee Ken-Yi, Chuang Yung-Yu, Chen Bing-Yu, and Ouhyoung Ming. 2009. Video stabilization using robust feature trajectories. In Proceedings of the 2009 IEEE 12th International Conference on Computer Vision. 13971404.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Lee Se-Ho, Kim Jin-Hwan, Choi Kwang Pyo, Sim Jae-Young, and Kim Chang-Su. 2014. Video saliency detection based on spatiotemporal feature learning. In Proceedings of the 2014 IEEE International Conference on Image Processing. 11201124.Google ScholarGoogle ScholarCross RefCross Ref
  20. [20] Ling Qiang, Deng Sibin, Li Feng, Huang Qinghua, and Li Xuelong. 2016. A feedback-based robust video stabilization method for traffic videos. In Proceedings of the IEEE Transactions on Circuits and Systems for Video Technology. 561572.Google ScholarGoogle Scholar
  21. [21] Liu Feng, Gleicher Michael, Jin Hailin, and Agarwala Aseem. 2009. Content-preserving warps for 3D video stabilization. ACM Transactions on Graphics 28, 3 (2009), 19.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. [22] Liu Feng, Gleicher Michael, Wang Jue, Jin Hailin, and Agarwala Aseem. 2011. Subspace video stabilization. In Proceedings of the ACM Transactions on Graphics. 110.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. [23] Liu Shuaicheng, Tan Ping, Yuan Lu, Sun Jian, and Zeng Bing. 2016. Meshflow: Minimum latency online video stabilization. In Proceedings of the European Conference on Computer Vision. 800815.Google ScholarGoogle ScholarCross RefCross Ref
  24. [24] Liu Shuaicheng, Wang Yinting, Yuan Lu, Bu Jiajun, Tan Ping, and Sun Jian. 2012. Video stabilization with a depth camera. In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition. 8995.Google ScholarGoogle Scholar
  25. [25] Liu Shuaicheng, Xu Binhan, Deng Chuang, Zhu Shuyuan, Zeng Bing, and Gabbouj Moncef. 2016. A hybrid approach for near-range video stabilization. In Proceedings of the IEEE Transactions on Circuits and Systems for Video Technology. 19221933.Google ScholarGoogle Scholar
  26. [26] Liu Shuaicheng, Yuan Lu, Tan Ping, and Sun Jian. 2013. Bundled camera paths for video stabilization. In Proceedings of the ACM Trans. Graph. 10 pages.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. [27] Liu Shuaicheng, Yuan Lu, Tan Ping, and Sun Jian. 2014. Steadyflow: Spatially smooth optical flow for video stabilization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 42094216.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. [28] Liu Yu-Lun, Lai Wei-Sheng, Yang Ming-Hsuan, Chuang Yung-Yu, and Huang Jia-Bin. 2021. Hybrid neural fusion for full-frame video stabilization. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 22992308.Google ScholarGoogle ScholarCross RefCross Ref
  29. [29] Ma Tiezheng, Nie Yongwei, Zhang Qing, Zhang Zhensong, Sun Hanqiu, and Li Guiqing. 2019. Effective video stabilization via joint trajectory smoothing and frame warping. IEEE Transactions on Visualization and Computer Graphics 26, 11 (2019), 31633176.Google ScholarGoogle Scholar
  30. [30] Matsushita Y., Ofek E., Ge Weina, Tang Xiaoou, and Shum Heung-Yeung. 2006. Full-frame video stabilization with motion inpainting. In Proceedings of the IEEE Transactions on Pattern Analysis and Machine Intelligence. 11501163.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. [31] Meng Quanling, Zhu Heyan, Zhang Weigang, Piao Xuefeng, and Zhang Aijie. 2020. Action recognition using form and motion modalities. ACM Transactions on Multimedia Computing, Communications, and Applications 16, 1s (2020), 116.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. [32] Morimoto Carlos and Chellappa Ramalingam. 1996. Fast electronic digital image stabilization. In Proceedings of the 13th International Conference on Pattern Recognition. 284288.Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Nie Yongwei, Su Tan, Zhang Zhensong, Sun Hanqiu, and Li Guiqing. 2017. Dynamic video stitching via shakiness removing. In Proceedings of the IEEE Transactions on Image Processing. 164178.Google ScholarGoogle Scholar
  34. [34] Ren Shaoqing, He Kaiming, Girshick Ross, and Sun Jian. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems, 91–99.Google ScholarGoogle Scholar
  35. [35] Hartley R. and Zisserman A.. 2003. Multiple view geometry in computer vision. In Proceedings of the Cambridge University Press.Google ScholarGoogle Scholar
  36. [36] Smith Brandon M, Zhang Li, Jin Hailin, and Agarwala Aseem. 2009. Light field video stabilization. In Proceedings of the 2009 IEEE 12th International Conference on Computer Vision. 341348.Google ScholarGoogle ScholarCross RefCross Ref
  37. [37] Tomasi Carlo and Kanade Takeo. 1991. Detection and tracking of point features. In Proceedings of the International Journal of Computer Vision.Google ScholarGoogle Scholar
  38. [38] Wang Miao, Yang Guo-Ye, Lin Jin-Kun, Zhang Song-Hai, Shamir Ariel, Lu Shao-Ping, and Hu Shi-Min. 2018. Deep online video stabilization with multi-grid warping transformation learning. IEEE Transactions on Image Processing 28, 5 (2018), 22832292.Google ScholarGoogle ScholarCross RefCross Ref
  39. [39] Wang Yu-Shuen, Liu Feng, Hsu Pu-Sheng, and Lee Tong-Yee. 2013. Spatially and temporally optimized video stabilization. In Proceedings of the IEEE Transactions on Visualization and Computer Graphics. 13541361.Google ScholarGoogle Scholar
  40. [40] Wang Zhong-Qiang, Zhang Lei, and Huang Hua. 2013. Multiplane video stabilization. In Proceedings of the Computer Graphics Forum. 265273.Google ScholarGoogle ScholarCross RefCross Ref
  41. [41] Wu Huicong, Xiao Liang, Lian Zhichao, and Shim Hiuk Jae. 2019. Locally low-rank regularized video stabilization with motion diversity constraints. In Proceedings of the IEEE Transactions on Circuits and Systems for Video Technology. 28732887.Google ScholarGoogle ScholarCross RefCross Ref
  42. [42] Wu Huicong, Xiao Liang, and Wei Zhihui. 2021. Simultaneous video stabilization and rolling shutter removal. IEEE Transactions on Image Processing 30 (2021), 46374652.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. [43] Wu Jingjing, Jiang Jianguo, Qi Meibin, Chen Cuiqun, and Liu Yimin. 2022. Improving feature discrimination for object tracking by structural-similarity-based metric learning. ACM Transactions on Multimedia Computing, Communications, and Applications 18, 4 (2022), 123.Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. [44] Xu Yufei, Zhang Jing, Maybank Stephen J., and Tao Dacheng. 2022. Dut: Learning video stabilization by simply watching unstable videos. IEEE Transactions on Image Processing 31 (2022), 43064320.Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. [45] Yang Junlan, Schonfeld Dan, and Mohamed Magdi. 2009. Robust video stabilization based on particle filter tracking of projected camera motion. 19, 7 (2009), 945954.Google ScholarGoogle Scholar
  46. [46] Yu Jiyang and Ramamoorthi Ravi. 2020. Learning video stabilization using optical flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 81598167.Google ScholarGoogle ScholarCross RefCross Ref
  47. [47] Zhang Fang-Lue, Wang Jue, Zhao Han, Martin Ralph R., and Hu Shi-Min. 2015. Simultaneous camera path optimization and distraction removal for improving amateur video. In Proceedings of the IEEE Transactions on Image Processing. 59825994.Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. [48] Zhang G., Hua W., Shao X. Qin Y., and Bao H.. 2009. Video stabilization based on a 3D perspective camera model. In Proceedings of the Visual Computer. 9971008.Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. [49] Zhang Lei, Xu Qian-Kun, and Huang Hua. 2015. A global approach to fast video stabilization. In Proceedings of the IEEE Transactions on Circuits and Systems for Video Technology. 225235.Google ScholarGoogle Scholar
  50. [50] Zhao Minda and Ling Qiang. 2020. PWStableNet: Learning pixel-wise warping maps for video stabilization. In Proceedings of the IEEE Transactions on Image Processing. 35823595.Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. [51] Zhao M. and Ling Q.. 2021. Adaptively meshed video stabilization. In Proceedings of the IEEE Transactions on Circuits and Systems for Video Technology. 35043517.Google ScholarGoogle ScholarCross RefCross Ref
  52. [52] Zhou Zihan, Jin Hailin, and Ma Yi. 2013. Plane-based content preserving warps for video stabilization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 22992306.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Robust Video Stabilization based on Motion Decomposition

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Multimedia Computing, Communications, and Applications
      ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 5
      September 2023
      262 pages
      ISSN:1551-6857
      EISSN:1551-6865
      DOI:10.1145/3585398
      • Editor:
      • Abdulmotaleb El Saddik
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 16 March 2023
      • Online AM: 30 January 2023
      • Accepted: 12 January 2023
      • Revised: 20 December 2022
      • Received: 6 April 2022
      Published in tomm Volume 19, Issue 5

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
    • Article Metrics

      • Downloads (Last 12 months)173
      • Downloads (Last 6 weeks)27

      Other Metrics

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!