Selectively de-animating video

Abstract

We present a semi-automated technique for selectively deanimating video to remove the large-scale motions of one or more objects so that other motions are easier to see. The user draws strokes to indicate the regions of the video that should be immobilized, and our algorithm warps the video to remove the large-scale motion of these regions while leaving finer-scale, relative motions intact. However, such warps may introduce unnatural motions in previously motionless areas, such as background regions. We therefore use a graph-cut-based optimization to composite the warped video regions with still frames from the input video; we also optionally loop the output in a seamless manner. Our technique enables a number of applications such as clearer motion visualization, simpler creation of artistic cinemagraphs (photos that include looping motions in some regions), and new ways to edit appearance and complicated motion paths in video by manipulating a de-animated representation. We demonstrate the success of our technique with a number of motion visualizations, cinemagraphs and video editing examples created from a variety of short input videos, as well as visual and numerical comparison to previous techniques.

References

1. Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Transactions on Graphics 23, 3 (Aug.), 294--302.
2. Agarwala, A., Zheng, K. C., Pal, C., Agrawala, M., Cohen, M., Curless, B., Salesin, D. H., and Szeliski, R. 2005. Panoramic video textures. ACM Transactions on Graphics 24, 3 (Aug.), 821--827.
3. Assa, J., Caspi, Y., and Cohen-Or, D. 2005. Action synopsis: pose selection and illustration. ACM Transactions on Graphics 24, 3 (Aug.), 667--676.
4. Bai, X., Wang, J., Simons, D., and Sapiro, G. 2009. Video snapcut: Robust video object cutout using localized classifiers. ACM Transactions on Graphics 28, 3 (July), 70:1--70:11.
5. Barnes, C., Goldman, D. B., Shechtman, E., and Finkelstein, A. 2010. Video tapestries with continuous temporal zoom. ACM Transactions on Graphics 29, 4 (July), 89:1--89:9.
6. Bennett, E. P., and McMillan, L. 2007. Computational time-lapse video. ACM Transactions on Graphics 26, 3 (July), 102:1--102:6.
7. Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 11 (Nov.), 1222--1239.
8. Caspi, Y., Axelrod, A., Matsushita, Y., and Gamliel, A. 2006. Dynamic stills and clip trailers. The Visual Computer 22, 9--10, 642--652.
9. Chuang, Y.-Y., Agarwala, A., Curless, B., Salesin, D. H., and Szeliski, R. 2002. Video matting of complex scenes. ACM Transactions on Graphics 21, 3 (July), 243--248.
10. Chuang, Y.-Y., Goldman, D. B., Zheng, K. C., Curless, B., Salesin, D. H., and Szeliski, R. 2005. Animating pictures with stochastic motion textures. ACM Transactions on Graphics 24, 3 (Aug.), 853--860.
11. Cohen, M. F., and Szeliski, R. 2006. The moment camera. Computer 39, 8, 40--45.
12. Correa, C. D., and Ma, K.-L. 2010. Dynamic video narratives. ACM Transactions on Graphics 29, 4 (July), 88:1--88:9.
13. Goldman, D. B., Curless, B., Salesin, D., and Seitz, S. M. 2006. Schematic storyboarding for video visualization and editing. ACM Transactions on Graphics 25, 3 (July), 862--871.
14. Kim, B., and Essa, I. 2005. Video-based nonphotorealistic and expressive illustration of motion. In Computer Graphics International 2005.
15. Kwatra, V., Schödl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: Image and video synthesis using graph cuts. ACM Transactions on Graphics 22, 3 (July), 277--286.
16. Liu, C., Torralba, A., Freeman, W. T., Durand, F., and Adelson, E. H. 2005. Motion magnification. ACM Transactions on Graphics 24, 3 (Aug.), 519--526.
17. Liu, F., Gleicher, M., Jin, H., and Agarwala, A. 2009. Content-preserving warps for 3d video stabilization. ACM Transactions on Graphics 28, 3 (July), 44:1--44:9.
18. Liu, F., Gleicher, M., Wang, J., Jin, H., and Agarwala, A. 2011. Subspace video stabilization. ACM Transactions on Graphics 30, 1 (Jan.), 4:1--4:10.
19. Lucas, B. D., and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. International Joint Conference on Artificial Intelligence.
20. Pritch, Y., Rav-Acha, A., and Peleg, S. 2008. Nonchronological video synopsis and indexing. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 11 (Nov.), 1971--1984.
21. Rav-Acha, A., Kohli, P., Rother, C., and Fitzgibbon, A. 2008. Unwrap mosaics: A new representation for video editing. ACM Transactions on Graphics 27, 3 (Aug.), 17:1--17:11.
22. Rubinstein, M., Liu, C., Sand, P., Durand, F., and Freeman, W. T. 2011. Motion denoising with application to time-lapse photography. IEEE Computer Vision and Pattern Recognition (CVPR) (June), 313--320.
23. Schödl, A., Szeliski, R., Salesin, D. H., and Essa, I. 2000. Video textures. In Proceedings of ACM SIGGRAPH 2000, Computer Graphics Proceedings, Annual Conference Series, 489--498.
24. Shi, J., and Tomasi, C. 1994. Good features to track. In Computer Vision and Pattern Recognition, 593--600.
25. Sutton, G. P., and Burrows, M. 2011. Biomechanics of jumping in the flea. J Exp Biol 214, 5 (Mar.), 836--847.
26. Tompkin, J., Pece, F., Subr, K., and Kautz, J. 2011. Towards moment imagery: Automatic cinemagraphs. Visual Media Production, Conference for 0, 87--93.
27. Truong, B. T., and Venkatesh, S. 2007. Video abstraction: A systematic review and classification. ACM Trans. Multimedia Comput. Commun. Appl. 3, 1 (Feb.).

Index Terms

• Computing methodologies

  • Computer graphics

    • Image manipulation