Content-preserving warps for 3D video stabilization

Abstract

We describe a technique that transforms a video from a hand-held video camera so that it appears as if it were taken with a directed camera motion. Our method adjusts the video to appear as if it were taken from nearby viewpoints, allowing 3D camera movements to be simulated. By aiming only for perceptual plausibility, rather than accurate reconstruction, we are able to develop algorithms that can effectively recreate dynamic scenes from a single source video. Our technique first recovers the original 3D camera motion and a sparse set of 3D, static scene points using an off-the-shelf structure-from-motion system. Then, a desired camera path is computed either automatically (e.g., by fitting a linear or quadratic path) or interactively. Finally, our technique performs a least-squares optimization that computes a spatially-varying warp from each input video frame into an output frame. The warp is computed to both follow the sparse displacements suggested by the recovered 3D structure, and avoid deforming the content in the video frame. Our experiments on stabilizing challenging videos of dynamic scenes demonstrate the effectiveness of our technique.

References

1. Alexa, M., Cohen-Or, D., and Levin, D. 2000. As-rigid-as-possible shape interpolation. In Proceedings of ACM SIGGRAPH 2000, Computer Graphics Proceedings, Annual Conference Series, 157--164.
2. Avidan, S., and Shamir, A. 2007. Seam carving for content-aware image resizing. ACM Transactions on Graphics 26, 3 (July), 10:1--10:9.
3. Beier, T., and Neely, S. 1992. Feature-based image metamorphosis. In Computer Graphics (Proceedings of SIGGRAPH 92), 35--42.
4. Bhat, P., Zitnick, C. L., Snavely, N., Agarwala, A., Agrawala, M., Cohen, M., Curless, B., and Kang, S. B. 2007. Using photographs to enhance videos of a static scene. In Rendering Techniques 2007: 18th Eurographics Workshop on Rendering, 327--338.
5. Bookstein, F. L. 1989. Principal warps: Thin-plate splines and the decomposition of deformations. IEEE Trans. Pattern Anal. Mach. Intell. 11, 6, 567--585.
6. Buehler, C., Bosse, M., and McMillan, L. 2001. Nonmetric image-based rendering for video stabilization. In 2001 Conference on Computer Vision and Pattern Recognition (CVPR 2001), 609--614.
7. Buehler, C., Bosse, M., McMillan, L., Gortler, S. J., and Cohen, M. F. 2001. Unstructured lumigraph rendering. In Proceedings of ACM SIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, 425--432.
8. Chuang, Y.-Y., Agarwala, A., Curless, B., Salesin, D. H., and Szeliski, R. 2002. Video matting of complex scenes. ACM Transactions on Graphics 21, 3 (July), 243--248.
9. Fitzgibbon, A., Wexler, Y., and Zisserman, A. 2005. Image-based rendering using image-based priors. International Journal of Computer Vision 63, 2 (July), 141--151.
10. Gal, R., Sorkine, O., and Cohen-Or, D. 2006. Feature-aware texturing. In Rendering Techniques 2006: 17th Eurographics Workshop on Rendering, 297--304.
11. Gleicher, M. L., and Liu, F. 2008. Re-cinematography: Improving the camerawork of casual video. ACM Transactions on Multimed. 5, 1, 1--28.
12. Gleicher, M., and Witkin, A. 1992. Through-the-lens camera control. In Computer Graphics (Proceedings of SIGGRAPH 92), 331--340.
13. Gomes, J., Darsa, L., Costa, B., and Velho, L. 1998. Warping and morphing of graphical objects. Morgan Kaufmann Publishers Inc., San Francisco, CA.
14. Hartley, R. I., and Zisserman, A. 2000. Multiple View Geometry in Computer Vision. Cambridge University Press.
15. Heckbert, P. S. 1989. Fundamentals of texture mapping and image warping. Tech. Rep. UCB/CSD-89-516, EECS Department, University of California, Berkeley, Jun.
16. Hoiem, D., Efros, A. A., and Hebert, M. 2005. Automatic photo pop-up. ACM Transactions on Graphics 24, 3 (Aug.), 577--584.
17. Igarashi, T., Moscovich, T., and Hughes, J. F. 2005. Asrigid-as-possible shape manipulation. ACM Transactions on Graphics 24, 3 (Aug.), 1134--1141.
18. Itti, L., Koch, C., and Niebur, E. 1998. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 20, 11 (Nov), 1254--1259.
19. Kawin, B. 1992. How Movies Work. Univ. of California Press.
20. Lee, J., and Shin, S. Y. 2002. General construction of time-domain filters for orientation data. IEEE Transactions on Visualization and Computer Graphics 8, 2 (April-June), 119--128.
21. Matsushita, Y., Ofek, E., Ge, W., Tang, X., and Shum, H.-Y. 2006. Full-frame video stabilization with motion inpainting. IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 7, 1150--1163.
22. Meingast, M., Geyer, C., and Sastry, S. 2005. Geometric models of rolling-shutter cameras. In 6th Int. workshop on Omnidirectional vision, Camera networks, and non-classical cameras.
23. Morimoto, C., and Chellappa, R. 1997. Evaluation of image stabilization algorithms. In DARPA Image Understanding Workshop DARPA97, 295--302.
24. Murray, R. M., Sastry, S. S., and Zexiang, L. 1994. A Mathematical Introduction to Robotic Manipulation. CRC Press, Inc., Boca Raton, FL, USA.
25. Nister, D. 2003. Preemptive RANSAC for live structure and motion estimation. IEEE International Conference on Computer Vision 1, 199--206.
26. Rubinstein, M., Shamir, A., and Avidan, S. 2008. Improved seam carving for video retargeting. ACM Transactions on Graphics 27, 3 (Aug.), 16:1--16:9.
27. Schaefer, S., McPhail, T., and Warren, J. 2006. Image deformation using moving least squares. ACM Transactions on Graphics 25, 3 (July), 533--540.
28. Thormählen, T., and Seidel, H.-P. 2008. 3D-modeling by ortho-image generation from image sequences. ACM Transactions on Graphics 27, 3 (Aug.), 86:1--86:5.
29. Torr, P. H. S., Fitzgibbon, A. W., and Zisserman, A. 1999. The problem of degeneracy in structure and motion recovery from uncalibrated image sequences. International Journal of Computer Vision 32, 1, 27--44.
30. Torresani, L., Hertzmann, A., and Bregler, C. 2008. Nonrigid structure-from-motion: Estimating shape and motion with hierarchical priors. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 5, 878--892.
31. van den Hengel, A., Dick, A., Thormählen, T., Ward, B., and Torr, P. H. S. 2007. Videotrace: Rapid interactive scene modelling from video. ACM Transactions on Graphics 26, 3 (July), 86:1--86:5.
32. Wang, Y.-S., Tai, C.-L., Sorkine, O., and Lee, T.-Y. 2008. Optimized scale-and-stretch for image resizing. ACM Transactions on Graphics 27, 5 (Dec.), 118:1--118:8.
33. Wexler, Y., Shechtman, E., and Irani, M. 2004. Space-time video completion. In 2004 Conference on Computer Vision and Pattern Recognition (CVPR 2004), 120--127.
34. Wolf, L., Guttmann, M., and Cohen-Or, D. 2007. Nonhomogeneous content-driven video-retargeting. In IEEE International Conference on Computer Vision, 1--6.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

      • Computer vision tasks

        • Scene understanding

  • Machine learning