Phase-based video motion processing

Abstract

We introduce a technique to manipulate small movements in videos based on an analysis of motion in complex-valued image pyramids. Phase variations of the coefficients of a complex-valued steerable pyramid over time correspond to motion, and can be temporally processed and amplified to reveal imperceptible motions, or attenuated to remove distracting changes. This processing does not involve the computation of optical flow, and in comparison to the previous Eulerian Video Magnification method it supports larger amplification factors and is significantly less sensitive to noise. These improved capabilities broaden the set of applications for motion processing in videos. We demonstrate the advantages of this approach on synthetic and natural video sequences, and explore applications in scientific analysis, visualization and video enhancement.

References

1. Bai, J., Agarwala, A., Agrawala, M., and Ramamoorthi, R. 2012. Selectively de-animating video. ACM Transactions on Graphics.
2. Bojanic, S., Simpson, T., and Bolger, C. 2001. Ocular microtremor: a tool for measuring depth of anaesthesia? British Journal of Anaesthesia 86, 4, 519--522.
3. Buades, A., Coll, B., and Morel, J.-M. 2008. Nonlocal image and movie denoising. International Journal of Computer Vision 76, 123--139.
4. Dabov, K., Foi, A., and Egiazarian, K. 2007. Video denoising by sparse 3d transform-domain collaborative filtering. In Proc. 15th European Signal Processing Conference, vol. 1, 7.
5. Fleet, D. J., and Jepson, A. D. 1990. Computation of component image velocity from local phase information. Int. J. Comput. Vision 5, 1 (Sept.), 77--104.
6. Freeman, W. T., Adelson, E. H., and Heeger, D. J. 1991. Motion without movement. SIGGRAPH Comput. Graph. 25 (Jul), 27--30.
7. Fuchs, M., Chen, T., Wang, O., Raskar, R., Seidel, H.-P., and Lensch, H. P. 2010. Real-time temporal shaping of high-speed video streams. Computers & Graphics 34, 5, 575--584.
8. Gautama, T., and Van Hulle, M. 2002. A phase-based approach to the estimation of the optical flow field using spatial filtering. Neural Networks, IEEE Transactions on 13, 5 (sep), 1127--1136.
9. Liu, C., and Freeman, W. 2010. A high-quality video denoising algorithm based on reliable motion estimation. In Computer Vision ECCV 2010, K. Daniilidis, P. Maragos, and N. Paragios, Eds., vol. 6313 of Lecture Notes in Computer Science. Springer Berlin Heidelberg, 706--719.
10. Liu, C., Torralba, A., Freeman, W. T., Durand, F., and Adelson, E. H. 2005. Motion magnification. ACM Trans. Graph. 24 (Jul), 519--526.
11. Portilla, J., and Simoncelli, E. P. 2000. A parametric texture model based on joint statistics of complex wavelet coefficients. Int. J. Comput. Vision 40, 1 (Oct.), 49--70.
12. Rolfs, M. 2009. Microsaccades: Small steps on a long way. Vision Research 49, 20, 2415--2441.
13. Rubinstein, M., Liu, C., Sand, P., Durand, F., and Freeman, W. T. 2011. Motion denoising with application to time-lapse photography. IEEE Computer Vision and Pattern Recognition (CVPR) (June), 313--320.
14. Simoncelli, E. P., and Freeman, W. T. 1995. The steerable pyramid: a flexible architecture for multi-scale derivative computation. In Proceedings of the 1995 International Conference on Image Processing (Vol. 3)-Volume 3 - Volume 3, IEEE Computer Society, Washington, DC, USA, ICIP '95, 3444--.
15. Simoncelli, E. P., Freeman, W. T., Adelson, E. H., and Heeger, D. J. 1992. Shiftable multi-scale transforms. IEEE Trans. Info. Theory 2, 38, 587--607.
16. Wang, J., Drucker, S. M., Agrawala, M., and Cohen, M. F. 2006. The cartoon animation filter. ACM Trans. Graph. 25, 1169--1173.
17. Wu, H.-Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., and Freeman, W. 2012. Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph. (Proc. SIGGRAPH) 31 (aug).

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Video segmentation

      • Computer vision tasks

        • Scene understanding