Practical temporal consistency for image-based graphics applications

Abstract

We present an efficient and simple method for introducing temporal consistency to a large class of optimization driven image-based computer graphics problems. Our method extends recent work in edge-aware filtering, approximating costly global regularization with a fast iterative joint filtering operation. Using this representation, we can achieve tremendous efficiency gains both in terms of memory requirements and running time. This enables us to process entire shots at once, taking advantage of supporting information that exists across far away frames, something that is difficult with existing approaches due to the computational burden of video data. Our method is able to filter along motion paths using an iterative approach that simultaneously uses and estimates per-pixel optical flow vectors. We demonstrate its utility by creating temporally consistent results for a number of applications including optical flow, disparity estimation, colorization, scribble propagation, sparse data up-sampling, and visual saliency computation.

References

1. Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., and Szeliski, R. 2011. A database and evaluation methodology for optical flow. International Journal of Computer Vision 92, 1, 1--31.
2. Bhat, P., Zitnick, C. L., Cohen, M. F., and Curless, B. 2010. Gradientshop: A gradient-domain optimization framework for image and video filtering. ACM Trans. Graph. 29, 2.
3. Chen, J., Paris, S., and Durand, F. 2007. Real-time edge-aware image processing with the bilateral grid. ACM Trans. Graph. 26, 3, 103.
4. Criminisi, A., Sharp, T., Rother, C., and Pérez, P. 2010. Geodesic image and video editing. ACM Trans. Graph. 29, 5, 134.
5. Dolson, J., Baek, J., Plagemann, C., and Thrun, S. 2010. Upsampling range data in dynamic environments. In CVPR, 1141--1148.
6. Durand, F., and Dorsey, J. 2002. Fast bilateral filtering for the display of high-dynamic-range images. ACM Trans. Graph. 21, 3, 257--266.
7. Gastal, E. S. L., and Oliveira, M. M. 2011. Domain transform for edge-aware image and video processing. ACM Trans. Graph. 30, 4, 69.
8. Guo, C., Ma, Q., and Zhang, L. 2008. Spatio-temporal saliency detection using phase spectrum of quaternion fourier transform. In CVPR, IEEE Computer Society.
9. He, K., Sun, J., and Tang, X. 2010. Guided image filtering. In ECCV (1), Springer, vol. 6311 of Lecture Notes in Computer Science, 1--14.
10. Höffken, M., Oberhoff, D., and Kolesnik, M. 2011. Temporal prediction and spatial regularization in differential optical flow. In ACIVS, Springer, vol. 6915 of Lecture Notes in Computer Science, 576--585.
11. Horn, B. K. P., and Schunck, B. G. 1981. Determining optical flow. Artif. Intell. 17, 1--3, 185--203.
12. Hosni, A., Rhemann, C., Bleyer, M., and Gelautz, M. 2011. Temporally consistent disparity and optical flow via efficient spatio-temporal filtering. In PSIVT (1), Springer, vol. 7087 of Lecture Notes in Computer Science, 165--177.
13. Kopf, J., Cohen, M. F., Lischinski, D., and Uyttendaele, M. 2007. Joint bilateral upsampling. ACM Trans. Graph. 26, 3, 96.
14. Krähenbühl, P., Lang, M., Hornung, A., and Gross, M. H. 2009. A system for retargeting of streaming video. ACM Trans. Graph. 28, 5.
15. Levin, A., Lischinski, D., and Weiss, Y. 2004. Colorization using optimization. ACM Trans. Graph. 23, 3, 689--694.
16. Levin, A., Lischinski, D., and Weiss, Y. 2006. A closed form solution to natural image matting. In CVPR (1), IEEE Computer Society, 61--68.
17. Nehab, D., Maximo, A., Lima, R. S., and Hoppe, H. 2011. Gpu-efficient recursive filtering and summed-area tables. ACM Trans. Graph. 30, 6, 176.
18. Paris, S., Kornprobst, P., and Tumblin, J. 2009. Bilateral Filtering. Now Publishers Inc., Hanover, MA, USA.
19. Perona, P., and Malik, J. 1990. Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell. 12, 7, 629--639.
20. Rhemann, C., Hosni, A., Bleyer, M., Rother, C., and Gelautz, M. 2011. Fast cost-volume filtering for visual correspondence and beyond. In CVPR, IEEE, 3017--3024.
21. Scharstein, D., and Szeliski, R. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision 47, 1--3, 7--42.
22. Sun, D., Roth, S., and Black, M. J. 2010. Secrets of optical flow estimation and their principles. In CVPR, 2432--2439.
23. Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In ICCV, 839--846.
24. Volz, S., Bruhn, A., Valgaerts, L., and Zimmer, H. 2011. Modeling temporal coherence for optical flow. In Proc. 13th International Conference on Computer Vision (ICCV), IEEE Computer Society Press, Barcelona.
25. Wang, O., Lang, M., Frei, M., Hornung, A., Smolic, A., and Gross, M. H. 2011. Stereobrush: Interactive 2d to 3d conversion using discontinuous warps. In SBM, Eurographics Association, 47--54.
26. Wildeboer, M. O., Yendo, T., Tehrani, M. P., and Tanimoto, M. 2010. A semi-automatic multi-view depth estimation method. Proceedings of SPIE, Visual Communications and Image Processing 7744.
27. Xiao, J., Cheng, H., Sawhney, H. S., Rao, C., and Isnardi, M. A. 2006. Bilateral filtering-based optical flow estimation with occlusion detection. In ECCV (1), Springer, vol. 3951 of Lecture Notes in Computer Science, 211--224.
28. Yang, Q., Yang, R., Davis, J., and Nistér, D. 2007. Spatial-depth super resolution for range images. In CVPR, IEEE Computer Society.
29. Zimmer, H., Bruhn, A., and Weickert, J. 2011. Optic flow in harmony. International Journal of Computer Vision 93, 3, 368--388.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Video segmentation

      • Computer vision tasks

        • Scene understanding

  • Computer graphics

    • Image manipulation

    • Rendering