Moving gradients: a path-based method for plausible image interpolation

Abstract

We describe a method for plausible interpolation of images, with a wide range of applications like temporal up-sampling for smooth playback of lower frame rate video, smooth view interpolation, and animation of still images. The method is based on the intuitive idea, that a given pixel in the interpolated frames traces out a path in the source images. Therefore, we simply move and copy pixel gradients from the input images along this path. A key innovation is to allow arbitrary (asymmetric) transition points, where the path moves from one image to the other. This flexible transition preserves the frequency content of the originals without ghosting or blurring, and maintains temporal coherence. Perhaps most importantly, our framework makes occlusion handling particularly simple. The transition points allow for matches away from the occluded regions, at any suitable point along the path. Indeed, occlusions do not need to be handled explicitly at all in our initial graph-cut optimization. Moreover, a simple comparison of computed path lengths after the optimization, allows us to robustly identify occluded regions, and compute the most plausible interpolation in those areas. Finally, we show that significant improvements are obtained by moving gradients and using Poisson reconstruction.

References

1. Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Trans. on Graphics (SIGGRAPH 2004) 23, 3 (aug), 294--302.
2. Alvarez, L., Deriche, R., Papadopoulo, T., and Sanchez, J. 2007. Symmetrical dense optical flow estimation with occlusions detection. Int. Journal of Computer Vision 75, 3 (December), 371--385.
3. Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., and Szeliski, R. 2007. A database and evaluation methodology for optical flow. In Proc. IEEE Int. Conf. Computer Vision, 1--8.
4. Barron, J., Fleet, D., and Beauchemin, S. 1994. Performance of optical flow techniques. Int. Journal of Computer Vision 12, 1, 43--77.
5. Belhumeur, P., and Mumford, D. 1992. A bayesian treatment of the stereo correspondence problem using half-occluded regions. Proc. IEEE Conf. Computer Vision and Pattern Recognition, 506--512.
6. Black, M., and Anandan, P. 1996. The robust estimation of multiple motions: Parametric and piecewise-smooth flow-fields. Computer Vision and Image Understanding 63, 1 (January), 75--104.
7. Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Analysis and Machine Intelligence 23, 11, 1222--1239.
8. Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. 2004. High accuracy optical flow estimation based on a theory for warping. In Proc. European Conf. Computer Vision, vol. 3024, 25--36.
9. Chen, S., and Williams, L. 1993. View interpolation for image synthesis. In Proc. SIGGRAPH 93, 279--288.
10. Fitzgibbon, A. W., Wexler, Y., and Zisserman, A. 2003. Image-based rendering using image-based priors. In Proc. IEEE Int. Conf. Computer Vision, 1176--1183.
11. Geiger, D., Ladendorf, B., and Yuille, A. 1992. Occlusions and binocular stereo. In Proc. European Conf. Computer Vision, 425--433.
12. Gonzalez, and Woods. 2002. Digital Image Processing. Prentice Hall Inc.
13. Jia, J., and Tang, C. 2008. Image stitching using structure deformation. IEEE Trans. Pattern Analysis and Machine Intelligence 30, 4, 617--631.
14. Jia, J., Sun, J., Tang, C., and Shum, H. 2006. Drag-and-drop pasting. ACM Trans. on Graphics (SIGGRAPH 2006) 25, 3 (jul), 631--637.
15. Kolmogorov, V., and Zabih, R. 2001. Computing visual correspondence with occlusions using graph cuts. Proc. IEEE Int. Conf. Computer Vision, 508--515.
16. Kwatra, V., Schödl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: Image and video synthesis using graph cuts. ACM Trans. on Graphics (SIGGRAPH 2003) 22, 3 (July), 277--286.
17. McMillan, L., and Bishop, G. 1995. Plenoptic modeling: An image-based rendering system. In In Proc. SIGGRAPH 95, 39--46.
18. Mémin, E., and Pérez, P. 2002. Hierarchical estimation and segmentation of dense motion fields. Int. Journal of Computer Vision 46, 2, 129--155.
19. Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. on Graphics (SIGGRAPH 2003) 22, 3 (jul), 313--318.
20. Scharstein, D., and Szeliski, R. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithm. Int. Journal of Computer Vision 47, 1, 7--42.
21. Schödl, A., Szeliski, R., Salesin, D., and Essa, I. 2000. Video textures. ACM Trans. on Graphics (SIGGRAPH 2000), 489--498.
22. Seitz, S., and Dyer, C. 1996. View morphing. In In Proc. SIGGRAPH 96, 21--30.
23. Sun, J., Li, Y., Kang, S., and Shum, H. 2005. Symmetric stereo matching for occlusion handling. Proc. IEEE Conf. Computer Vision and Pattern Recognition 2, 399--406.
24. Wang, H., Raskar, R., and Ahuja, N. 2004. Seamless video editing. In Int. Conf. on Pattern Recognition, vol. 3, 858--861.
25. Wang, H., Xu, N., Raskar, R., and Ahuja, N. 2005. Videoshop: A new framework for spatio-temporal video editing in gradient domain. Proc. IEEE Conf. Computer Vision and Pattern Recognition 2, 1201.
26. Wolberg, G. 1990. Digital Image Warping. IEEE Press.
27. Xiao, J., Cheng, H., Sawhney, H., Rao, C., and Isnardi, M. 2006. Bilateral filtering-based optical flow estimation with occlusion detection. In Proc. European Conf. Computer Vision, vol. 1, 211--224.
28. Zitnick, C. L., Jojic, N., and Kang, S. B. 2005. Consistent segmentation for optical flow estimation. In Proc. IEEE Int. Conf. Computer Vision, 1308--1315.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

  • Computer graphics

    • Image manipulation

      • Image processing