Video deblurring for hand-held cameras using patch-based synthesis

Abstract

Videos captured by hand-held Cameras often contain significant camera shake, causing many frames to be blurry. Restoring shaky videos not only requires smoothing the camera motion and stabilizing the content, but also demands removing blur from video frames. However, video blur is hard to remove using existing single or multiple image deblurring techniques, as the blur kernel is both spatially and temporally varying. This paper presents a video deblurring method that can effectively restore sharp frames from blurry ones caused by camera shake. Our method is built upon the observation that due to the nature of camera shake, not all video frames are equally blurry. The same object may appear sharp on some frames while blurry on others. Our method detects sharp regions in the video, and uses them to restore blurry regions of the same content in nearby frames. Our method also ensures that the deblurred frames are both spatially and temporally coherent using patch-based synthesis. Experimental results show that our method can effectively remove complex video blur under the presence of moving objects and other outliers, which cannot be achieved using previous deconvolution-based approaches.

References

1. Agrawal, A., Xu, Y., and Raskar, R. 2009. Invertible motion blur in video. ACM Trans. Graphics 28, 3, 95:1--95:8.
2. Baker, S., and Matthews, I. 2004. Lucas-kanade 20 years on: A unifying framework. International Journal of Computer Vision (IJCV) 56, 3, 221--255.
3. Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. B. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graphics 28, 3, 24:1--24:11.
4. Barnes, C., Shechtman, E., Goldman, D. B., and Finkelstein, A. 2010. The generalized PatchMatch correspondence algorithm. In Proc. ECCV 2010, 29--43.
5. Buades, A., and Coll, B. 2005. A non-local algorithm for image denoising. In Proc. CVPR 2006, 60--65.
6. Cai, J.-F., Ji, H., Liu, C., and Shen, Z. 2009. Blind motion deblurring using multiple images. J. Comput. Phys. 228, 5057--5071.
7. Chen, J., Yuan, L., Tang, C.-K., and Quan, L. 2008. Robust dual motion deblurring. In Proc. CVPR 2008, 1--8.
8. Cho, S., and Lee, S. 2009. Fast motion deblurring. ACM Trans. Graphics 28, 5, 145:1--145:8.
9. Cho, S., Matsushita, Y., and Lee, S. 2007. Removing non-uniform motion blur from images. In Proc. ICCV 2007, 1--8.
10. Cho, S., Wang, J., and Lee, S. 2011. Handling outliers in non-blind image deconvolution. In Proc. ICCV 2011, 495--502.
11. Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. Proc. ACM SIGGRAPH 2001, 341--346.
12. Fergus, R., Singh, B., Hertzmann, A., Roweis, S. T., and Freeman, W. T. 2006. Removing camera shake from a single photograph. ACM Trans. Graphics 25, 3, 787--794.
13. Freedman, G., and Fattal, R. 2011. Image and video up-scaling from local self-examples. ACM Trans. Graphics 30, 2, 12:1--12:11.
14. Fried, D. L. 1978. Probability of getting a lucky short-exposure image through turbulence. J. Opt. Soc. Am. 68, 12, 1651--1657.
15. Grundmann, M., Kwatra, V., and Essa, I. 2011. Auto-directed video stabilization with robust L1 optimal camera paths. In Proc. CVPR 2011, 225--232.
16. Gupta, A., Joshi, N., Zitnick, C. L., Cohen, M., and Curless, B. 2010. Single image deblurring using motion density functions. In Proc. ECCV 2010, 171--184.
17. HaCohen, Y., Shechtman, E., Goldman, D. B., and Lischinski, D. 2011. Non-rigid dense correspondence with applications for image enhancement. ACM Trans. Graphics 30, 4, 70:1--70:10.
18. Hirsch, M., Schuler, C. J., Harmeling, S., and Schölkopf, B. 2011. Fast removal of non-uniform camera shake. In Proc. ICCV 2011, 463--470.
19. Joshi, N., and Cohen, M. 2010. Seeing mt. rainier: Lucky imaging for multi-image denoising, sharpening, and haze removal. In Proc. ICCP 2010, 1--8.
20. Kwatra, V., Essa, I., Bobick, A., and Kwatra, N. 2005. Texture optimization for example-based synthesis. ACM Trans. Graphics 24, 3, 795--802.
21. Levin, A., Weiss, Y., Durand, F., and Freeman, W. T. 2011. Efficient marginal likelihood optimization in blind deconvolution. In Proc. CVPR 2011, 2657--2664.
22. Li, Y., Kang, S. B., Joshi, N., Seitz, S. M., and Huttenlocher, D. P. 2010. Generating sharp panoramas from motion-blurred videos. In Proc. CVPR 2010, 2424--2431.
23. Liu, C., and Freeman, W. T. 2010. A high-quality video denoising algorithm based on reliable motion estimation. In Proc. ECCV 2010, 706--719.
24. Liu, F., Gleicher, M., Wang, J., Jin, H., and Agarwala, A. 2011. Subspace video stabilization. ACM Trans. Graphics 30, 1, 4:1--4:10.
25. Matsushita, Y., Ofek, E., Ge, W., Tang, X., and Shum, H.-Y. 2006. Full-frame video stabilization with motion inpainting. IEEE Trans. Pattern Analysis Machine Intelligence 28, 7, 1150--1163.
26. Osher, S., and Rudin, L. I. 1990. Feature-oriented image enhancement using shock filters. SIAM Journal on Numerical Analysis 27, 4, 919--940.
27. Shan, Q., Jia, J., and Agarwala, A. 2008. High-quality motion deblurring from a single image. ACM Trans. Graphics 27, 3, 73:1--73:10.
28. Shi, J., and Tomasi, C. 1994. Good features to track. In Proc. CVPR 1994, 593--600.
29. Simakov, D., Caspi, Y., Shechtman, E., and Irani, M. 2008. Summarizing visual data using bidirectional similarity. In Proc. CVPR 2008, 1--8.
30. Tai, Y.-W., Tan, P., and Brown, M. S. 2011. Richardson-lucy deblurring for scenes under a projective motion path. IEEE Trans. Pattern Analysis Machine Intelligence 33, 8, 1603--1618.
31. Whyte, O., Sivic, J., Zisserman, A., and Ponce, J. 2010. Non-uniform deblurring for shaken images. In Proc. CVPR 2010, 491--498.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

  • Computer graphics

    • Image manipulation