Invertible motion blur in video

Abstract

We show that motion blur in successive video frames is invertible even if the point-spread function (PSF) due to motion smear in a single photo is non-invertible. Blurred photos exhibit nulls (zeros) in the frequency transform of the PSF, leading to an ill-posed deconvolution. Hardware solutions to avoid this require specialized devices such as the coded exposure camera or accelerating sensor motion. We employ ordinary video cameras and introduce the notion of null-filling along with joint-invertibility of multiple blur-functions. The key idea is to record the same object with varying PSFs, so that the nulls in the frequency component of one frame can be filled by other frames. The combined frequency transform becomes null-free, making deblurring well-posed. We achieve jointly-invertible blur simply by changing the exposure time of successive frames. We address the problem of automatic deblurring of objects moving with constant velocity by solving the four critical components: preservation of all spatial frequencies, segmentation of moving parts, motion estimation of moving parts, and non-degradation of the static parts of the scene. We demonstrate several challenging cases of object motion blur including textured backgrounds and partial occluders.

References

1. Agrawal, A., and Raskar, R. 2007. Resolving Objects at Higher Resolution from a Single Motion-blurred Image. In Proc. Conf. Comp. Vision and Pattern Recognition, 1--8.
2. Bascle, B., Blake, A., and Zisserman, A. 1996. Motion Deblurring and Super-resolution from an Image Sequence. In Proc. European Conf. Computer Vision, vol. 2, 573--582.
3. Ben-Ezra, M., and Nayar, S. 2004. Motion-based Motion Deblurring. IEEE Trans. Pattern Anal. Machine Intell. 26, 6 (Jun), 689--698.
4. Berenstein, C., and Patrick, E. 1990. Exact Deconvolution for Multiple Convolution Operators-an Overview, plus Performance Characterizations for Imaging Sensors. Proc. of the IEEE 78 (Apr.), 723--734.
5. Chen, W.-G., Nandhakumar, N., and Martin, W. N. 1996. Image Motion Estimation from Motion Smear-A New Computational Model. IEEE Trans. Pattern Anal. Mach. Intell. 18, 4 (Apr.), 412--425.
6. Chen, J., Yuan, L., Tang, C.-K., and Quan, L. 2008. Robust Dual Motion Deblurring. In Proc. Conf. Comp. Vision and Pattern Recognition, 1--8.
7. Cho, S., Matsushita, Y., and Lee, S. 2007. Removing Non-Uniform Motion Blur from Images. In Proc. Int'l Conf. Computer Vision, 1--8.
8. Dai, S., and Wu, Y. 2008. Motion from Blur. In Proc. Conf. Comp. Vision and Pattern Recognition, 1--8.
9. Debevec, P. E., and Malik, J. 1997. Recovering High Dynamic Range Radiance Maps from Photographs. In Proc. SIGGRAPH 97, 369--378.
10. Dowski, E. R., and Cathey, W. 1995. Extended Depth of Field through Wavefront Coding. Appl. Optics 34, 11 (Apr.), 1859--1866.
11. Fergus, R., Singh, B., Hertzmann, A., Roweis, S. T., and Freeman, W. T. 2006. Removing Camera Shake from a Single Photograph. ACM Trans. Graph. 25, 3 (jul), 787--794.
12. Grossberg, M., and Nayar, S. 2003. High Dynamic Range from Multiple Images: Which Exposures to Combine? In ICCV Workshop on Color and Photometric Methods in Computer Vision (CPMCV).
13. Jansson, P. 1997. Deconvolution of Image and Spectra, 2nd ed. Academic Press.
14. Jia, J. 2007. Single Image Motion Deblurring using Transparency. In Proc. Conf. Comp. Vision and Pattern Recognition, 1--8.
15. Joshi, N., Szeliski, R., and Kriegman, D. 2008. PSF Estimation using Sharp Edge Prediction. In Proc. Conf. Comp. Vision and Pattern Recognition, 1--8.
16. Levin, A., Fergus, R., Durand, F., and Freeman, W. T. 2007. Image and Depth from a Conventional Camera with a Coded Aperture. ACM Trans. Graph. 26, 3 (Jul.), 70.
17. Levin, A., Lischinski, D., and Weiss, Y. 2008. A Closed-Form Solution to Natural Image Matting. IEEE Trans. Pattern Anal. Mach. Intell. 30, 2, 228--242.
18. Levin, A., Sand, P., Cho, T. S., Durand, F., and Freeman, W. T. 2008. Motion-Invariant Photography. ACM Trans. Graph. 27, 3 (Aug.), 71.
19. Lucy, L. 1974. An iterative technique for the rectification of observed distributions. J. Astronomy 79, 745--754.
20. Mann, S., and Picard, R. W. 1995. Being Undigital with Digital Cameras: Extending Dynamic Range by Combining Differently Exposed Pictures. In Proc. of IS&T 48th Annual Conference, 422--428.
21. Nagahara, H., Kuthirummal, S., Zhou, C., and Nayar, S. 2008. Flexible Depth of Field Photography. In Proc. European Conf. Computer Vision, 60--73.
22. Piccardi, M. 2004. Background Subtraction Techniques: a Review. In Proc. IEEE SMC Intl. Conf. Systems, Man and Cybernetics.
23. Raskar, R., Agrawal, A., and Tumblin, J. 2006. Coded Rxposure Photography: Motion Deblurring using Fluttered Shutter. ACM Trans. Graph. 25, 3 (Jul.), 795--804.
24. Rav-Acha, A., and Peleg, S. 2005. Two Motion-blurred Images are Better than One. Pattern Recognition Letters 26, 3, 311--317.
25. Richardson, W. 1972. Bayesian-based iterative method of image restoration. J. Opt. Soc. of America 62, 1, 55--59.
26. Schultz, R. R., and Stevenson, R. L. 1996. Extraction of High-Resolution Frames from Video Sequences. IEEE Trans. Image Processing 5 (jun), 996--1011.
27. Sellent, A., Eisemann, M., and Magnor, M. 2008. Calculating Motion Fields from Images with Two Different Exposure Times. Tech. rep., Computer Graphics Lab, Technical University of Braunschweig, 5.
28. Shan, Q., Jia, J., and Agarwala, A. 2008. High-Quality Motion Deblurring from a Single Image. ACM Trans. Graph. 27, 3 (Aug.), 73.
29. Shechtman, E., Caspi, Y., and Irani, M. 2002. Increasing Space-Time Resolution in Video. In Proc. European Conf. Computer Vision, 753--768.
30. Tai., W. Y., Hao, D., Brown, M. S., and Lin, S. 2008. Image/Video Deblurring using a Hybrid Camera. In Proc. Conf. Comp. Vision and Pattern Recognition, 1--8.
31. Telleen, J., Sullivan, A., Yee, J., Gunawardane, P., Wang, O., Collins, I., and Davis, J. 2007. Synthetic Shutter Speed Imaging. In Proc. Eurographics, 591--598.
32. Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High Performance Imaging using Large Camera Arrays. ACM Trans. Graph. 24, 3 (Jul.), 765--776.
33. Yuan, L., Sun, J., Quan, L., and Shum, H.-Y. 2007. Image Deblurring with Blurred/Noisy Image Pairs. ACM Trans. Graph. 26, 3 (Jul.), 1.
34. Yuan, L., Sun, J., Quan, L., and Shum, H.-Y. 2008. Progressive Inter-Scale and Intra-Scale Non-Blind Image Deconvolution. ACM Trans. Graph. 27, 3 (Aug.), 74.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Interest point and salient region detections

      • Computer vision tasks

        • Video summarization

      • Image and video acquisition

        • Motion capture

  • Computer graphics

    • Animation

      • Motion capture

      • Motion processing

    • Image manipulation

  • Modeling and simulation

    • Model development and analysis

      • Model verification and validation

      • Modeling methodologies