High-quality image deblurring with panchromatic pixels

Abstract

Image deblurring has been a very challenging problem in recent decades. In this article, we propose a high-quality image deblurring method with a novel image prior based on a new imaging system. The imaging system has a newly designed sensor pattern achieved by adding panchromatic (pan) pixels to the conventional Bayer pattern. Since these pan pixels are sensitive to all wavelengths of visible light, they collect a significantly higher proportion of the light striking the sensor. A new demosaicing algorithm is also proposed to restore full-resolution images from pixels on the sensor. The shutter speed of pan pixels is controllable to users. Therefore, we can produce multiple images with different exposures. When long exposure is needed under dim light, we read pan pixels twice in one shot: one with short exposure and the other with long exposure. The long-exposure image is often blurred, while the short-exposure image can be sharp and noisy. The short-exposure image plays an important role in deblurring, since it is sharp and there is no alignment problem for the one-shot image pair. For the algorithmic aspect, our method runs in a two-step maximum-a-posteriori (MAP) fashion under a joint minimization of the blur kernel and the deblurred image. The algorithm exploits a combined image prior with a statistical part and a spatial part, which is powerful in ringing controls. Extensive experiments under various conditions and settings are conducted to demonstrate the performance of our method.

References

1. Agrawal, A., Xu, Y., and Raskar, R. 2009. Invertible motion blur in video. ACM Trans. Graph. 28, 3, 95:1--95:8.
2. Alleysson, D., Süsstrunk, S., and Hérault, J. 2005. Linear demosaicing inspired by the human visual system. IEEE Trans. Image Process. 14, 4, 439--449.
3. Bayer, B. 1976. Color imaging array. US Patent 3,971,065.
4. Ben-Ezra, M. and Nayar, S. 2003. Motion deblurring using hybrid imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 657--664.
5. Border, J., Anderson, T., and Deever, A. 2009. Determining and correcting for imaging device motion during an exposure. US Patent application US2009/0021588A1.
6. Cai, J., Ji, H., Liu, C., and Shen, Z. 2009a. Blind motion deblurring from a single image using sparse approximation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 104--111.
7. Cai, J., Ji, H., Liu, C., and Shen, Z. 2009b. High-quality curvelet-based motion deblurring from an image pair. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1566--1573.
8. Chen, J., Yuan, L., Tang, C., and Quan, L. 2008. Robust dual motion deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1--8.
9. Cho, S. and Lee, S. 2009. Fast motion deblurring. ACM Trans. Graph. 28, 5, 145:1--145:8.
10. Compton, J. and Hamilton, J. 2007. Capturing images under varying lighting conditions. US Patent application 2007/0046807A1.
11. Fergus, R., Singh, B., Hertzmann, A., Roweis, S., and Freeman, W. 2006. Removing camera shake from a single photograph. ACM Trans. Graph. 25, 3, 787--794.
12. Hou, T., Wang, S., and Qin, H. 2011. Image deconvolution with multi-stage convex relaxation and its perceptual evaluation. IEEE Trans. Image Process. 20, 12, 3383--3392.
13. Joshi, N., Kang, S., Zitnick, C., and Szeliski, R. 2010. Image deblurring using inertial measurement sensors. ACM Trans. Graph. 29, 4, 30:1--30:9.
14. Joshi, N., Zitnick, C., Szeliski, R., and Kriegman, D. 2009. Image deblurring and denoising using color priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1550--1557.
15. Krishnan, D. and Fergus, R. 2009. Fast image deconvolution using hyper-laplacian priors. In Proceedings of the Neural Information Processing Systems Conference.
16. Levin, A., Fergus, R., Durand, F., and Freeman, W. 2007. Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph. 26, 6, 70--77.
17. Levin, A., Sand, P., Cho, T., Durand, F., and Freeman, W. 2008. Motion-invariant photograph. ACM Trans. Graph. 27, 3, 71:1--71:9.
18. Levin, A., Weiss, Y., Durand, F., and Freeman, W. 2009. Understanding and evaluating blind deconvolution algorithms. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1964--1971.
19. Lim, S. and Silverstein, D. 2006. Method for deblurring an image. US Patent application 2006/0187308A1.
20. Lucy, L. 1974. Bayesian-Based iterative method of image restoration. J. Astron. 79, 745--754.
21. Muramatsu, A. 1989. Color video signal generating device using monochrome and color image sensors having different resolutions to form a luminance signal. US Patent 4,876,591.
22. Nayar, S. and Narasimhan, S. 2002. Assorted pixels: Multi-Sampled imaging with structural models. In Proceedings of the European Conference on Computer Vision. 472--479.
23. Raskar, R., Agrawal, A., and Tumblin, J. 2006. Coded exposure photography:motion deblurring using fluttered shutter. ACM Trans. Graph. 25, 3, 795--804.
24. Shan, Q., Jia, J., and Agarwala, A. 2008. High-quality motion deblurring from a single image. ACM Trans. Graph. 27, 3, 73:1--73:10.
25. Susanu, G., Petrescu, S., Nanu, F., Capata, A., and Corcoran, P. 2009. Rgbw sensor array. US Patent application 2009/0167893A1.
26. Tomasi, C. and Manduchi, R. 1998. Bilateral filtering for gray and color images. In Proceedings of the International Conference on Computer Vision. 839--846.
27. Wang, S., Hou, T., and Miller, R. 2011. Image deblurring using panchromatic pixels. US Patent application 2011/0090378A1.
28. Whyte, O., Sivc, J., Zisserman, A., and Ponce, J. 2010. Non-uniform deblurring for shaken images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 491--498.
29. Xu, L. and Jia, J. 2010. Two-phase kernel estimation for robust motion deblurring. In Proceedings of the European Conference on Computer Vision. 157--170.
30. Yuan, L., Sun, J., Quan, L., and Shum, H. 2007. Image deblurring with blurred/noisy image pairs. ACM Trans. Graph. 26, 3.
31. Yuan, L., Sun, J., Quan, L., and Shum, H. 2008. Progressive inter-scale and intra-scale non-blind image deconvolution. ACM Trans. Graph. 27, 3, 74:1--74:10.
32. Zhou, C. and Nayar, S. 2009. What are good apertures for defocus deblurring? In Proceedings of the International Conference on Computational Photography. 1--8.

Index Terms

• Applied computing

  • Document management and text processing

    • Document capture

      • Document scanning

      • Graphics recognition and interpretation

• Computing methodologies

  • Artificial intelligence

    • Computer vision

  • Computer graphics

    • Image manipulation