Handheld multi-frame super-resolution

Abstract

Compared to DSLR cameras, smartphone cameras have smaller sensors, which limits their spatial resolution; smaller apertures, which limits their light gathering ability; and smaller pixels, which reduces their signal-to-noise ratio. The use of color filter arrays (CFAs) requires demosaicing, which further degrades resolution. In this paper, we supplant the use of traditional demosaicing in single-frame and burst photography pipelines with a multiframe super-resolution algorithm that creates a complete RGB image directly from a burst of CFA raw images. We harness natural hand tremor, typical in handheld photography, to acquire a burst of raw frames with small offsets. These frames are then aligned and merged to form a single image with red, green, and blue values at every pixel site. This approach, which includes no explicit demosaicing step, serves to both increase image resolution and boost signal to noise ratio. Our algorithm is robust to challenging scene conditions: local motion, occlusion, or scene changes. It runs at 100 milliseconds per 12-megapixel RAW input burst frame on mass-produced mobile phones. Specifically, the algorithm is the basis of the Super-Res Zoom feature, as well as the default merge method in Night Sight mode (whether zooming or not) on Google's flagship phone.

References

1. Adobe. 2012. Digital Negative (DNG) Specification. https://www.adobe.com/content/dam/acom/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf.
2. Simon Baker and Takeo Kanade. 2002. Limits on super-resolution and how to break them. IEEE Trans. PAMI 24, 9 (2002), 1167--1183.
3. Bryce E Bayer. 1976. Color imaging array. US Patent 3,971,065.
4. Stefanos P Belekos, Nikolaos P Galatsanos, and Aggelos K Katsaggelos. 2010. Maximum a posteriori video super-resolution using a new multichannel image prior. IEEE Trans. Image Processing 19, 6 (2010), 1451--1464.
5. Moshe Ben-Ezra, Assaf Zomet, and Shree K Nayar. 2005. Video super-resolution using controlled subpixel detector shifts. IEEE Trans. PAMI 27, 6 (2005), 977--987.
6. Josef Bigün, Goesta H. Granlund, and Johan Wiklund. 1991. Multidimensional orientation estimation with applications to texture analysis and optical flow. IEEE Trans. PAMI 8 (1991), 775--790.
7. James F Blinn. 1982. A generalization of algebraic surface drawing. ACM TOG 1, 3 (1982), 235--256.
8. Edward Chang, Shiufun Cheung, and Davis Y Pan. 1999. Color filter array recovery using a threshold-based variable number of gradients. In Sensors, Cameras, and Applications for Digital Photography, Vol. 3650. 36--44.
9. CIPA. 2018. CIPA Report. www.cipa.jp/stats/report_e.html. {Online; accessed 29-Nov-2018}.
10. Sabrina Dammertz and Alexander Keller. 2008. Image synthesis by rank-1 lattices. In Monte Carlo and Quasi-Monte Carlo Methods 2006. 217--236.
11. Michael Drobot. 2014. Hybrid reconstruction anti-aliasing. In ACM SIGGRAPH Courses.
12. Joan Duran and Antoni Buades. 2014. Self-similarity and spectral correlation adaptive algorithm for color demosaicking. IEEE Trans. Image Processing 23, 9 (2014), 4031--4040.
13. Michael Elad and Arie Feuer. 1997. Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Trans. Image Processing 6, 12 (1997), 1646--1658.
14. Sina Farsiu, Michael Elad, and Peyman Milanfar. 2006. Multiframe demosaicing and super-resolution of color images. IEEE Trans. Image Processing 15, 1 (2006), 141--159.
15. David J Fleet and Allan D Jepson. 1990. Computation of component image velocity from local phase information. IJCV 5, 1 (1990), 77--104.
16. Flickr. 2017. Top Devices of 2017 on Flickr. https://blog.flickr.net/en/2017/12/07/top-devices-of-2017/. {Online; accessed 11-Jan-2019}.
17. Alessandro Foi, Mejdi Trimeche, Vladimir Katkovnik, and Karen Egiazarian. 2008. Practical Poissonian-Gaussian noise modeling and fitting for single-image raw-data. IEEE Trans. Image Processing 17, 10 (2008), 1737--1754.
18. Michaël Gharbi, Gaurav Chaurasia, Sylvain Paris, and Frédo Durand. 2016. Deep joint demosaicking and denoising. ACM TOG 35, 6 (2016), 191.
19. Clément Godard, Kevin Matzen, and Matt Uyttendaele. 2018. Deep burst denoising. In Proc. ECCV, Vol. 11219. 560--577.
20. Tomomasa Gotoh and Masatoshi Okutomi. 2004. Direct super-resolution and registration using raw CFA images. In Proc. CVPR, Vol. 2.
21. Chris Harris and Mike Stephens. 1988. A combined corner and edge detector.. In Alvey Vision Conference, Vol. 15. 10--5244.
22. Samuel W Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM TOG 35, 6 (2016), 192.
23. Sung Hee Park and Marc Levoy. 2014. Gyro-based multi-image deconvolution for removing handshake blur. In Proc. ICCV. 3366--3373.
24. Felix Heide, Markus Steinberger, Yun-Ta Tsai, Mushfiqur Rouf, Dawid Pajkk, Dikpal Reddy, Orazio Gallo, Jing Liu, Wolfgang Heidrich, et al. 2014. FlexISP: A flexible camera image processing framework. ACM TOG 33, 6 (2014), 231.
25. Carl W Helstrom. 1969. Detection and resolution of incoherent objects by a background-limited optical system. JOSA 59, 2 (1969), 164--175.
26. Robert Herzog, Elmar Eisemann, Karol Myszkowski, and H-P Seidel. 2010. Spatio-temporal upsampling on the GPU. In Proceedings of the 2010 ACM SIGGRAPH symposium on Interactive 3D Graphics and Games. 91--98.
27. Keigo Hirakawa and Thomas W Parks. 2005. Adaptive homogeneity-directed demosaicing algorithm. IEEE Trans. Image Processing 14, 3 (2005), 360--369.
28. Keigo Hirakawa and Thomas W Parks. 2006. Joint demosaicing and denoising. IEEE Trans. Image Processing 15, 8 (2006), 2146--2157.
29. Terence D Hunt. 2004. Image Super-Resolution Using Adaptive 2-D Gaussian Basis Function Interpolation. Technical Report. Air Force Inst of Tech Wright-Patterson AFB OH School of Engineering.
30. Michal Irani and Shmuel Peleg. 1991. Improving resolution by image registration. CVGIP: Graphical models and image processing 53, 3 (1991), 231--239.
31. Jorge Jimenez, Diego Gutierrez, Jason Yang, Alexander Reshetov, Pete Demoreuille, Tobias Berghoff, Cedric Perthuis, Henry Yu, Morgan McGuire, Timothy Lottes, Hugh Malan, Emil Persson, Dmitry Andreev, and Tiago Sousa. 2011. Filtering Approaches for Real-Time Anti-Aliasing. In ACM SIGGRAPH Courses.
32. Takeo Kanade and Masatoshi Okutomi. 1991. A stereo matching algorithm with an adaptive window: Theory and experiment. In Proc. IEEE ICRA. IEEE, 1088--1095.
33. Brian Karis. 2014. High-Quality Temporal Supersampling. In ACM SIGGRAPH Courses.
34. Darwin T Kuan, Alexander A Sawchuk, Timothy C Strand, and Pierre Chavel. 1985. Adaptive noise smoothing filter for images with signal-dependent noise. IEEE Trans. PAMI 2 (1985), 165--177.
35. Yeon Ju Lee and Jungho Yoon. 2010. Nonlinear image upsampling method based on radial basis function interpolation. IEEE Trans. Image Processing 19, 10 (2010), 2682--2692.
36. Brian Leung, Gwanggil Jeon, and Eric Dubois. 2011. Least-squares luma-chroma demultiplexing algorithm for Bayer demosaicking. IEEE Trans. Image Processing 20, 7 (2011), 1885--1894.
37. Tzu-Mao Li, Michaël Gharbi, Andrew Adams, Frédo Durand, and Jonathan Ragan-Kelley. 2018. Differentiable programming for image processing and deep learning in Halide. ACM Trans. Graph. (Proc. SIGGRAPH) 37, 4 (2018), 139:1--139:13.
38. Xin Li, Bahadir Gunturk, and Lei Zhang. 2008. Image demosaicing: A systematic survey. In Visual Communications and Image Processing 2008, Vol. 6822. 68221J.
39. Ce Liu and Deqing Sun. 2011. A Bayesian approach to adaptive video super resolution. In Proc. CVPR. IEEE, 209--216.
40. Xiao-Ming Lu, Hari Krovi, Ranjith Nair, Saikat Guha, and Jeffrey H Shapiro. 2018. Quantum-optimal detection of one-versus-two incoherent optical sources with arbitrary separation. arXiv preprint arXiv:1802.02300 (2018).
41. Bruce D Lucas and Takeo Kanade. 1981. An iterative image registration technique with an application to stereo vision. (1981).
42. Rastislav Lukac and Konstantinos N Plataniotis. 2004. Normalized color-ratio modeling for CFA interpolation. IEEE Trans. Consumer Electronics 50, 2 (2004), 737--745.
43. Hugh Malan. 2012. Real-Time Global Illumination and Reflections in Dust 514. In ACM SIGGRAPH Courses.
44. John Marshall and E Geoffrey Walsh. 1956. Physiological tremor. Journal of neurology, neurosurgery, and psychiatry 19, 4 (1956), 260.
45. Ben Mildenhall, Jonathan T Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, and Robert Carroll. 2018. Burst denoising with kernel prediction networks. In Proc. CVPR. 2502--2510.
46. Yusuke Monno, Daisuke Kiku, Masayuki Tanaka, and Masatoshi Okutomi. 2015. Adaptive residual interpolation for color image demosaicking. In Proc. IEEE ICIP. 3861--3865.
47. Matthias Müller, Barbara Solenthaler, Richard Keiser, and Markus Gross. 2005. Particle-based fluid-fluid interaction. In Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation. 237--244.
48. NIH. 2018. Tremor Fact. www.ninds.nih.gov/Disorders/Patient-Caregiver-Education/Fact-Sheets/Tremor-Fact-Sheet. {Online; accessed 29-Nov-2018}.
49. Harry Nyquist. 1928. Certain topics in telegraph transmission theory. Transactions of the American Institute of Electrical Engineers 47, 2 (1928), 617--644.
50. Athanasios Papoulis. 1977. Generalized sampling expansion. IEEE Trans. Circuits and Systems 24, 11 (1977), 652--654.
51. Cameron N Riviere, R Scott Rader, and Nitish V Thakor. 1998. Adaptive cancelling of physiological tremor for improved precision in microsurgery. IEEE Trans. Biomedical Engineering 45, 7 (1998), 839--846.
52. Dirk Robinson and Peyman Milanfar. 2004. Fundamental performance limits in image registration. IEEE Trans. Image Processing 13, 9 (2004), 1185--1199.
53. Dirk Robinson and Peyman Milanfar. 2006. Statistical performance analysis of super-resolution. IEEE Trans. Image Processing 15, 6 (2006), 1413--1428.
54. Yaniv Romano, John Isidoro, and Peyman Milanfar. 2017. RAISR: Rapid and accurate image super resolution. IEEE Trans. Computational Imaging 3, 1 (2017), 110--125.
55. Mehdi S. M. Sajjadi, Raviteja Vemulapalli, and Matthew Brown. 2018. Frame-recurrent video super-resolution. In Proc. CVPR. 6626--6634.
56. Marco Salvi. 2016. An excursion in temporal super sampling. GDC2016 From the Lab Bench: Real-Time Rendering Advances from NVIDIA Research (2016).
57. E. A. Schäfer. 1886. On the rhythm of muscular response to volitional impulses in man. The Journal of Physiology 7, 2 (1886), 111--117.
58. Morteza Shahram and Peyman Milanfar. 2006. Statistical and information-theoretic analysis of resolution in imaging. IEEE Trans. Information Theory 52, 8 (2006), 3411--3437.
59. Masao Shimiziu and Masatoshi Okutomi. 2005. Sub-pixel estimation error cancellation on area-based matching. IJCV 63 (2005), 207--224. Issue 3.
60. Tiago Sousa. 2013. Graphics Gems CryENGINE3. In ACM SIGGRAPH Courses.
61. Molly M Sturman, David E Vaillancourt, and Daniel M Corcos. 2005. Effects of aging on the regularity of physiological tremor. Journal of Neurophysiology 93, 6 (2005), 3064--3074.
62. H Takeda, S Farsiu, and P Milanfar. 2006. Robust kernel regression for restoration and reconstruction of images from sparse noisy data. Proc. IEEE ICIP (2006), 1257--1260.
63. H Takeda, S Farsiu, and P Milanfar. 2007. Kernel regression for image processing and reconstruction. IEEE Trans. Image Processing 16, 2 (2007), 349.
64. Hanlin Tan, Xiangrong Zeng, Shiming Lai, Yu Liu, and Maojun Zhang. 2017. Joint demosaicing and denoising of noisy bayer images with ADMM. In Proc. IEEE ICIP. 2951--2955.
65. R.Y. Tsai and T.S. Huang. 1984. Multiframe image restoration and registration. Advance Computer Visual and Image Processing 1 (1984), 317--339.
66. Patrick Vandewalle, Karim Krichane, David Alleysson, and Sabine Süsstrunk. 2007. Joint demosaicing and super-resolution imaging from a set of unregistered aliased images. In Digital Photography III, Vol. 6502. International Society for Optics and Photonics, 65020A.
67. Neal Wadhwa, Rahul Garg, David E Jacobs, Bryan E Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attias, Jonathan T Barron, Yael Pritch, and Marc Levoy. 2018. Synthetic depth-of-field with a single-camera mobile phone. ACM TOG 37, 4 (2018), 64.
68. Hermann Weyl. 1910. Über die Gibbs'sche Erscheinung und verwandte Konvergenzphänomene. Rendiconti del Circolo Matematico di Palermo (1884--1940) 30, 1 (01 Dec 1910), 377--407.
69. Bartlomiej Wronski and Peyman Milanfar. 2018. See Better and Further with Super Res Zoom on the Pixel 3. https://ai.googleblog.com/2018/10/see-better-and-further-with-super-res.html.
70. Xiaolin Wu and Lei Zhang. 2006. Temporal color video demosaicking via motion estimation and data fusion. IEEE Trans. Circuits and Systems for Video Technology 16 (2006). Issue 2.
71. J Yen. 1956. On nonuniform sampling of bandwidth-limited signals. IRE Trans. Circuit Theory 3, 4 (1956), 251--257.
72. Jihun Yu and Greg Turk. 2013. Reconstructing surfaces of particle-based fluids using anisotropic kernels. ACM TOG 32, 1 (2013), 5.
73. Lei Zhang, Xiaolin Wu, Antoni Buades, and Xin Li. 2011. Color demosaicking bylocal directional interpolation and nonlocal adaptive thresholding. Journal of Electronic imaging 20, 2 (2011), 023016.

Index Terms

• Computing methodologies

  • Computer graphics

    • Image manipulation

      • Computational photography

      • Image processing