High dynamic range and super-resolution from raw image bursts

Abstract

Photographs captured by smartphones and mid-range cameras have limited spatial resolution and dynamic range, with noisy response in underexposed regions and color artefacts in saturated areas. This paper introduces the first approach (to the best of our knowledge) to the reconstruction of highresolution, high-dynamic range color images from raw photographic bursts captured by a handheld camera with exposure bracketing. This method uses a physically-accurate model of image formation to combine an iterative optimization algorithm for solving the corresponding inverse problem with a learned image representation for robust alignment and a learned natural image prior. The proposed algorithm is fast, with low memory requirements compared to state-of-the-art learning-based approaches to image restoration, and features that are learned end to end from synthetic yet realistic data. Extensive experiments demonstrate its excellent performance with super-resolution factors of up to ×4 on real photographs taken in the wild with hand-held cameras, and high robustness to low-light conditions, noise, camera shake, and moderate object motion.

References

1. Cecilia Aguerrebere, Julie Delon, Yann Gousseau, and Pablo Musé. 2014. Best Algorithms for HDR Image Generation. A Study of Performance Bounds. SIAM Journal on Imaging Science 7, 1 (2014), 1--34.
2. Paul E. Anuta. 1970. Spatial Registration of Multispectral and Multitemporal Digital Imagery Using Fast Fourier Transform Techniques. IEEE Transactions on Geoscience eletronics 8, 4 (1970), 353--368.
3. Tunç Ozan Aydin, Rafal Mantiuk, and Hans-Peter Seidel. 2008. Extending quality metrics to full luminance range images. In Proceedings of Human Vision and Electronic Imaging (SPIE Proceedings), Bernice E. Rogowitz and Thrasyvoulos N. Pappas (Eds.), Vol. 6806. SPIE, 68060B.
4. Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. 2018. Automatic differentiation in machine learning: a survey. Journal of Machine Learning Research (JMLR) 18 (2018), 1--43.
5. Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. 2021a. Deep Burst Super-Resolution. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 9209--9218.
6. Goutam Bhat, Martin Danelljan, Fisher Yu, Luc Van Gool, and Radu Timofte. 2021b. Deep Reparametrization of Multi-Frame Super-Resolution and Denoising. (2021), 2460--2470.
7. Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T. Barron. 2019. Unprocessing Images for Learned Raw Denoising. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 11036--11045.
8. Che-Han Chang, Chun-Nan Chou, and Edward Y Chang. 2017. CLKN: Cascaded lucaskanade networks for image alignment. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 2213--2221.
9. Jongseong Choi, Min Kyu Park, and Moon Gi Kang. 2009. High Dynamic Range Image Reconstruction with Spatial Resolution Enhancement. Computer Journal 52, 1 (2009), 114--125.
10. Roger N. Clark. 2006. Digital Camera Reviews and Sensor Performance Summary. "https://clarkvision.com/articles/digital.sensor.performance.summary/".
11. Ryan Dahl, Mohammad Norouzi, and Jonathon Shlens. 2017. Pixel Recursive Super Resolution. In Proceedings of the International Conference on Computer Vision (ICCV).
12. Paul E. Debevec and Jitendra Malik. 1997. Recovering high dynamic range radiance maps from photographs. In SIGGRAPH. ACM, 369--378.
13. Xin Deng, Yutong Zhang, Mai Xu, Shuhang Gu, and Yiping Duan. 2021. Deep Coupled Feedback Network for Joint Exposure Fusion and Image Super-Resolution. IEEE Transactions on Image Processing (TIP) 30 (2021), 3098--3112.
14. Akshay Dudhane, Syed Waqas Zamir, Salman Khan, Fahad Khan, and Ming-Hsuan Yang. 2021. Burst Image Restoration and Enhancement. arXiv preprint arXiv:2110.03680 (2021).
15. Gabriel Eilertsen, Saghi Hajisharif, Param Hanji, Apostolia Tsirikoglou, Rafal K. Mantiuk, and Jonas Unger. 2021. How to cheat with metrics in single-image HDR reconstruction. In Proceedings of the workshops of the International Conference on Computer Vision (ICCVW). 3981--3990.
16. Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafal K. Mantiuk, and Jonas Unger. 2017. HDR image reconstruction from a single exposure using deep CNNs. ACM Transactions on Graphics 36, 6 (2017), 178:1--178:15.
17. Yuki Endo, Yoshihiro Kanamori, and Jun Mitani. 2017. Deep reverse tone mapping. ACM Transactions on Graphics (ToG) 36, 6 (2017), 177:1--177:10.
18. Manfred Ernst and Bartlomiej Wronski. 2021. HDR+ with Bracketing on Pixel Phones. "https://ai.googleblog.com/2021/04/hdr-with-bracketing-on-pixel-phones.html".
19. Sina Farsiu, Michael Elad, and Peyman Milanfar. 2006. Multiframe demosaicing and super-resolution of color images. IEEE Transactions on Image Processing (TIP) 15, 1 (2006), 141--159.
20. Alessandro Foi, Mejdi Trimeche, Vladimir Katkovnik, and Karen O. Egiazarian. 2008. Practical Poissonian-Gaussian Noise Modeling and Fitting for Single-Image Raw-Data. IEEE Transactions on Image Processing (TIP) 17, 10 (2008), 1737--1754.
21. Orazio Gallo, Marius Tico, Roberto Manduchi, Natasha Gelfand, and Kari Pulli. 2012. Metering for Exposure Stacks. Computer Graphics Forum 31, 2 (2012), 479--488.
22. Orazio Gallo, Alejandro J. Troccoli, Jun Hu, Kari Pulli, and Jan Kautz. 2015. Locally non-rigid registration for mobile HDR photography. In (CVPRW). IEEE Computer Society, 48--55.
23. Donald Geman and Chengda Yang. 1995. Nonlinear image recovery with half-quadratic regularization. IEEE Transactions on Image Processing (TIP) 5, 7 (1995), 932--946.
24. Miguel Granados, Boris Ajdin, Michael Wand, Christian Theobalt, Hans-Peter Seidel, and Hendrik P. A. Lensch. 2010. Optimal HDR reconstruction with linear digital cameras. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, 215--222.
25. Bahadir K. Gunturk and Murat Gevrekci. 2006. High-resolution image reconstruction from multiple differently exposed images. IEEE Signal Processing Letters 13, 4 (2006), 197--200.
26. Param Hanji, Fangcheng Zhong, and Rafal K. Mantiuk. 2020. Noise-Aware Merging of High Dynamic Range Image Stacks Without Camera Calibration. In Proceedings of the workshops of the European Conference on Computer Vision (ECCVW). 376--391.
27. Samuel W. Hasinoff, Frédo Durand, and William T. Freeman. 2010. Noise-optimal capture for high dynamic range photography. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 553--560.
28. Samuel W. Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T. Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Transactions on Graphics (ToG) 35, 6 (2016), 192:1--192:12.
29. Felix Heide, Markus Steinberger, Yun-Ta Tsai, Mushfiqur Rouf, Dawid Pajak, Dikpal Reddy, Orazio Gallo, Jing Liu abd Wolfgang Heidrich, Karen Egiazarian, Jan Kautz, and Kari Pulli. 2014. FlexISP: A flexible camera image processing framework. ACM Transactions on Graphics (ToG) 33, 6 (2014), 231:1--231:13.
30. Andrey Ignatov, Luc Van Gool, and Radu Timofte. 2020. Replacing mobile camera isp with a single deep learning model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 536--537.
31. Nima Khademi Kalantari and Ravi Ramamoorthi. 2017. Deep high dynamic range imaging of dynamic scenes. ACM Transactions on Graphics (ToG) 36, 4 (2017), 144:1--144:12.
32. Soo Ye Kim, Jihyong Oh, and Munchurl Kim. 2019. Deep SR-ITM: Joint Learning of Super-Resolution and Inverse Tone-Mapping for 4K UHD HDR Applications. In Proceedings of the International Conference on Computer Vision (ICCV). 3116--3125.
33. Bruno Lecouat, Jean Ponce, and Julien Mairal. 2021. Lucas-Kanade Reloaded: End-to-End Super-Resolution from Raw Image Bursts. In Proceedings of the International Conference on Computer Vision (ICCV).
34. Anat Levin, Robert Fergus, Frédo Durand, and William T. Freeman. 2007. Image and depth from a conventional camera with a coded aperture. ACM Transactions on Graphics (ToG) 26, 3 (2007), 70.
35. Orly Liba, Kiran Murthy, Yun-Ta Tsai, Tim Brooks, Tianfan Xue, Nikhil Karnad, Qiurui He, Jonathan T. Barron, Dillon Sharlet, Ryan Geiss, Samuel W. Hasinoff, Yael Pritch, and Marc Levoy. 2019. Handheld mobile photography in very low light. ACM Transactions on Graphics (ToG) 38, 6 (2019), 164:1--164:16.
36. Yu-Lun Liu, Wei-Sheng Lai, Yu-Sheng Chen, Yi-Lung Kao, Ming-Hsuan Yang, Yung-Yu Chuang, and Jia-Bin Huang. 2020. Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 1648--1657.
37. Bruce D. Lucas and Takeo Kanade. 1981. An Iterative Image Registration Technique with an Application to Stereo Vision. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). 674--679.
38. Guillermo Luijk. 2007. Dcraw tutorial. "http://guillermoluijk.com/tutorial/dcraw/index_en.htm".
39. Ziwei Luo, Lei Yu, Xuan Mo, Youwei Li, Lanpeng Jia, Haoqiang Fan, Jian Sun, and Shuaicheng Liu. 2021. EBSR: Feature enhanced burst super-resolution with deformable alignment. In Proceedings of the Conference on Computer Vision and Pattern Recognition Workshops. 471--478.
40. Kede Ma, Hui Li, Hongwei Yong, Zhou Wang, Deyu Meng, and Lei Zhang. 2017. Robust Multi-Exposure Image Fusion: A Structural Patch Decomposition Approach. IEEE Transactions on Image Processing (TIP) 26, 5 (2017), 2519--2532.
41. Henrique S. Malvar, Li-wei He, and Ross Cutler. 2004. High-quality linear interpolation for demosaicing of Bayer-patterned color images. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP). 485--488.
42. Steve Mann and Rosalind W. Picard. 1995. On being 'undigital' with digital cameras: Extending dynamic range by combining differently exposed pictures. In Proceedings of Is&T. 442--448.
43. Julien NP Martel, Lorenz K Mueller, Stephen J Carey, Piotr Dudek, and Gordon Wetzstein. 2020. Neural sensors: Learning pixel exposures for HDR imaging and video compressive sensing with programmable sensors. IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 7 (2020), 1642--1653.
44. Emil Martinec. 2008. Noise, Dynamic Range and Bit Depth in Digital SLRs. "https://photonstophotos.net/EmilMartinec/noise.html".
45. Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. 2020. PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
46. Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul Srinivasan, and Jonathan T Barron. 2021. NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images. arXiv preprint arXiv:2111.13679 (2021).
47. Antoine Monod, Julie Delon, and Thomas Veit. 2021. An Analysis and Implementation of the HDR+ Burst Denoising Method. Image Processing On Line 11 (2021), 142--169.
48. Manish Narwaria, Rafal K. Mantiuk, Matthieu Perreira Da Silva, and Patrick Le Callet. 2015. HDR-VDP-2.2: a calibrated method for objective quality prediction of high-dynamic range and standard images. Journal on Electronic Imaging 24, 1 (2015), 010501.
49. Shree K Nayar and Tomoo Mitsunaga. 2000. High dynamic range imaging: Spatially varying pixel exposures. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1. IEEE, 472--479.
50. Yuzhen Niu, Jianbin Wu, Wenxi Liu, Wenzhong Guo, and Rynson W. H. Lau. 2021. HDR-GAN: HDR Image Reconstruction From Multi-Exposed LDR Images With Large Motions. IEEE Transactions on Image Processing (TIP) 30 (2021), 3885--3896.
51. Neal Parikh and Stephen P. Boyd. 2014. Proximal Algorithms. Foundations and Trends in Optimization 1, 3 (2014), 127--239.
52. Eduardo Pérez-Pellitero, Sibi Catley-Chandar, Ales Leonardis, and Radu Timofte. 2021. NTIRE 2021 Challenge on High Dynamic Range Imaging: Dataset, Methods and Results. In CVPR Workshops. 691--700.
53. Tobias Plötz and Stefan Roth. 2017. Benchmarking Denoising Algorithms with Real Photographs. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 2750--2759.
54. Ali Ajdari Rad, Laurence Meylan, Patrick Vandewalle, and Sabine Süsstrunk. 2007. Multidimensional image enhancement from a set of unregistered and differently exposed images. In Computational Imaging (SPIE Proceedings), Vol. 6498. SPIE, 649808.
55. Erik Reinhard, Michael M. Stark, Peter Shirley, and James A. Ferwerda. 2002. Photographic tone reproduction for digital images. ACM Transactions on Graphics (ToG) 21, 3 (2002), 267--276.
56. Javier Sanchez. 2016. The inverse compositional algorithm for parametric registration. Image Processing On Line (2016).
57. Marcel Santana Santos, Tsang Ing Ren, and Nima Khademi Kalantari. 2020. Single image HDR reconstruction using a CNN with masked features and perceptual loss. ACM Transactions on Graphics (ToG) 39, 4 (2020), 80.
58. Pradeep Sen, Nima Khademi Kalantari, Maziar Yaesoubi, Soheil Darabi, Dan B. Goldman, and Eli Shechtman. 2012. Robust patch-based HDR reconstruction of dynamic scenes. ACM Transactions on Graphics (ToG) 31, 6 (2012), 203:1--203:11.
59. Ana Serrano, Felix Heide, Diego Gutierrez, Gordon Wetzstein, and Belen Masia. 2016. Convolutional sparse coding for high dynamic range imaging. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 153--163.
60. Hiroyuki Takeda, Sina Farsiu, and Peyman Milanfar. 2007. Kernel Regression for Image Processing and Reconstruction. IEEE Transactions on Image Processing (TIP) 16, 2 (2007), 349--366.
61. Yann Traonmilin and Cecilia Aguerrebere. 2014. Simultaneous High Dynamic Range and Superresolution Imaging without Regularization. SIAM Journal on Imaging Science 7, 3 (2014), 1624--1644.
62. Okan Tarhan Tursun, Ahmet Oguz Akyüz, Aykut Erdem, and Erkut Erdem. 2016. An Objective Deghosting Quality Metric for HDR Images. Computer Graphics Forum 35, 2 (2016), 139--152.
63. Patrick Vandewalle, Sabine Süsstrunk, and Martin Vetterli. 2006. A Frequency Domain Approach to Registration of Aliased Images with Application to Super-resolution. EURASIP Journal on Advances in Signal Processing 2006 (2006).
64. Subeesh Vasu, Abhijeet Shenoi, and A. N. Rajagopalan. 2018. Joint HDR and SuperResolution Imaging in Motion Blur. In Proceedings of the International Conference on Image Processing (ICIP). 2885--2889.
65. Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg. 2013. Plug-and-play priors for model based reconstruction. In Proceedings of the Global Conference on Signal and Information Processing. 945--948.
66. Greg Ward. 2003. Fast, Robust Image Registration for Compositing High Dynamic Range Photographs from Hand-Held Exposures. Journal on Graphics, GPU, & Game Tools 8, 2 (2003), 17--30.
67. Bartlomiej Wronski, Ignacio Garcia-Dorado, Manfred Ernst, Damien Kelly, Michael Krainin, Chia-Kai Liang, Marc Levoy, and Peyman Milanfar. 2019. Handheld multiframe super-resolution. ACM Transactions on Graphics (ToG) 38, 4 (2019), 28:1--28:18.
68. Shangzhe Wu, Jiarui Xu, Yu-Wing Tai, and Chi-Keung Tang. 2018. Deep High Dynamic Range Imaging with Large Foreground Motions. In Proceedings of the European Conference on Computer Vision (ECCV). 120--135.
69. Qingsen Yan, Dong Gong, Javen Qinfeng Shi, Anton van den Hengel, Chunhua Shen, Ian Reid, and Yanning Zhang. 2021. Dual-attention-guided network for ghost-free high dynamic range imaging. International Journal of Computer Vision (IJCV) (2021), 1--19.
70. Qingsen Yan, Lei Zhang, Yu Liu, Yu Zhu, Jinqiu Sun, Qinfeng Shi, and Yanning Zhang. 2020. Deep HDR Imaging via A Non-Local Network. IEEE Transactions on Image Processing (TIP) 29 (2020), 4308--4322.
71. Kai Zhang, Luc Van Gool, and Radu Timofte. 2020. Deep unfolding network for image super-resolution. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 3217--3226.
72. Henning Zimmer, Andrés Bruhn, and Joachim Weickert. 2011. Freehand HDR Imaging of Moving Scenes with Simultaneous Resolution Enhancement. Computer Graphics Forum 30, 2 (2011), 405--414.

Index Terms

• Computing methodologies

  • Computer graphics

    • Image manipulation

      • Image processing