Wei-Sheng Lai
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Motion blur of fast-moving subjects is a longstanding problem in photography and very common on mobile phones due to limited light collection efficiency, particularly in low-light conditions. While we have witnessed great progress in image deblurring in recent years, most methods require significant computational power and have limitations in processing high-resolution photos with severe local motions. To this end, we develop a novel face deblurring system based on the dual camera fusion technique for mobile phones. The system detects subject motion to dynamically enable a reference camera, e.g., ultrawide angle camera commonly available on recent premium phones, and captures an auxiliary photo with faster shutter settings. While the main shot is low noise but blurry (Figure 1(a)), the reference shot is sharp but noisy (Figure 1(b)). We learn ML models to align and fuse these two shots and output a clear photo without motion blur (Figure 1(c)). Our algorithm runs efficiently on Google Pixel 6, which takes 463 ms overhead per shot. Our experiments demonstrate the advantage and robustness of our system against alternative single-image, multi-frame, face-specific, and video deblurring algorithms as well as commercial products. To the best of our knowledge, our work is the first mobile solution for face motion deblurring that works reliably and robustly over thousands of images in diverse motion and lighting conditions.
Supplemental Material
Available for Download
- 2021. Android Camera2 API. https://developer.android.com/guide/topics/media/camera.Google Scholar
- Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-Scale Machine Learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). 265--283.Google ScholarDigital Library
- Miika Aittala and Frédo Durand. 2018. Burst image deblurring using permutation invariant convolutional neural networks. In ECCV. 731--747.Google Scholar
- Giacomo Boracchi and Alessandro Foi. 2012. Modeling the performance of image restoration from motion blur. IEEE TIP 21, 8 (2012), 3502--3517.Google Scholar
- Daniel J Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black. 2012. A naturalistic open source movie for optical flow evaluation. In ECCV. 611--625.Google Scholar
- Meng Chang, Huajun Feng, Zhihai Xu, and Qi Li. 2021. Low-light image restoration with short-and long-exposure raw pairs. IEEE Transactions on Multimedia.Google Scholar
- Qifeng Chen and Vladlen Koltun. 2017. Photographic image synthesis with cascaded refinement networks. In ICCV. 1511--1520.Google Scholar
- Sunghyun Cho and Seungyong Lee. 2009. Fast motion deblurring. In ACM TOG. 1--8.Google Scholar
- Sunghyun Cho, Jue Wang, and Seungyong Lee. 2012. Video deblurring for hand-held cameras using patch-based synthesis. ACM SIGGRAPH 31, 4.Google ScholarDigital Library
- Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung, and Sung-Jea Ko. 2021. Rethinking coarse-to-fine approach in single image deblurring. In ICCV. 4641--4650.Google Scholar
- Mauricio Delbracio, Ignacio Garcia-Dorado, Sungjoon Choi, Damien Kelly, and Peyman Milanfar. 2021. Polyblur: Removing mild blur by polynomial reblurring. IEEE Transactions on Computational Imaging 7, 837--848.Google ScholarCross Ref
- Mauricio Delbracio and Guillermo Sapiro. 2015a. Burst deblurring: Removing camera shake through Fourier burst accumulation. In CVPR. 2385--2393.Google Scholar
- Mauricio Delbracio and Guillermo Sapiro. 2015b. Hand-held video deblurring via efficient fourier aggregation. Transactions on Computational Imaging 1, 4.Google Scholar
- Senyou Deng, Wenqi Ren, Yanyang Yan, Tao Wang, Fenglong Song, and Xiaochun Cao. 2021. Multi-Scale Separable Network for Ultra-High-Definition Video Deblurring. In CVPR. 14030--14039.Google Scholar
- Henry Dietz and Paul Eberhart. 2019. Shuttering methods and the artifacts they produce. Electronic Imaging 2019, 4.Google Scholar
- Jiangxin Dong, Jinshan Pan, Zhixun Su, and Ming-Hsuan Yang. 2017. Blind image deblurring with outlier handling. In ICCV. 2478--2486.Google Scholar
- Jana Ehmann, Lun-Cheng Chu, Sung-Fang Tsai, and Chia-Kai Liang. 2018. Real-time video denoising on mobile phones. In ICIP. 505--509.Google Scholar
- Rob Fergus, Barun Singh, Aaron Hertzmann, Sam T Roweis, and William T Freeman. 2006. Removing camera shake from a single photograph. In ACM SIGGRAPH. 787--794.Google Scholar
- Alessandro Foi, Mejdi Trimeche, Vladimir Katkovnik, and Karen Egiazarian. 2008.Google Scholar
- Practical Poissonian-Gaussian noise modeling and fitting for single-image raw-data. IEEE TIP 17, 10 (2008), 1737--1754.Google Scholar
- Hongyun Gao, Xin Tao, Xiaoyong Shen, and Jiaya Jia. 2019. Dynamic scene deblurring with parameter selective sharing and nested skip connections. In CVPR. 3848--3856.Google Scholar
- Amnon Geifman. 2020. The Correct Way to Measure Inference Time of Deep Neural Networks. https://deci.ai/resources/blog/measure-inference-time-deep-neural-networks/.Google Scholar
- Dong Gong, Jie Yang, Lingqiao Liu, Yanning Zhang, Ian Reid, Chunhua Shen, Anton Van Den Hengel, and Qinfeng Shi. 2017. From motion blur to motion flow: A deep learning solution for removing heterogeneous motion blur. In CVPR. 2319--2328.Google Scholar
- Chunzhi Gu, Xuequan Lu, Ying He, and Chao Zhang. 2020. Blur removal via blurrednoisy image pair. IEEE TIP 30, 345--359.Google Scholar
- Yoav Hacohen, Eli Shechtman, and Dani Lischinski. 2013. Deblurring by example using dense correspondence. In ICCV. 2384--2391.Google Scholar
- Samuel W Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM SIGGRAPH 35, 6.Google ScholarDigital Library
- Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861.Google Scholar
- Xiaobin Hu, Wenqi Ren, Kaicheng Yu, Kaihao Zhang, Xiaochun Cao, Wei Liu, and Bjoern Menze. 2021. Pyramid architecture search for real-time image deblurring. In ICCV. 4298--4307.Google Scholar
- Zhe Hu, Sunghyun Cho, Jue Wang, and Ming-Hsuan Yang. 2014. Deblurring low-light images with light streaks. In ICCV. 3382--3389.Google Scholar
- Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In CVPR. 4700--4708.Google Scholar
- Tae Hyun Kim and Kyoung Mu Lee. 2015. Generalized video deblurring for dynamic scenes. In CVPR. 5426--5434.Google Scholar
- Tae Hyun Kim, Kyoung Mu Lee, Bernhard Scholkopf, and Michael Hirsch. 2017. Online video deblurring via dynamic temporal blending network. In ICCV. 4038--4047.Google Scholar
- Adam Kaufman and Raanan Fattal. 2020. Deblurring using analysis-synthesis networks pair. In CVPR. 5811--5820.Google Scholar
- Tae Hyun Kim, Mehdi SM Sajjadi, Michael Hirsch, and Bernhard Scholkopf. 2018. Spatio-temporal transformer network for video restoration. In ECCV. 106--122.Google Scholar
- Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980.Google Scholar
- Jaihyun Koh, Jangho Lee, and Sungroh Yoon. 2021. Single-image deblurring with neural networks: A comparative survey. CVIU 203.Google Scholar
- Dilip Krishnan, Terence Tay, and Rob Fergus. 2011. Blind deconvolution using a normalized sparsity measure. In CVPR. 233--240.Google Scholar
- Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiří Matas. 2018. Deblurgan: Blind motion deblurring using conditional adversarial networks. In CVPR. 8183--8192.Google Scholar
- Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. 2019. DeblurGANv2: Deblurring (orders-of-magnitude) faster and better. In ICCV. 8878--8887.Google Scholar
- Wei-Sheng Lai, Jian-Jiun Ding, Yen-Yu Lin, and Yung-Yu Chuang. 2015. Blur kernel estimation using normalized color-line prior. In ICCV. 64--72.Google Scholar
- Lerenhan Li, Jinshan Pan, Wei-Sheng Lai, Changxin Gao, Nong Sang, and Ming-Hsuan Yang. 2018. Learning a discriminative prior for blind image deblurring. In CVPR. 6616--6625.Google Scholar
- Lerenhan Li, Jinshan Pan, Wei-Sheng Lai, Changxin Gao, Nong Sang, and Ming-Hsuan Yang. 2020. Dynamic scene deblurring by depth guided model. IEEE TIP 29, 5273--5288.Google Scholar
- Orly Liba, Kiran Murthy, Yun-Ta Tsai, Tim Brooks, Tianfan Xue, Nikhil Karnad, Qiurui He, Jonathan T Barron, Dillon Sharlet, Ryan Geiss, et al. 2019. Handheld mobile photography in very low light. ACM SIGGRAPH 38, 6.Google ScholarDigital Library
- Songnan Lin, Jiawei Zhang, Jinshan Pan, Yicun Liu, Yongtian Wang, Jing Chen, and Jimmy Ren. 2020. Learning to deblur face images via sketch synthesis. In AAAI, Vol. 34.Google ScholarCross Ref
- Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. 2016. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In CVPR. 4040--4048.Google Scholar
- Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR. 3883--3891.Google Scholar
- Seungjun Nah, Sanghyun Son, and Kyoung Mu Lee. 2019. Recurrent neural networks with intra-frame iterations for video deblurring. In CVPR. 8102--8111.Google Scholar
- Jinshan Pan, Zhe Hu, Zhixun Su, and Ming-Hsuan Yang. 2014. Deblurring face images with exemplars. In ECCV. 47--62.Google Scholar
- Jinshan Pan, Zhouchen Lin, Zhixun Su, and Ming-Hsuan Yang. 2016a. Robust kernel estimation with outliers handling for image deblurring. In ICCV. 2800--2808.Google Scholar
- Jinshan Pan, Deqing Sun, Hanspeter Pfister, and Ming-Hsuan Yang. 2016b. Blind image deblurring using dark channel prior. In CVPR. 1628--1636.Google Scholar
- Tobias Plotz and Stefan Roth. 2017. Benchmarking denoising algorithms with real photographs. In CVPR. 1586--1595.Google Scholar
- Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter Shirley. 2001. Color transfer between images. IEEE Computer graphics and applications 21, 5 (2001), 34--41.Google ScholarDigital Library
- Wenqi Ren, Xiaochun Cao, Jinshan Pan, Xiaojie Guo, Wangmeng Zuo, and Ming-Hsuan Yang. 2016. Image deblurring via enhanced low-rank prior. IEEE TIP 25, 7.Google Scholar
- Wenqi Ren, Jinshan Pan, Xiaochun Cao, and Ming-Hsuan Yang. 2017. Video deblurring via semantic segmentation and pixel-wise non-linear kernel. In ICCV. 1077--1085.Google Scholar
- Wenqi Ren, Jiaolong Yang, Senyou Deng, David Wipf, Xiaochun Cao, and Xin Tong. 2019. Face video deblurring using 3d facial priors. In CVPR. 9388--9397.Google Scholar
- Jaesung Rim, Haeyun Lee, Jucheol Won, and Sunghyun Cho. 2020. Real-world blur dataset for learning and benchmarking deblurring algorithms. In ECCV. 184--201.Google Scholar
- Qi Shan, Jiaya Jia, and Aseem Agarwala. 2008. High-quality motion deblurring from a single image. ACM SIGGRAPH 27, 3.Google ScholarDigital Library
- Ziyi Shen, Wei-Sheng Lai, Tingfa Xu, Jan Kautz, and Ming-Hsuan Yang. 2018. Deep semantic face deblurring. In CVPR. 8260--8269.Google Scholar
- Ziyi Shen, Wei-Sheng Lai, Tingfa Xu, Jan Kautz, and Ming-Hsuan Yang. 2020. Exploiting semantics for face image deblurring. IJCV 128, 7.Google ScholarCross Ref
- Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556.Google Scholar
- Hyeongseok Son, Junyong Lee, Jonghyeop Lee, Sunghyun Cho, and Seungyong Lee. 2021. Recurrent video deblurring with blur-invariant motion estimation and pixel volumes. ACM SIGGRAPH 40, 5.Google ScholarDigital Library
- Shuochen Su, Mauricio Delbracio, Jue Wang, Guillermo Sapiro, Wolfgang Heidrich, and Oliver Wang. 2017. Deep video deblurring for hand-held cameras. In CVPR. 1279--1288.Google Scholar
- Maitreya Suin, Kuldeep Purohit, and AN Rajagopalan. 2020. Spatially-attentive patchhierarchical network for adaptive motion deblurring. In CVPR. 3606--3615.Google Scholar
- Deqing Sun, Daniel Vlasic, Charles Herrmann, Varun Jampani, Michael Krainin, Huiwen Chang, Ramin Zabih, William T Freeman, and Ce Liu. 2021. AutoFlow: Learning a Better Training Set for Optical Flow. In CVPR. 10093--10102.Google Scholar
- Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. 2018. PWC-net: CNNs for optical flow using pyramid, warping, and cost volume. In CVPR. 8934--8943.Google Scholar
- Jian Sun, Wenfei Cao, Zongben Xu, and Jean Ponce. 2015. Learning a convolutional neural network for non-uniform motion blur removal. In ICCV. 769--777.Google Scholar
- Libin Sun, Sunghyun Cho, Jue Wang, and James Hays. 2013. Edge-based blur kernel estimation using patch priors. In ICCP. 1--8.Google Scholar
- Hossein Talebi and Peyman Milanfar. 2018. NIMA: Neural image assessment. IEEE TIP 27, 8.Google Scholar
- Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. 2018. Scale-recurrent network for deep image deblurring. In CVPR. 8174--8182.Google Scholar
- Zachary Teed and Jia Deng. 2020. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV. 402--419.Google Scholar
- Neal Wadhwa, Rahul Garg, David E Jacobs, Bryan E Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attias, Jonathan T Barron, Yael Pritch, and Marc Levoy. 2018. Synthetic depth-of-field with a single-camera mobile phone. ACM SIGGRAPH 37, 4.Google ScholarDigital Library
- Xintao Wang, Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy. 2019. EDVR: Video restoration with enhanced deformable convolutional networks. In CVPR.Google Scholar
- Oliver Whyte, Josef Sivic, and Andrew Zisserman. 2014. Deblurring shaken and partially saturated images. IJCV 110, 2.Google ScholarDigital Library
- Li Xu and Jiaya Jia. 2010. Two-phase kernel estimation for robust motion deblurring. In ECCV. 157--170.Google Scholar
- Li Xu, Shicheng Zheng, and Jiaya Jia. 2013. Unnatural L0 sparse representation for natural image deblurring. In CVPR. 1107--1114.Google Scholar
- Xiangyu Xu, Deqing Sun, Jinshan Pan, Yujin Zhang, Hanspeter Pfister, and Ming-Hsuan Yang. 2017. Learning to super-resolve blurry face and text images. In ICCV. 251--260.Google Scholar
- Liuge Yang and Hui Ji. 2019. A variational EM framework with adaptive edge selection for blind motion deblurring. In CVPR. 10167--10176.Google Scholar
- Rajeev Yasarla, Federico Perazzi, and Vishal M Patel. 2020. Deblurring face images using uncertainty guided multi-stream semantic networks. IEEE TIP 29 (2020), 6251--6263.Google Scholar
- Amir Yazdanbakhsh, Kiran Seshadri, Berkin Akin, James Laudon, and Ravi Narayanaswami. 2021. An evaluation of edge TPU accelerators for convolutional neural networks. arXiv:2102.10423.Google Scholar
- Lu Yuan, Jian Sun, Long Quan, and Heung-Yeung Shum. 2007. Image deblurring with blurred/noisy image pairs. In ACM SIGGRAPH.Google Scholar
- Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. 2021. Multi-stage progressive image restoration. In CVPR. 14821--14831.Google Scholar
- Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. 2019. Deep stacked hierarchical multi-patch network for image deblurring. In CVPR. 5978--5986.Google Scholar
- Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Bjorn Stenger, Wei Liu, and Hongdong Li. 2020. Deblurring by realistic blurring. In CVPR. 2737--2746.Google Scholar
- Shangchen Zhou, Jiawei Zhang, Jinshan Pan, Haozhe Xie, Wangmeng Zuo, and Jimmy Ren. 2019. Spatio-temporal filter adaptive network for video deblurring. In CVPR. 2482--2491.Google Scholar
- Shaojie Zhuo, Dong Guo, and Terence Sim. 2010. Robust flash deblurring. In CVPR. 2440--2447.Google Scholar
Index Terms
Face deblurring using dual camera fusion on mobile phones
Recommendations
Joint Face Hallucination and Deblurring via Structure Generation and Detail Enhancement
We address the problem of restoring a high-resolution face image from a blurry low-resolution input. This problem is difficult as super-resolution and deblurring need to be tackled simultaneously. Moreover, existing algorithms cannot handle face images ...
Motion Deblurring of Faces
Face analysis lies at the heart of computer vision with remarkable progress in the past decades. Face recognition and tracking are tackled by building invariance to fundamental modes of variation such as illumination, 3D pose. A much less standing mode ...
Accurate motion deblurring using camera motion tracking and scene depth
WACV '13: Proceedings of the 2013 IEEE Workshop on Applications of Computer Vision (WACV)In this paper, we propose an estimation algorithm for spatially-variant blur due to camera motion. To estimate the most accurate latent image, we integrated depth sensor (Microsoft Kinect) and IMU sensor with the camera. The joint analysis of the blurry ...
Comments