Distortion-free wide-angle portraits on camera phones

Abstract

Photographers take wide-angle shots to enjoy expanding views, group portraits that never miss anyone, or composite subjects with spectacular scenery background. In spite of the rapid proliferation of wide-angle cameras on mobile phones, a wider field-of-view (FOV) introduces a stronger perspective distortion. Most notably, faces are stretched, squished, and skewed, to look vastly different from real-life. Correcting such distortions requires professional editing skills, as trivial manipulations can introduce other kinds of distortions. This paper introduces a new algorithm to undistort faces without affecting other parts of the photo. Given a portrait as an input, we formulate an optimization problem to create a content-aware warping mesh which locally adapts to the stereographic projection on facial regions, and seamlessly evolves to the perspective projection over the background. Our new energy function performs effectively and reliably for a large group of subjects in the photo. The proposed algorithm is fully automatic and operates at an interactive rate on the mobile platform. We demonstrate promising results on a wide range of FOVs from 70° to 120°.

References

1. Michael D Abràmoff, Paulo J Magalhàes, and Sunanda J Ram. 2004. Image processing with ImageJ. Biophotonics international 11, 7 (2004), 36--42.
2. Sameer Agarwal, Keir Mierle, and Others. 2012. Ceres Solver. http://ceres-solver.org.
3. Aseem Agarwala, Maneesh Agrawala, Michael Cohen, David Salesin, and Richard Szeliski. 2006. Photographing long scenes with multi-viewpoint panoramas. In ACM TOG, Vol 25. 853--861.
4. Dimitri P Bertsekas. 1999. Nonlinear programming.
5. Duane C Brown. 1966. Decentering distortion of lenses. Photogrammetric Engineering and Remote Sensing (1966).
6. Robert Carroll, Aseem Agarwala, and Maneesh Agrawala. 2010. Image warps for artistic perspective manipulation. In ACM TOG, Vol. 29. 127.
7. Robert Carroll, Maneesh Agrawal, and Aseem Agarwala. 2009. Optimizing content-preserving projections for wide-angle images. In ACM TOG, Vol. 28. 43.
8. Che-Han Chang and Yung-Yu Chuang. 2012. A line-structure-preserving approach to image resizing. In Proc. CVPR. 1075--1082.
9. Che-Han Chang, Chia-Kai Liang, and Yung-Yu Chuang. 2011. Content-aware display adaptation and interactive editing for stereoscopic images. IEEE TMM 13, 4 (2011), 589--601.
10. Che-Han Chang, Yoichi Sato, and Yung-Yu Chuang. 2014. Shape-preserving half-projective warps for image stitching. In Proc. CVPR. 3254--3261.
11. Yu-Sheng Chen and Yung-Yu Chuang. 2016. Natural image stitching with the global similarity prior. In Proc. ECCV. 186--201.
12. Song-Pei Du, Shi-Min Hu, and Ralph R Martin. 2013. Changing perspective in stereoscopic images. IEEE TVCG 19, 8 (2013), 1288--1297.
13. Claude E Duchon. 1979. Lanczos filtering in one and two dimensions. Journal of applied meteorology 18, 8 (1979), 1016--1022.
14. DxO. 2018. Correcting volume deformation with DxO ViewPoint. https://www.dxo.com/dxo-viewpoint/features/.
15. Ohad Fried, Eli Shechtman, Dan B Goldman, and Adam Finkelstein. 2016. Perspective-aware manipulation of portrait photos. ACM TOG 35, 4 (2016), 128.
16. Ran Gal, Olga Sorkine, and Daniel Cohen-Or. 2006. Feature-aware texturing. Rendering Techniques 2006, 17 (2006), 2.
17. GoPro. 2017. HERO6 Black Field of View (FOV) Information. https://gopro.com/help/articles/question_answer/hero6-black-field-of-view-fov-information.
18. Heng Guo, Shuaicheng Liu, Tong He, Shuyuan Zhu, Bing Zeng, and Moncef Gabbouj. 2016. Joint video stitching and stabilization from moving cameras. IEEE TIP 25, 11, 5491--5503.
19. Kaiming He, Huiwen Chang, and Jian Sun. 2013. Rectangling panoramic images via warping. ACM TOG 32, 4 (2013), 79.
20. Chintan Intwala and Aseem Agarwala. 2015. Perspective warp. US Patent 9,117,253.
21. Hamid Izadinia, Bryan C Russell, Ali Farhadi, Matthew D Hoffman, and Aaron Hertzmann. 2015. Deep classifiers from image tags in the wild. In Proc. 2015 Workshop on Community-Organized Multimodal Mining: Opportunities for Novel Solutions. 13--18.
22. Yoshihiro Kanamori, Nguyen Huu Cuong, and Tomoyuki Nishita. 2011. Local optimization of distortions in wide-angle images using moving least-squares. In Proceedings of the 27th Spring Conference on Computer Graphics. ACM, 51--56.
23. Johannes Kopf, Dani Lischinski, Oliver Deussen, Daniel Cohen-Or, and Michael Cohen. 2009. Locally adapted projections to reduce panorama distortions. In CGF, Vol. 28. 1083--1089.
24. Hyunjoon Lee, Eli Shechtman, Jue Wang, and Seungyong Lee. 2012. Automatic upright adjustment of photographs. In Proc. CVPR. 877--884.
25. Tommer Leyvand, Daniel Cohen-Or, Gideon Dror, and Dani Lischinski. 2008. Data-driven enhancement of facial attractiveness. ACM TOG 27, 3 (2008), 38.
26. Dongping Li, Kaiming He, Jian Sun, and Kun Zhou. 2015. A geodesic-preserving method for image warping. In Proc. CVPR. 213--221.
27. Feng Liu, Michael Gleicher, Hailin Jin, and Aseem Agarwala. 2009. Content-preserving warps for 3D video stabilization. In ACM TOG, Vol. 28. ACM, 44.
28. Feng Liu, Michael Gleicher, Jue Wang, Hailin Jin, and Aseem Agarwala. 2011. Subspace video stabilization. ACM TOG 30, 1 (2011), 4.
29. Darko Pavić, Volker Schönefeld, and Leif Kobbelt. 2006. Interactive image completion with perspective correction. The Visual Computer 22, 9--11 (2006), 671--681.
30. Samsung. 2017. GalaxyS7: About Shape Correction. www.samsung.com/nz/support/mobile-devices/galaxy-s7-about-shape-correction/.
31. Thomas K Sharpless, Bruno Postle, and Daniel M German. 2010. Pannini: a new projection for rendering wide angle perspective images. In Proc. 6th international conference on Computational Aesthetics in Graphics, Visualization and Imaging. 9--16.
32. John Parr Snyder. 1987. Map projections-A working manual. Vol. 1395. US Government Printing Office.
33. Mahdi Abbaspour Tehrani, Aditi Majumder, and M Gopi. 2016. Correcting perceived perspective distortions using object specific planar transformations. In Proc. ICCP.
34. TensorFlow. 2018. TensorFlow-Mobile/Lite. https://www.tensorflow.org/mobile/.
35. Kim H Veltman. 1986. Perspective, anamorphosis and vision. Marburger Jahrbuch für (1986).
36. Dhanraj Vishwanath, Ahna R Girshick, and Martin S Banks. 2005. Why pictures look right when viewed from the wrong place. Nature neuroscience 8, 10 (2005), 1401.
37. Neal Wadhwa, Rahul Garg, David Jacobs, Bryan Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attias, Jonathan Barron, Yael Pritch, and Marc Levoy. 2018. Synthetic depth-of-field with a single-camera mobile phone. ACM TOG 37, 4 (2018).
38. Yu-Shuen Wang, Chiew-Lan Tai, Olga Sorkine, and Tong-Yee Lee. 2008. Optimized scale-and-stretch for image resizing. In ACM TOG, Vol. 27. 118.
39. Jin Wei, Chen-Feng Li, Shi-Min Hu, Ralph R Martin, and Chiew-Lan Tai. 2012. Fisheye video correction. IEEE TVCG 18, 10 (2012), 1771--1783.
40. Lior Wolf, Moshe Guttmann, and Daniel Cohen-Or. 2007. Non-homogeneous content-driven video-retargeting. (2007).
41. Lihi Zelnik-Manor, Gabriele Peters, and Pietro Perona. 2005. Squaring the circle in panoramas. In Proc. ICCV, Vol. 2. 1292--1299.
42. David Zhang, Qijun Zhao, and Fangmei Chen. 2011. Quantitative analysis of human facial beauty using geometric features. Pattern Recognition 44, 4 (2011), 940--950.
43. Fang-Lue Zhang, Xian Wu, Hao-Tian Zhang, Jue Wang, and Shi-Min Hu. 2016. Robust background identification for dynamic video editing. ACM TOG 35, 6 (2016), 197.
44. Zhengyou Zhang. 2000. A flexible new technique for camera calibration. IEEE Trans. PAMI 22 (2000).
45. Denis Zorin and Alan H Barr. 1995. Correction of geometric perceptual distortions in pictures. In SIGGRAPH. 257--264.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

        • Computational photography

  • Computer graphics

    • Image manipulation

      • Image processing