Relighting humans: occlusion-aware inverse rendering for full-body human images

Abstract

Relighting of human images has various applications in image synthesis. For relighting, we must infer albedo, shape, and illumination from a human portrait. Previous techniques rely on human faces for this inference, based on spherical harmonics (SH) lighting. However, because they often ignore light occlusion, inferred shapes are biased and relit images are unnaturally bright particularly at hollowed regions such as armpits, crotches, or garment wrinkles. This paper introduces the first attempt to infer light occlusion in the SH formulation directly. Based on supervised learning using convolutional neural networks (CNNs), we infer not only an albedo map, illumination but also a light transport map that encodes occlusion as nine SH coefficients per pixel. The main difficulty in this inference is the lack of training datasets compared to unlimited variations of human portraits. Surprisingly, geometric information including occlusion can be inferred plausibly even with a small dataset of synthesized human figures, by carefully preparing the dataset so that the CNNs can exploit the data coherency. Our method accomplishes more realistic relighting than the occlusion-ignored formulation.

References

1. Miika Aittala, Timo Aila, and Jaakko Lehtinen. 2016. Reflectance modeling by neural texture synthesis. ACM Trans. Graph. 35, 4 (2016), 65:1--65:13.
2. Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. 2005. SCAPE: shape completion and animation of people. ACM Trans. Graph. 24, 3 (2005), 408--416.
3. Alexandru O. Balan, Leonid Sigal, Michael J. Black, James E. Davis, and Horst W. Haussecker. 2007. Detailed Human Shape and Pose from Images. In 2007 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2007).
4. Jonathan T. Barron and Jitendra Malik. 2015. Shape, Illumination, and Reflectance from Shading. IEEE Trans. Pattern Anal. Mach. Intell. 37, 8 (2015), 1670--1687.
5. H. G. Barrow and J. M. Tenenbaum. 1978. Recovering intrinsic scene characteristics from images. Comp. Vis. Sys. (1978).
6. Anil S. Baslamisli, Hoang-An Le, and Theo Gevers. 2018. CNN based Learning using Reflection and Retinex Models for Intrinsic Image Decomposition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018).
7. R. Basri and D. W. Jacobs. 2003. Lambertian reflectance and linear subspaces. IEEE Transactions on Pattern Analysis and Machine Intelligence 25, 2 (Feb 2003), 218--233.
8. Sean Bell, Kavita Bala, and Noah Snavely. 2014. Intrinsic images in the wild. ACM Trans. Graph. 33, 4 (2014), 159:1--159:12.
9. Volker Blanz and Thomas Vetter. 1999. A Morphable Model for the Synthesis of 3D Faces. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 1999). 187--194.
10. Nicolas Bonneel, Balazs Kovacs, Sylvain Paris, and Kavita Bala. 2017. Intrinsic Decompositions for Image Editing. Comput. Graph. Forum 36, 2 (2017), 593--609.
11. Menglei Chai, Linjie Luo, Kalyan Sunkavalli, Nathan Carr, Sunil Hadap, and Kun Zhou. 2015. High-quality hair modeling from a single portrait photo. ACM Trans. Graph. 34, 6 (2015), 204:1--204:10.
12. Manmohan Krishna Chandraker and Ravi Ramamoorthi. 2011. What an image reveals about material reflectance. In IEEE International Conference on Computer Vision (ICCV 2011). 1076--1083.
13. R. Danerek, Endri Dibra, A. Cengiz Öztireli, Remo Ziegler, and Markus H. Gross. 2017. DeepGarment: 3D Garment Shape Estimation from a Single Image. Comput. Graph. Forum 36, 2 (2017), 269--280.
14. Paul Debevec. 2004. Light Probe Image Gallery. (2004). http://www.pauldebevec.com/Probes/.
15. Paul E. Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. 2000. Acquiring the reflectance field of a human face. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 2000). 145--156.
16. Yuki Endo, Yoshihiro Kanamori, and Jun Mitani. 2017. Deep reverse tone mapping. ACM Trans. Graph. 36, 6 (2017), 177:1--177:10.
17. Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean-François Lalonde. 2017. Learning to predict indoor illumination from a single image. ACM Trans. Graph. 36, 6 (2017), 176:1--176:14.
18. Pablo Garrido, Levi Valgaerts, Chenglei Wu, and Christian Theobalt. 2013. Reconstructing detailed dynamic face geometry from monocular video. ACM Trans. Graph. 32, 6 (2013), 158:1--158:10.
19. Stamatios Georgoulis, Konstantinos Rematas, Tobias Ritschel, Efstratios Gavves, Mario Fritz, Luc Van Gool, and Tinne Tuytelaars. 2018. Reflectance and Natural Illumination from Single-Material Specular Objects Using Deep Learning. IEEE Trans. Pattern Anal. Mach. Intell. 40, 8 (2018), 1932--1947.
20. Peng Guan, Alexander Weiss, Alexandru O. Balan, and Michael J. Black. 2009. Estimating human shape and pose from a single image. In IEEE 12th International Conference on Computer Vision (ICCV 2009). 1381--1388.
21. Yannick Hold-Geoffroy, Kalyan Sunkavalli, Sunil Hadap, Emiliano Gambaretto, and Jean-François Lalonde. 2017. Deep Outdoor Illumination Estimation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). 2373--2382.
22. Berthold K. P. Horn. 1989. Shape from Shading. MIT Press, Chapter Obtaining Shape from Shading Information, 123--171.
23. Carlo Innamorati, Tobias Ritschel, Tim Weyrich, and Niloy J. Mitra. 2017. Decomposing Single Images for Layered Photo Retouching. Comput. Graph. Forum 36, 4 (2017), 15--25.
24. Micah K. Johnson and Edward H. Adelson. 2011. Shape estimation in natural illumination. In The 24th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2011). 2553--2560.
25. Ira Kemelmacher-Shlizerman and Ronen Basri. 2011. 3D Face Reconstruction from a Single Image Using a Single Reference Face Shape. IEEE Trans. Pattern Anal. Mach. Intell. 33, 2 (2011), 394--405.
26. Natasha Kholgade, Tomas Simon, Alexei A. Efros, and Yaser Sheikh. 2014. 3D object manipulation in a single photograph using stock 3D models. ACM Trans. Graph. 33, 4 (2014), 127:1--127:12.
27. Edwin H. Land and John J. McCann. 1971. Lightness and Retinex Theory. J. Opt. Soc. Am. 61, 1 (Jan 1971), 1--11.
28. Guannan Li, Chenglei Wu, Carsten Stoll, Yebin Liu, Kiran Varanasi, Qionghai Dai, and Christian Theobalt. 2013. Capturing Relightable Human Performances under General Uncontrolled Illumination. Comput. Graph. Forum 32, 2 (2013), 275--284.
29. Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2017. Modeling surface appearance from a single photograph using self-augmented convolutional neural networks. ACM Trans. Graph. 36, 4 (2017), 45:1--45:11.
30. Jorge Lopez-Moreno, Elena Garces, Sunil Hadap, Erik Reinhard, and Diego Gutierrez. 2013. Multiple Light Source Estimation in a Single Image. Comput. Graph. Forum 32, 8 (2013), 170--182.
31. Zhaoliang Lun, Matheus Gadelha, Evangelos Kalogerakis, Subhransu Maji, and Rui Wang. 2017. 3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks. In 2017 International Conference on 3D Vision (3DV 2017).
32. Takuya Narihira, Michael Maire, and Stella X. Yu. 2015. Direct Intrinsics: Learning Albedo-Shading Decomposition by Convolutional Regression. In 2015 IEEE International Conference on Computer Vision (ICCV 2015). 2992.
33. Geoffrey Oxholm and Ko Nishino. 2012. Shape and Reflectance from Natural Illumination. In 12th European Conference on Computer Vision (ECCV 2012), Proceedings, Part I. 528--541.
34. Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. 2009. A 3D Face Model for Pose and Illumination Invariant Face Recognition. In Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS 2009). 296--301.
35. Ravi Ramamoorthi and Pat Hanrahan. 2001. An efficient representation for irradiance environment maps. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2001. 497--500.
36. Andreas Schneider, Sandro Schönborn, Bernhard Egger, Lavrenti Frobeen, and Thomas Vetter. 2017. Efficient Global Illumination for Morphable Models. In IEEE International Conference on Computer Vision (ICCV 2017). 3885--3893.
37. Soumyadip Sengupta, Angjoo Kanazawa, Carlos D. Castillo, and David W. Jacobs. 2018. SfSNet: Learning Shape, Reflectance and Illuminance of Faces 'in the Wild'. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018).
38. Jian Shi, Yue Dong, Hao Su, and Stella X. Yu. 2017. Learning Non-Lambertian Object Intrinsics Across ShapeNet Categories. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). 5844--5853.
39. Zhixin Shu, Sunil Hadap, Eli Shechtman, Kalyan Sunkavalli, Sylvain Paris, and Dimitris Samaras. 2017a. Portrait Lighting Transfer Using a Mass Transport Approach. ACM Trans. Graph. 36, 4, Article 145a (Oct. 2017).
40. Zhixin Shu, Ersin Yumer, Sunil Hadap, Kalyan Sunkavalli, Eli Shechtman, and Dimitris Samaras. 2017b. Neural Face Editing with Intrinsic Image Disentangling. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). 5444--5453.
41. Peter-Pike J. Sloan, Jan Kautz, and John Snyder. 2002. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. ACM Trans. Graph. 21, 3 (2002), 527--536.
42. Ayush Tewari, Michael Zollhöfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Pérez, and Christian Theobalt. 2017. MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction. In IEEE International Conference on Computer Vision (ICCV 2017). 3735--3744.
43. Su Xue, Aseem Agarwala, Julie Dorsey, and Holly E. Rushmeier. 2012. Understanding and improving the realism of image composites. ACM Trans. Graph. 31, 4 (2012), 84:1--84:10.
44. Shugo Yamaguchi, Shunsuke Saito, Koki Nagano, Yajie Zhao, Weikai Chen, Kyle Olszewski, Shigeo Morishima, and Hao Li. 2018. High-fidelity Facial Reflectance and Geometry Inference from an Unconstrained Image. ACM Trans. Graph. 37, 4, Article 162 (July 2018), 162:1--162:14 pages.
45. Chao Zhang, Sergi Pujades, Michael J. Black, and Gerard Pons-Moll. 2017. Detailed, Accurate, Human Shape Estimation from Clothed 3D Scan Sequences. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). 5484--5493.
46. Bin Zhou, Xiaowu Chen, Qiang Fu, Kan Guo, and Ping Tan. 2013. Garment Modeling from a Single Image. Comput. Graph. Forum 32, 7 (2013), 85--91.
47. Sergey Zhukov, Andrei Iones, and Grigorij Kronin. 1998. An Ambient Light Illumination Model. In Rendering Techniques '98, Proceedings of the Eurographics Workshop. 45--56.

Index Terms

• Computing methodologies

  • Computer graphics

    • Image manipulation

      • Image-based rendering

    • Rendering