Abstract
Video-streaming services usually feature post-processing effects to replace the background. However, these often yield inconsistent lighting. Machine-learning-based relighting methods can address this problem, but, at real-time rates, are restricted to a low resolution and can result in an unrealistic skin appearance. Physically-based rendering techniques require complex skin models that can only be acquired using specialised equipment. Our method is lightweight and uses only a standard smartphone. By correcting imperfections during capture, we extract a convincing physically-based skin model. In combination with suitable acceleration techniques, we achieve real-time rates on commodity hardware.
Supplemental Material
Available for Download
Supplemental movie, appendix, image and software files for, Real-Time Relighting of Human Faces with a Low-Cost Setup
- Jean-Yves Bouguet et al. 2001. Pyramidal implementation of the affine lucas kanade feature tracker description of the algorithm. Intel corporation 5, 1--10 (2001), 4.Google Scholar
- Brian Cabral, Marc Olano, and Philip Nemec. 1999. Reflection space image based rendering. Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1999 (1999), 165--171. https://doi.org/10.1145/311535.311553Google Scholar
Digital Library
- Paul Debevec. 2012. The Light Stages and Their Applications to Photoreal Digital Actors. In SIGGRAPH Asia. Singapore.Google Scholar
- Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. 2000. Acquiring the reflectance field of a human face. Proceedings of the ACM SIGGRAPH Conference on Computer Graphics (2000), 145--156. https://doi.org/10.1145/344779.344855Google Scholar
Digital Library
- Per Einarsson, Charles-Felix Chabert, Andrew Jones, Wan-Chun Ma, Bruce Lamond, Tim Hawkins, Mark Bolas, Sebastian Sylwan, and Paul Debevec. 2006. Relighting Human Locomotion with Flowed Reflectance Fields. In Proceedings of the 17th Eurographics Conference on Rendering Techniques (Nicosia, Cyprus) (EGSR '06). Eurographics Association, Goslar, DEU, 183--194.Google Scholar
Digital Library
- Martin Fuchs, Hendrik PA Lensch, Volker Blanz, and Hans-Peter Seidel. 2007. Superresolution reflectance fields: Synthesizing images for intermediate light directions. In Computer Graphics Forum, Vol. 26. Wiley Online Library, 447--456.Google Scholar
- Graham Fyffe and Paul Debevec. 2015. Single-Shot Reflectance Measurement from Polarized Color Gradient Illumination. 2015 IEEE International Conference on Computational Photography, ICCP 2015 - Proceedings (2015). https://doi.org/10.1109/ICCPHOT.2015.7168375Google Scholar
Cross Ref
- G. Fyffe, P. Graham, B. Tunwattanapong, A. Ghosh, and P. Debevec. 2016. Near-instant capture of high-resolution facial geometry and reflectance. Computer Graphics Forum 35, 2 (2016), 353--363. https://doi.org/10.1111/cgf.12837Google Scholar
Cross Ref
- Gaurav Garg, Eino-Ville Talvala, Marc Levoy, and Hendrik P Lensch. 2006. Symmetric Photography: Exploiting Data-sparseness in Reflectance Fields. Eurographics Symposium on Rendering (EGSR) (2006), 251--262.Google Scholar
Digital Library
- Jiahao Geng, Yanlin Weng, Lvdi Wang, and Kun Zhou. 2021. Single-view facial reflectance inference with a differentiable renderer. Science China Information Sciences 64, 11 (2021), 1--17.Google Scholar
Cross Ref
- Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Debevec. 2011. Multiview Face Capture Using Polarized Spherical Gradient Illumination. In Proceedings of the 2011 SIGGRAPH Asia Conference (Hong Kong, China) (SA '11). ACM, New York, NY, USA, Article 129, 10 pages. https://doi.org/10.1145/2024156.2024163Google Scholar
Digital Library
- Paulo Gotardo, Jérémy Riviere, Derek Bradley, Abhijeet Ghosh, and Thabo Beeler. 2018. Practical dynamic facial appearance modeling and acquisition. SIGGRAPH Asia 2018 Technical Papers, SIGGRAPH Asia 2018 37, 6 (2018). https://doi.org/10.1145/3272127.3275073Google Scholar
- Jianzhu Guo, Xiangyu Zhu, and Zhen Lei. 2018. 3DDFA. https://github.com/cleardusk/3DDFA.Google Scholar
- Jianzhu Guo, Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei, and Stan Z Li. 2020. Towards Fast, Accurate and Stable 3D Dense Face Alignment. In Proceedings of the European Conference on Computer Vision (ECCV).Google Scholar
Digital Library
- Tim Hawkins, Jonathan Cohen, and Paul Debevec. 2001. A Photometric Approach to Digitizing Cultural Artifacts. In Proceedings of the 2001 Conference on Virtual Reality, Archeology, and Cultural Heritage (Glyfada, Greece) (VAST '01). ACM, New York, NY, USA, 333--342. https://doi.org/10.1145/584993.585053Google Scholar
Digital Library
- Tim Hawkins, Andreas Wenger, Chris Tchou, Andrew Gardner, Fredrik Göransson, and Paul Debevec. 2004. Animatable Facial Reflectance Fields. In Proceedings of the Fifteenth Eurographics Conference on Rendering Techniques (Norrköping, Sweden) (EGSR'04). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 309--319. https://doi.org/10.2312/EGWR/EGSR04/309-319Google Scholar
Digital Library
- Chloe Legendre, Wan Chun Ma, Rohit Pandey, Sean Fanello, Christoph Rhemann, Jason Dourgarian, Jay Busch, and Paul Debevec. 2020. Learning Illumination from Diverse Portraits. SIGGRAPH Asia 2020 Technical Communications, SA 2020 (2020). https://doi.org/10.1145/3410700.3425432 arXiv:2008.02396Google Scholar
- Jingtang Liao, Bert Buchholz, Jean-Marc Thiery, Pablo Bauszat, and Elmar Eisemann. 2017. Indoor Scene Reconstruction Using Near-light Photometric Stereo. IEEE Transactions on Image Processing 26, 3 (March 2017), 1089--1101. http://graphics.tudelft.nl/Publications-new/2017/LBTBE17 DOI: 10.1109/TIP.2016.2636661.Google Scholar
Digital Library
- Wc Ma, Tim Hawkins, and P Peers. 2007. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. Proceedings of the 18th Eurographics conference on Rendering Techniques (2007), 183--194. http://dl.acm.org/citation.cfm?id=2383873Google Scholar
Digital Library
- Tom Malzbender, Dan Gelb, and Hans Wolters. 2001. Polynomial texture maps. Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2001 (2001), 519--528. https://doi.org/10.1145/383259.383320Google Scholar
Digital Library
- Morgan McGuire. 2017. Computer Graphics Archive. https://casual-effects.com/dataGoogle Scholar
- Abhimitra Meka, Christian Haene, Rohit Pandey, Michael Zollhoefer, Sean Fanello, Graham Fyffe, Adarsh Kowdle, Xueming Yu, Jay Busch, Jason Dourgarian, Peter Denny, Sofien Bouaziz, Peter Lincoln, Matt Whalen, Geoff Harvey, Jonathan Taylor, Shahram Izadi, Andrea Tagliasacchi, Paul Debevec, Christian Theobalt, Julien Valentin, and Christoph Rhemann. 2019. Deep Reflectance Fields - High-Quality Facial Reflectance Field Inference From Color Gradient Illumination. ACM Transactions on Graphics (Proceedings SIGGRAPH) 38, 4. https://doi.org/10.1145/3306346.3323027Google Scholar
- Thomas Nestmeyer, Jean-François Lalonde, Iain Matthews, and Andreas M Lehrmann. 2020. Learning Physics-guided Face Relighting under Directional Light. In Conference on Computer Vision and Pattern Recognition. IEEE/CVF, 5123--5132.Google Scholar
Cross Ref
- Rohit Pandey, Sergio Orts Escolano, Chloe Legendre, Christian Haene, Sofien Bouaziz, Christoph Rhemann, Paul Debevec, and Sean Fanello. 2021. Total relighting: learning to relight portraits for background replacement. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1--21.Google Scholar
Digital Library
- Pieter Peers, Naoki Tamura, Wojciech Matusik, and Paul Debevec. 2007. Post-production facial performance relighting using reflectance transfer. ACM Transactions on Graphics 26, 99 (2007), 52. https://doi.org/10.1145/1239451.1239503Google Scholar
Digital Library
- Jérémy Riviere, Paulo Gotardo, Derek Bradley, Abhijeet Ghosh, and Thabo Beeler. 2020. Single-Shot High-Quality Facial Geometry and Skin Appearance Capture. ACM Trans. Graph. 39, 4, Article 81 (July 2020), 12 pages. https://doi.org/10.1145/3386569.3392464Google Scholar
Digital Library
- Soumyadip Sengupta, Angjoo Kanazawa, Carlos D Castillo, and David W Jacobs. 2018. SfSNet: Learning Shape, Reflectance and Illuminance of Facesin the Wild'. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6296--6305.Google Scholar
Cross Ref
- Jianbo Shi et al. 1994. Good features to track. In 1994 Proceedings of IEEE conference on computer vision and pattern recognition. IEEE, 593--600.Google Scholar
- Zhixin Shu, Sunil Hadap, Eli Shechtman, Kalyan Sunkavalli, Sylvain Paris, and Dimitris Samaras. 2017. Portrait Lighting Transfer Using a Mass Transport Approach. ACM Trans. Graph. 37, 1, Article 2 (Oct. 2017), 15 pages. https://doi.org/10.1145/3095816Google Scholar
- Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. 2019. First Order Motion Model for Image Animation. In Conference on Neural Information Processing Systems (NeurIPS).Google Scholar
- Manminder Singh and AS Arora. 2016. Varying illumination and pose conditions in face recognition. Procedia Computer Science 85 (2016), 691--695.Google Scholar
Cross Ref
- Tiancheng Sun, Jonathan T. Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul Debevec, and Ravi Ramamoorthi. 2019. Single Image Portrait Relighting. ACM Trans. Graph. 38, 4, Article 79 (July 2019), 12 pages. https://doi.org/10.1145/3306346.3323008Google Scholar
Digital Library
- Tiancheng Sun, Zexiang Xu, Xiuming Zhang, Sean Fanello, Christoph Rhemann, Paul Debevec, Yun Ta Tsai, Jonathan T. Barron, and Ravi Ramamoorthi. 2020. Light Stage Super-Resolution: Continuous High-Frequency Relighting. ACM Transactions on Graphics 39, 6 (2020). https://doi.org/10.1145/3414685.3417821 arXiv:2010.08888Google Scholar
Digital Library
- Paul Viola and Michael J Jones. 2004. Robust real-time face detection. International journal of computer vision 57, 2 (2004), 137--154.Google Scholar
Digital Library
- Zhibo Wang, Xin Yu, Ming Lu, Quan Wang, Chen Qian, and Feng Xu. 2020. Single image portrait relighting via explicit multiple reflectance channel modeling. ACM Transactions on Graphics 39, 6 (2020). https://doi.org/10.1145/3414685.3417824Google Scholar
Digital Library
- Tim Weyrich, Wojciech Matusik, Hanspeter Pfister, Bernd Bickel, Craig Donner, Chien Tu, Janet McAndless, Jinho Lee, Addy Ngan, Henrik Wann Jensen, and Markus Gross. 2006. Analysis of Human Faces using a Measurement-Based Skin Reflectance Model. ACM Trans. on Graphics (Proc. SIGGRAPH 2006) 25, 3 (2006), 1013--1024. https://doi.org/10.1145/1179352.1141987Google Scholar
Digital Library
- Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi Ramamoorthi. 2018. Deep image-based relighting from optimal sparse samples. ACM Transactions on Graphics 37, 4 (2018), 1--13. https://doi.org/10.1145/3197517.3201313Google Scholar
Digital Library
- Shugo Yamaguchi, Shunsuke Saito, Koki Nagano, Yajie Zhao, Weikai Chen, Kyle Olszewski, Shigeo Morishima, and Hao Li. 2018. High-fidelity facial reflectance and geometry inference from an unconstrained image. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1--14.Google Scholar
Digital Library
- Xiuming Zhang, Sean Fanello, Yun Ta Tsai, Tiancheng Sun, Tianfan Xue, Rohit Pandey, Sergio Orts-Escolano, Philip Davidson, Christoph Rhemann, Paul Debevec, Jonathan T. Barron, Ravi Ramamoorthi, and William T. Freeman. 2021. Neural Light Transport for Relighting and View Synthesis. ACM Transactions on Graphics 40, 1 (2021). https://doi.org/10.1145/3446328 arXiv:2008.03806Google Scholar
- Hao Zhou, Sunil Hadap, Kalyan Sunkavalli, and David Jacobs. 2019. Deep single-image portrait relighting. Proceedings of the IEEE International Conference on Computer Vision 2019-Octob (2019), 7193--7201. https://doi.org/10.1109/ICCV.2019.00729Google Scholar
Cross Ref
Index Terms
Real-Time Relighting of Human Faces with a Low-Cost Setup
Recommendations
Total relighting: learning to relight portraits for background replacement
We propose a novel system for portrait relighting and background replacement, which maintains high-frequency boundary details and accurately synthesizes the subject's appearance as lit by novel illumination, thereby producing realistic composite images ...
Interactive relighting with dynamic BRDFs
We present a technique for interactive relighting in which source radiance, viewing direction, and BRDFs can all be changed on the fly. In handling dynamic BRDFs, our method efficiently accounts for the effects of BRDF modification on the reflectance ...
Interactive relighting with dynamic BRDFs
SIGGRAPH '07: ACM SIGGRAPH 2007 papersWe present a technique for interactive relighting in which source radiance, viewing direction, and BRDFs can all be changed on the fly. In handling dynamic BRDFs, our method efficiently accounts for the effects of BRDF modification on the reflectance ...






Comments