Facial hair tracking for high fidelity performance capture

Abstract

Facial hair is a largely overlooked topic in facial performance capture. Most production pipelines in the entertainment industry do not have a way to automatically capture facial hair or track the skin underneath it. Thus, actors are asked to shave clean before face capture, which is very often undesirable. Capturing the geometry of individual facial hairs is very challenging, and their presence makes it harder to capture the deforming shape of the underlying skin surface. Some attempts have already been made at automating this task, but only for static faces with relatively sparse 3D hair reconstructions. In particular, current methods lack the temporal correspondence needed when capturing a sequence of video frames depicting facial performance. The problem of robustly tracking the skin underneath also remains unaddressed. In this paper, we propose the first multiview reconstruction pipeline that tracks both the dense 3D facial hair, as well as the underlying 3D skin for entire performances. Our method operates with standard setups for face photogrammetry, without requiring dense camera arrays. For a given capture subject, our algorithm first reconstructs a dense, high-quality neutral 3D facial hairstyle by registering sparser hair reconstructions over multiple frames that depict a neutral face under quasi-rigid motion. This custom-built, reference facial hairstyle is then tracked throughout a variety of changing facial expressions in a captured performance, and the result is used to constrain the tracking of the 3D skin surface underneath. We demonstrate the proposed capture pipeline on a variety of different facial hairstyles and lengths, ranging from sparse and short to dense full-beards.

References

1. Sameer Agarwal, Keir Mierle, and Others. 2016. Ceres Solver. http://ceres-solver.org.
2. Thabo Beeler, Bernd Bickel, Paul Beardsley, Bob Sumner, and Markus Gross. 2010. High-Quality Single-Shot Capture of Facial Geometry. ACM Trans. Graphics (Proc. SIGGRAPH) 29, 4, Article 40 (2010).
3. Thabo Beeler, Bernd Bickel, Gioacchino Noris, Paul Beardsley, Steve Marschner, Robert W. Sumner, and Markus Gross. 2012. Coupled 3D Reconstruction of Sparse Facial Hair and Skin. ACM Trans. Graphics (Proc. SIGGRAPH) 31, 4, Article 117 (2012).
4. Thabo Beeler, Fabian Hahn, Derek Bradley, Bernd Bickel, Paul Beardsley, Craig Gotsman, Robert W. Sumner, and Markus Gross. 2011. High-Quality Passive Facial Performance Capture Using Anchor Frames. ACM Trans. Graphics (Proc. SIGGRAPH) 30, 4, Article 75 (2011).
5. Pascal Bérard, Derek Bradley, Markus Gross, and Thabo Beeler. 2016. Lightweight Eye Capture Using a Parametric Model. ACM Trans. Graphics (Proc. SIGGRAPH) 35, 4, Article 117 (2016).
6. Amit Bermano, Thabo Beeler, Yeara Kozlov, Derek Bradley, Bernd Bickel, and Markus Gross. 2015. Detailed Spatio-Temporal Reconstruction of Eyelids. ACM Trans. Graphics (Proc. SIGGRAPH) 34, 4, Article 44 (2015).
7. Derek Bradley, Wolfgang Heidrich, Tiberiu Popa, and Alla Sheffer. 2010. High Resolution Passive Facial Performance Capture. ACM Trans. Graphics (Proc. SIGGRAPH) 29, 4, Article 41 (2010).
8. Menglei Chai, Lvdi Wang, Yanlin Weng, Xiaogang Jin, and Kun Zhou. 2013. Dynamic Hair Manipulation in Images and Videos. ACM Trans. Graphics (Proc. SIGGRAPH) 32, 4, Article 75 (2013).
9. Prashanth Chandran, Derek Bradley, Markus Gross, and Thabo Beeler. 2020. Semantic Deep Face Models. In Int. Conf. on 3D Vision. 345--354.
10. Graham Fyffe. 2012. High Fidelity Facial Hair Capture. In ACM SIGGRAPH 2012 Talks. Article 23.
11. Graham Fyffe, Koki Nagano, Loc Huynh, Shunsuke Saito, Jay Busch, Andrew Jones, Hao Li, and Paul Debevec. 2017. Multi-View Stereo on Consistent Face Topology. Comp. Graphics Forum (Proc. Eurographics) 36, 2 (2017), 295--309.
12. Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Niessner. 2021. Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction. In IEEE Computer Vision and Pattern Recognition (CVPR). 8649--8658.
13. Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Debevec. 2011. Multiview face capture using polarized spherical gradient illumination. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 30, 6 (2011), 1--10.
14. Paulo Gotardo, Jérémy Riviere, Derek Bradley, Abhijeet Ghosh, and Thabo Beeler. 2018. Practical Dynamic Facial Appearance Modeling and Acquisition. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 37, 6, Article 232 (2018).
15. Stéphane Grabli, François X. Sillion, Stephen R. Marschner, and Jerome E. Lengyel. 2002. Image-Based Hair Capture by Inverse Lighting. In Proc. of Graphics Interface (GI). 51--58.
16. Tomas Lay Herrera, Arno Zinke, and Andreas Weber. 2012. Lighting Hair from the inside: A Thermal Approach to Hair Reconstruction. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 31, 6, Article 146 (2012).
17. Tomas Lay Herrera, Arno Zinke, Andreas Weber, and Thomas Vetter. 2010. Toward Image-Based Facial Hair Modeling. In Proc. of the 26th Spring Conf. on Computer Graphics. 93--100.
18. Osamu Hirose. 2021. A Bayesian Formulation of Coherent Point Drift. IEEE TPAMI 43, 7 (2021), 2269--2286.
19. Liwen Hu, Derek Bradley, Hao Li, and Thabo Beeler. 2017. Simulation-Ready Hair Capture. Comp. Graphics Forum (Proc. Eurographics) 36, 2 (2017), 281--294.
20. Liwen Hu, Chongyang Ma, Linjie Luo, and Hao Li. 2014. Robust Hair Capture Using Simulated Examples. ACM Trans. Graphics (Proc. SIGGRAPH) 33, 4, Article 126 (2014).
21. Takahito Ishikawa, Yosuke Kazama, Eiji Sugisaki, and Shigeo Morishima. 2007. Hair Motion Reconstruction Using Motion Capture System. In ACM SIGGRAPH 2007 Posters. 78--es.
22. Wenzel Jakob, Jonathan T. Moon, and Steve Marschner. 2009. Capturing Hair Assemblies Fiber by Fiber. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 28, 5 (2009), 1--9.
23. Samuli Laine, Tero Karras, Timo Aila, Antti Herva, Shunsuke Saito, Ronald Yu, Hao Li, and Jaakko Lehtinen. 2017. Production-Level Facial Performance Capture Using Deep Convolutional Neural Networks. In Proc. of Eurographics Symposium on Computer Animation. Article 10.
24. Chloe LeGendre, Loc Hyunh, Shanhe Wang, and Paul Debevec. 2017. Modeling Vellus Facial Hair from Asperity Scattering Silhouettes. In ACM SIGGRAPH 2017 Talks.
25. Tianye Li, Shichen Liu, Timo Bolkart, Jiayi Liu, Hao Li, and Yajie Zhao. 2021. Topologically Consistent Multi-View Face Inference Using Volumetric Sampling. In IEEE Int. Conf. on Computer Vision (ICCV). 3824--3834.
26. Shu Liang, Xiufeng Huang, Xianyu Meng, Kunyao Chen, Linda G. Shapiro, and Ira Kemelmacher-Shlizerman. 2018. Video to Fully Automatic 3D Hair Model. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 37, 6, Article 206 (2018).
27. Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. 2019. Neural Volumes: Learning Dynamic Renderable Volumes from Images. ACM Trans. Graphics (Proc. SIGGRAPH) 38, 4, Article 65 (2019).
28. Linjie Luo, Hao Li, and Szymon Rusinkiewicz. 2013. Structure-Aware Hair Capture. ACM Trans. Graphics (Proc. SIGGRAPH) 32, 4, Article 76 (2013).
29. Wan-Chun Ma, Tim Hawkins, Pieter Peers, Charles-Felix Chabert, Malte Weiss, and Paul Debevec. 2007. Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination. In Proc. Eurographics Conf. on Rendering Techniques. 183--194.
30. Masayuki Nakajima, Kong Wai Ming, and Hiroki Takashi. 1997. Generation of 3d hair model from multiple pictures. IEEE Comp. Graphics and Applications (1997).
31. Giljoo Nam, Chenglei Wu, Min H. Kim, and Yaser Sheikh. 2019. Strand-Accurate Multi-View Hair Capture. In IEEE Computer Vision and Pattern Recognition (CVPR). 155--164.
32. Sylvain Paris, Hector M. Briceño, and François X. Sillion. 2004. Capture of Hair Geometry from Multiple Images. ACM Trans. Graphics (Proc. SIGGRAPH) 23, 3 (2004), 712--719.
33. Sylvain Paris, Will Chang, Oleg I. Kozhushnyan, Wojciech Jarosz, Wojciech Matusik, Matthias Zwicker, and Frédo Durand. 2008. Hair Photobooth: Geometric and Photometric Acquisition of Real Hairstyles. ACM Trans. Graphics (Proc. SIGGRAPH) 27, 3 (2008), 1--9.
34. Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. 2021. HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 40, 6, Article 238 (2021).
35. Jérémy Riviere, Paulo Gotardo, Derek Bradley, Abhijeet Ghosh, and Thabo Beeler. 2020. Single-Shot High-Quality Facial Geometry and Skin Appearance Capture. ACM Trans. Graphics (Proc. SIGGRAPH) 39, 4, Article 81 (2020).
36. Gemma Rotger, Francesc Moreno-Noguer, Felipe Lumbreras, and Antonio Agudo. 2019. Single View Facial Hair 3D Reconstruction. In Pattern Rec. and Image Anal. 423--436.
37. Olga Sorkine, Daniel Cohen-Or, Yaron Lipman, Marc Alexa, Christian Rössl, and Hans-Peter Seidel. 2004. Laplacian Surface Editing. In Proc. of the Symposium on Geometry Processing. 175--184.
38. Tiancheng Sun, Giljoo Nam, Carlos Aliaga, Christophe Hery, and Ravi Ramamoorthi. 2021. Human Hair Inverse Rendering using Multi-View Photometric data. In Eurographics Symposium on Rendering.
39. Ayush Tewari, Mohamed Elgharib, Mallikarjun B R, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, and Christian Theobalt. 2020. PIE: Portrait Image Embedding for Semantic Control. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 39, 6, Article 223 (2020).
40. Ayush Tewari, Michael Zollhofer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Christian Theobalt. 2017. MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction. In Proc. ICCV Workshops.
41. Chenglei Wu, Derek Bradley, Pablo Garrido, Michael Zollhöfer, Christian Theobalt, Markus Gross, and Thabo Beeler. 2016a. Model-Based Teeth Reconstruction. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 35, 6, Article 220 (2016).
42. Chenglei Wu, Derek Bradley, Markus Gross, and Thabo Beeler. 2016b. An Anatomically-Constrained Local Deformation Model for Monocular Face Capture. ACM Trans. Graphics (Proc. SIGGRAPH) 35, 4, Article 115 (2016).
43. Zexiang Xu, Hsiang-Tao Wu, Lvdi Wang, Changxi Zheng, Xin Tong, and Yue Qi. 2014. Dynamic Hair Capture Using Spacetime Optimization. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 33, 6, Article 224 (2014).
44. Tatsuhisa Yamaguchi, Bennett Wilburn, and Eyal Ofek. 2009. Video-Based Modeling of Dynamic Hair. In Adv. in Image and Video Technology. 585--596.
45. Lingchen Yang, Zefeng Shi, Youyi Zheng, and Kun Zhou. 2019. Dynamic Hair Modeling from Monocular Videos Using Deep Neural Networks. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 38, 6, Article 235 (2019).
46. Meng Zhang, Menglei Chai, Hongzhi Wu, Hao Yang, and Kun Zhou. 2017. A Data-Driven Approach to Four-View Image-Based Hair Modeling. ACM Trans. Graphics (Proc. SIGGRAPH) 36, 4, Article 156 (2017).
47. Gaspard Zoss, Thabo Beeler, Markus Gross, and Derek Bradley. 2019. Accurate Markerless Jaw Tracking for Facial Performance Capture. ACM Trans. Graphics (Proc. SIGGRAPH) 38, 4, Article 50 (2019).

Index Terms

• Computing methodologies

  • Computer graphics

    • Animation

      • Motion capture

      • Motion processing