Learning active quasistatic physics-based models from data

Abstract

Humans and animals can control their bodies to generate a wide range of motions via low-dimensional action signals representing high-level goals. As such, human bodies and faces are prime examples of active objects, which can affect their shape via an internal actuation mechanism. This paper explores the following proposition: given a training set of example poses of an active deformable object, can we learn a low-dimensional control space that could reproduce the training set and generalize to new poses? In contrast to popular machine learning methods for dimensionality reduction such as auto-encoders, we model our active objects in a physics-based way. We utilize a differentiable, quasistatic, physics-based simulation layer and combine it with a decoder-type neural network. Our differentiable physics layer naturally fits into deep learning frameworks and allows the decoder network to learn actuations that reach the desired poses after physics-based simulation. In contrast to modeling approaches where users build anatomical models from first principles, medical literature or medical imaging, we do not presume knowledge of the underlying musculature, but learn the structure and control of the actuation mechanism directly from the input data. We present a training paradigm and several scalability-oriented enhancements that allow us to train effectively while accommodating high-resolution volumetric models, with as many as a quarter million simulation elements. The prime demonstration of the efficacy of our example-driven modeling framework targets facial animation, where we train on a collection of input expressions while generalizing to unseen poses, drive detailed facial animation from sparse motion capture input, and facilitate expression sculpting via direct manipulation.

References

1. Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein generative adversarial networks. In International conference on machine learning. 214--223.
2. Timur Bagautdinov, Chenglei Wu, Jason Saragih, Pascal Fua, and Yaser Sheikh. 2018. Modeling Facial Geometry Using Compositional VAEs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
3. Stephen W. Bailey, Dalton Omens, Paul Dilorenzo, and James F. O'Brien. 2020. Fast and Deep Facial Deformations. ACM Trans. Graph. 39, 4, Article 94 (July 2020).
4. Michael Bao, Matthew Cong, Stephane Grabli, and Ronald Fedkiw. 2019. High-Quality Face Capture Using Anatomical Muscles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
5. GG Barbarino, M Jabareen, J Trzewik, A Nkengne, G Stamatas, and E Mazza. 2009. Development and validation of a three-dimensional finite element model of the face. Journal of biomechanical engineering 131, 4 (2009), 041006.
6. James Bern, Grace Kumagai, and Stelian Coros. 2017b. Fabrication, Modeling, and Control of Plush Robots. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 3739 -- 3746. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2017); Conference Location: Vancouver, Canada; Conference Date: September 24-28, 2017.
7. James M Bern, Pol Banzet, Roi Poranne, and Stelian Coros. 2019. Trajectory optimization for cable-driven soft robot locomotion. Proceedings of Robotics: Science and Systems (2019).
8. James M. Bern, Kai-Hung Chang, and Stelian Coros. 2017a. Interactive Design of Animated Plushies. ACM Trans. Graph. 36, 4, Article 80 (July 2017), 11 pages.
9. Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques. 187--194.
10. Silvia Salinas Blemker. 2004. 3D modeling of complex muscle architecture and geometry. Ph.D. Dissertation. Stanford University.
11. Sofien Bouaziz, Sebastian Martin, Tiantian Liu, Ladislav Kavan, and Mark Pauly. 2014. Projective dynamics: fusing constraint projections for fast simulation. Proc. of ACM SIGGRAPH 33, 4 (2014).
12. Matthew Cong, Kiran S Bhat, and Ronald Fedkiw. 2016. Art-directed muscle simulation for high-end facial animation. 119--127.
13. Stelian Coros, Sebastian Martin, Bernhard Thomaszewski, Christian Schumacher, Robert Sumner, and Markus Gross. 2012. Deformable objects alive! ACM Transactions on Graphics (TOG) 31, 4 (2012), 69.
14. Kai Ding, Libin Liu, Michiel van de Panne, and KangKang Yin. 2015. Learning Reduced-Order Feedback Policies for Motion Skills. In Proceedings of the 14th ACM SIGGRAPH / Eurographics Symposium on Computer Animation (Los Angeles, California) (SCA '15). Association for Computing Machinery, New York, NY, USA, 83--92.
15. Carl Doersch. 2016. Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05908 (2016).
16. Tao Du, Kui Wu, Pingchuan Ma, Sebastien Wah, Andrew Spielberg, Daniela Rus, and Wojciech Matusik. 2021. DiffPD: Differentiable Projective Dynamics with Contact. arXiv:2101.05917 [cs.LG]
17. Paul Ekman and Wallace V Friesen. 1977. Facial action coding system. (1977).
18. Ye Fan, Joshua Litven, and Dinesh K Pai. 2014. Active volumetric musculoskeletal systems. Proc. of ACM SIGGRAPH 33, 4 (2014), 152.
19. François Faure, Christian Duriez, Hervé Delingette, Jérémie Allard, Benjamin Gilles, Stéphanie Marchesseau, Hugo Talbot, Hadrien Courtecuisse, Guillaume Bousquet, Igor Peterlik, et al. 2012. Sofa: A multi-model framework for interactive physical simulation. In Soft tissue biomechanical modeling for computer assisted surgery. Springer, 283--321.
20. Cormac Flynn, Ian Stavness, John Lloyd, and Sidney Fels. 2015. A finite element model of the face including an orthotropic skin model under in vivo tension. Computer methods in biomechanics and biomedical engineering 18, 6 (2015), 571--582.
21. Moritz Geilinger, David Hahn, Jonas Zehnder, Moritz Bächer, Bernhard Thomaszewski, and Stelian Coros. 2020. ADD: analytically differentiable dynamics for multi-body systems with frictional contact. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1--15.
22. Zhenglin Geng, Daniel Johnson, and Ronald Fedkiw. 2020. Coercing machine learning to output physically accurate results. J. Comput. Phys. 406 (Apr 2020), 109099.
23. Evgeny Gladilin. 2003. Biomechanical modeling of soft tissue and facial expressions for craniofacial surgery planning. Freien University, Berlin (2003).
24. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680.
25. David Hahn, Pol Banzet, James M Bern, and Stelian Coros. 2019. Real2sim: Visco-elastic parameter estimation from dynamic motion. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1--13.
26. I. Higgins, Loïc Matthey, A. Pal, C. Burgess, Xavier Glorot, M. Botvinick, S. Mohamed, and Alexander Lerchner. 2017. beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. In ICLR.
27. Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand. 2020. DiffTaichi: Differentiable Programming for Physical Simulation. In International Conference on Learning Representations.
28. Yuanming Hu, Jiancheng Liu, Andrew Spielberg, Joshua B Tenenbaum, William T Freeman, Jiajun Wu, Daniela Rus, and Wojciech Matusik. 2019. Chainqueen: A real-time differentiable physical simulator for soft robotics. In 2019 International Conference on Robotics and Automation (ICRA). IEEE, 6265--6271.
29. Alexandru Ichim, Ladislav Kavan, Merlin Nimier-David, and Mark Pauly. 2016. Building and Animating User-Specific Volumetric Face Rigs.
30. Alexandru-Eugen Ichim, Petr Kadleček, Ladislav Kavan, and Mark Pauly. 2017. Phace: Physics-based face modeling and animation. ACM Transactions on Graphics (TOG) 36, 4 (2017), 153.
31. Petr Kadlecek and Ladislav Kavan. 2019. Building Accurate Physics-Based Face Models from Data. In Symposium on Computer Animation.
32. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR.
33. Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
34. Gergely Klár, Andrew Moffat, Ken Museth, and Eftychios Sifakis. 2020. Shape Targeting: A Versatile Active Elasticity Constitutive Model. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Talks (SIGGRAPH '20). Association for Computing Machinery, Article 59, 2 pages.
35. Yeara Kozlov, Derek Bradley, Moritz Bächer, Bernhard Thomaszewski, Thabo Beeler, and Markus Gross. 2017. Enriching Facial Blendshape Rigs with Physical Simulation. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 75--84.
36. Lana Lan, Matthew Cong, and Ronald Fedkiw. 2017. Lessons from the evolution of an anatomical facial muscle model. In Proceedings of the ACM SIGGRAPH Digital Production Symposium. ACM, 11.
37. John P Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Frederic H Pighin, and Zhigang Deng. 2014. Practice and Theory of Blendshape Facial Models. Eurographics (State of the Art Reports) 1, 8 (2014), 2.
38. Jiaman Li, Zhengfei Kuang, Yajie Zhao, Mingming He, Karl Bladin, and Hao Li. 2020b. Dynamic facial asset and rig generation from a single scan. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1--18.
39. Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara, Owen Ingraham, Pengda Xiang, Xinglei Ren, Pratusha Prasad, Bipin Kishore, Jun Xing, et al. 2020a. Learning Formation of Physically-Based Face Attributes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3410--3419.
40. John E Lloyd, Ian Stavness, and Sidney Fels. 2012. ArtiSynth: A fast interactive biomechanical modeling toolkit combining multibody and finite element simulation. In Soft tissue biomechanical modeling for computer assisted surgery. Springer, 355--394.
41. Stephen Lombardi, Jason Saragih, Tomas Simon, and Yaser Sheikh. 2018. Deep appearance models for face rendering. ACM Transactions on Graphics (TOG) 37, 4 (2018), 68.
42. Steve A Maas, Gerard A Ateshian, and Jeffrey A Weiss. 2017. FEBio: History and advances. Annual review of biomedical engineering 19 (2017), 279--299.
43. Sebastian Martin, Bernhard Thomaszewski, Eitan Grinspun, and Markus Gross. 2011. Example-based elastic materials. In ACM Transactions on Graphics (TOG), Vol. 30. ACM, 72.
44. Aleka McAdams, Yongning Zhu, Andrew Selle, Mark Empey, Rasmus Tamstorf, Joseph Teran, and Eftychios Sifakis. 2011. Efficient elasticity for character skinning with contact and collisions. ACM Transactions on Graphics (TOG) 30, 4 (2011), 37.
45. Antoine McNamara, Adrien Treuille, Zoran Popović, and Jos Stam. 2004. Fluid Control Using the Adjoint Method. ACM Trans. Graph. 23, 3 (Aug. 2004), 449--456.
46. Nathan Mitchell, Court Cutting, and Eftychios Sifakis. 2015. GRIDiron: An interactive authoring and cognitive training foundation for reconstructive plastic surgery procedures. ACM Trans. Graph. (Proceedings of ACM SIGGRAPH) (2015).
47. Paola Nardinocchi and Luciano Teresi. 2007. On the active response of soft living tissues. Journal of Elasticity 88, 1 (2007), 27--39.
48. Thomas Neumann, Kiran Varanasi, Stephan Wenger, Markus Wacker, Marcus Magnor, and Christian Theobalt. 2013. Sparse Localized Deformation Components. ACM Trans. Graph. 32, 6, Article 179 (Nov. 2013), 10 pages.
49. Michael Schmidt and Hod Lipson. 2009. Distilling Free-Form Natural Laws from Experimental Data. Science 324, 5923 (2009), 81--85.
50. Gabriel Schwartz, Shih-En Wei, Te-Li Wang, Stephen Lombardi, Tomas Simon, Jason Saragih, and Yaser Sheikh. 2020. The eyes have it: an integrated eye and face model for photorealistic facial animation. ACM Transactions on Graphics (TOG) 39, 4 (2020), 91--1.
51. Eftychios Sifakis and Jernej Barbič. 2012. FEM Simulation of 3D Deformable Solids: A practitioner's guide to theory, discretization and model reduction. http://www.femdefo.org.
52. Eftychios Sifakis, Igor Neverov, and Ronald Fedkiw. 2005. Automatic determination of facial muscle activations from sparse motion capture marker data. In Proc. of ACM SIGGRAPH, Vol. 24. 417--425.
53. Eftychios Sifakis, Tamar Shinar, Geoffrey Irving, and Ronald Fedkiw. 2007. Hybrid simulation of deformable solids. In Proceedings of the 2007 ACM SIGGRAPH/Eurographics symposium on Computer animation. Eurographics Association, 81--90.
54. Breannan Smith, Chenglei Wu, He Wen, Patrick Peluse, Yaser Sheikh, Jessica K Hodgins, and Takaaki Shiratori. 2020. Constraining dense hand surface tracking with elasticity. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1--14.
55. Martin Spüler, Nerea Irastorza Landa, Andrea Sarasola Sanz, and Ander Ramos-Murguialday. 2016. Extracting Muscle Synergy Patterns from EMG Data Using Autoencoders. In Artificial Neural Networks and Machine Learning - ICANN 2016. 47--54.
56. Ian Stavness, Mohammad Ali Nazari, Cormac Flynn, Pascal Perrier, Yohan Payan, John E Lloyd, and Sidney Fels. 2014. Coupled biomechanical modeling of the face, jaw, skull, tongue, and hyoid bone. In 3D Multiscale Physiological Human. Springer, 253--274.
57. Jie Tan, Greg Turk, and C Karen Liu. 2012. Soft body locomotion. ACM Transactions on Graphics (TOG) 31, 4 (2012), 26.
58. J Rafael Tena, Fernando De la Torre, and Iain Matthews. 2011. Interactive region-based linear 3d face models. In ACM SIGGRAPH 2011 papers. 1--10.
59. Joseph Teran, Sylvia Blemker, V Hing, and Ronald Fedkiw. 2003. Finite volume methods for the simulation of skeletal muscle. Eurographics Association, 68--74.
60. Joseph Teran, Eftychios Sifakis, Silvia S Blemker, Victor Ng-Thow-Hing, Cynthia Lau, and Ronald Fedkiw. 2005. Creating and simulating skeletal muscle from the visible human data set. IEEE TVCG 11, 3 (2005), 317--328.
61. Shih-En Wei, Jason Saragih, Tomas Simon, Adam W Harley, Stephen Lombardi, Michal Perdoch, Alexander Hypes, Dawei Wang, Hernan Badino, and Yaser Sheikh. 2019. Vr facial animation via multiview image translation. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--16.
62. Jeffrey A Weiss, Bradley N Maker, and Sanjay Govindjee. 1996. Finite element implementation of incompressible, transversely isotropic hyperelasticity. Computer methods in applied mechanics and engineering 135, 1 (1996), 107--128.
63. Chris Wojtan, Peter J. Mucha, and Greg Turk. 2006. Keyframe control of complex particle systems using the adjoint method. In SCA '06: Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on Computer animation (Vienna, Austria). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 15--23.
64. Chenglei Wu, Derek Bradley, Markus Gross, and Thabo Beeler. 2016. An anatomically-constrained local deformation model for monocular face capture. ACM transactions on graphics (TOG) 35, 4 (2016), 1--12.
65. Felix E Zajac. 1989. Muscle and tendon Properties models scaling and application to biomechanics and motor. Critical reviews in biomedical engineering 17, 4 (1989), 359--411.
66. Xiaotian Zhang, Fan Kiat Chan, Tejaswin Parthasarathy, and Mattia Gazzola. 2019. Modeling and simulation of complex dynamic musculoskeletal architectures. Nature communications 10, 1 (2019), 1--12.

Index Terms

• Computing methodologies

  • Computer graphics

    • Animation

      • Physical simulation