Contact-centric deformation learning

Abstract

We propose a novel method to machine-learn highly detailed, nonlinear contact deformations for real-time dynamic simulation. We depart from previous deformation-learning strategies, and model contact deformations in a contact-centric manner. This strategy shows excellent generalization with respect to the object's configuration space, and it allows for simple and accurate learning. We complement the contact-centric learning strategy with two additional key ingredients: learning a continuous vector field of contact deformations, instead of a discrete approximation; and sparsifying the mapping between the contact configuration and contact deformations. These two ingredients further contribute to the accuracy, efficiency, and generalization of the method. We integrate our learning-based contact deformation model with subspace dynamics, showing real-time dynamic simulations with fine contact deformation detail.

References

1. Thiemo Alldieck, Hongyi Xu, and Cristian Sminchisescu. 2021. imGHUM: Implicit Generative Models of 3D Human Shape and Articulated Pose. In Proc. of Computer Vision and Pattern Recognition (CVPR). 5461--5470.
2. Steven S. An, Theodore Kim, and Doug L. James. 2008. Optimizing Cubature for Efficient Integration of Subspace Deformations. ACM Trans. Graph. 27, 5, Article 165 (2008).
3. Uri M. Ascher and Eddy Boxerman. 2003. On the Modified Conjugate Gradient Method in Cloth Simulation. Vis. Comput. 19, 7--8 (2003).
4. Matan Atzmon and Yaron Lipman. 2020. SAL: Sign Agnostic Learning of Shapes from Raw Data. In Proc. of Computer Vision and Pattern Recognition (CVPR).
5. Stephen W. Bailey, Dave Otte, Paul Dilorenzo, and James F. O'Brien. 2018. Fast and Deep Deformation Approximations. ACM Trans. Graph. 37, 4 (2018).
6. Jernej Barbič and Doug L. James. 2005. Real-Time Subspace Integration for St. Venant-Kirchhoff Deformable Models. ACM Trans. Graph. 24, 3 (July 2005), 982--990.
7. Jernej Barbič and Yili Zhao. 2011. Real-Time Large-Deformation Substructuring. In ACM SIGGRAPH 2011 Papers (Vancouver, British Columbia, Canada) (SIGGRAPH '11). Association for Computing Machinery, New York, NY, USA, Article 91, 8 pages.
8. Nuri Benbarka, Timon Höfer, Hamd ul-Moqeet Riaz, and Andreas Zell. 2022. Seeing Implicit Neural Representations As Fourier Series. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 2041--2050.
9. Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. 2020. LoopReg: Self-supervised Learning of Implicit Surface Correspondences, Pose and Shape for 3D Human Mesh Registration. In Advances in Neural Information Processing Systems (NeurIPS).
10. Christopher Brandt, Elmar Eisemann, and Klaus Hildebrandt. 2018. Hyper-Reduced Projective Dynamics. ACM Trans. Graph. 37, 4, Article 80 (2018).
11. Dan Casas and Miguel A Otaduy. 2018. Learning nonlinear soft-tissue dynamics for interactive avatars. Proceedings of the ACM on Computer Graphics and Interactive Techniques 1, 1 (2018), 10.
12. Peter Yichen Chen, Maurizio Chiaramonte, Eitan Grinspun, and Kevin Carlberg. 2021. Model reduction for the material point method via learning the deformation map and its spatial-temporal gradients. arXiv preprint arXiv:2109.12390 (2021).
13. Zhiqin Chen and Hao Zhang. 2019. Learning implicit fields for generative shape modeling. In Proc. of Computer Vision and Pattern Recognition (CVPR). 5939--5948.
14. Julian Chibane, Aymen Mir, and Gerard Pons-Moll. 2020. Neural Unsigned Distance Fields for Implicit Function Learning. In Advances in Neural Information Processing Systems (NeurIPS).
15. Enric Corona, Albert Pumarola, Guillem Alenyà, Gerard Pons-Moll, and Francesc Moreno-Noguer. 2021. SMPLicit: Topology-aware Generative Model for Clothed People. In Proc. of Computer Vision and Pattern Recognition (CVPR).
16. Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, and Andrea Tagliasacchi. 2020. NASA: Neural Articulated Shape Approximation. In Proc. of European Conference on Computer Vision (ECCV).
17. Eric Ferley, Marie-Paule Cani, and Jean-Dominique Gascuel. 1999. Practical Volumetric Sculpting. The Visual Computer 16 (09 1999).
18. Lawson Fulton, Vismay Modi, David Duvenaud, David I. W. Levin, and Alec Jacobson. 2019. Latent-space Dynamics for Reduced Deformable Simulation. Computer Graphics Forum 38, 2 (2019), 379--391.
19. Theodore F Gast, Craig Schroeder, Alexey Stomakhin, Chenfanfu Jiang, and Joseph M Teran. 2015. Optimization integrator for large time steps. IEEE Transactions on Visualization and Computer Graphics (TVCG) 21, 10 (2015), 1103--1115.
20. Benjamin Gilles, Guillaume Bousquet, Francois Faure, and Dinesh K. Pai. 2011. Frame-Based Elastic Models. ACM Trans. Graph. 30, 2 (2011).
21. Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. 2020. Implicit Geometric Regularization for Learning Shapes. In Proceedings of Machine Learning and Systems 2020. 3569--3579.
22. Fabian Hahn, Sebastian Martin, Bernhard Thomaszewski, Robert Sumner, Stelian Coros, and Markus Gross. 2012. Rig-Space Physics. ACM Trans. Graph. 31, 4, Article 72 (July 2012), 8 pages.
23. Fabian Hahn, Bernhard Thomaszewski, Stelian Coros, Robert W. Sumner, Forrester Cole, Mark Meyer, Tony DeRose, and Markus Gross. 2014. Subspace Clothing Simulation Using Adaptive Bases. ACM Trans. Graph. 33, 4 (2014).
24. David Harmon and Denis Zorin. 2013. Subspace Integration with Local Deformations. ACM Trans. Graph. 32, 4, Article 107 (July 2013), 10 pages.
25. Daniel Holden, Bang Chi Duong, Sayantan Datta, and Derek Nowrouzezahrai. 2019. Subspace Neural Physics: Fast Data-Driven Interactive Simulation. In Proceedings of the 18th Annual ACM SIGGRAPH/Eurographics Symposium on Computer Animation.
26. Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, and Tony Tung. 2020. ARCH: Animatable Reconstruction of Clothed Humans. In Proc. of Computer Vision and Pattern Recognition (CVPR). 3093--3102.
27. Alec Jacobson, Zhigang Deng, Ladislav Kavan, and JP Lewis. 2014. Skinning: Real-time Shape Deformation. In ACM SIGGRAPH 2014 Courses.
28. Korrawe Karunratanakul, Adrian Spurr, Zicong Fan, Otmar Hilliges, and Siyu Tang. 2021. A Skeleton-Driven Neural Occupancy Representation for Articulated Hands. In International Conference on 3D Vision (3DV).
29. Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael J Black, Krikamol Muandet, and Siyu Tang. 2020. Grasping Field: Learning Implicit Representations for Human Grasps. In International Conference on 3D Vision.
30. Meekyoung Kim, Gerard Pons-Moll, Sergi Pujades, Seungbae Bang, Jinwook Kim, Michael J. Black, and Sung-Hee Lee. 2017. Data-Driven Physics for Human Soft Tissue Animation. ACM Trans. Graph. 36, 4, Article 54 (2017), 12 pages.
31. Theodore Kim and Doug L. James. 2011. Physics-Based Character Skinning Using Multi-Domain Subspace Deformations. In Proceedings of the 2011 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 63--72.
32. P. Krysl, S. Lall, and J. E. Marsden. 2001. Dimensional model reduction in non-linear finite element dynamics of solids and structures. Internat. J. Numer. Methods Engrg. 51, 4 (2001), 479--504.
33. Tsuneya Kurihara and Natsuki Miyata. 2004. Modeling Deformable Human Hands from Medical Images. In Proc. of ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 355--363.
34. Lei Lan, Ran Luo, Marco Fratarcangeli, Weiwei Xu, Huamin Wang, Xiaohu Guo, Junfeng Yao, and Yin Yang. 2020. Medial Elastics: Efficient and Collision-Ready Deformation via Medial Axis Transform. ACM Trans. Graph. 39, 3, Article 20 (2020).
35. Kookjin Lee and Kevin T. Carlberg. 2021. Deep Conservation: A Latent-Dynamics Model for Exact Satisfaction of Physical Conservation Laws. In AAAI.
36. J. P. Lewis, Matt Cordner, and Nickson Fong. 2000. Pose Space Deformation: A Unified Approach to Shape Interpolation and Skeleton-Driven Deformation. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH). ACM Press/Addison-Wesley Publishing Co., USA, 165--172.
37. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2015. SMPL: A Skinned Multi-person Linear Model. ACM Trans. Graph. 34, 6, Article 248 (Oct. 2015), 16 pages.
38. Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J. Black. 2020. Learning to Dress 3D People in Generative Clothing. In Proc. of Computer Vision and Pattern Recognition (CVPR).
39. N. Magnenat-Thalmann, R. Laperrière, and D. Thalmann. 1988. Joint-dependent Local Deformations for Hand Animation and Object Grasping. In Proceedings on Graphics Interface '88 (Edmonton, Alberta, Canada). Canadian Information Processing Society, Toronto, Ont., Canada, Canada, 26--33.
40. Sebastian Martin, Bernhard Thomaszewski, Eitan Grinspun, and Markus Gross. 2011. Example-Based Elastic Materials. ACM Trans. Graph. 30, 4, Article 72 (2011).
41. Aleka McAdams, Yongning Zhu, Andrew Selle, Mark Empey, Rasmus Tamstorf, Joseph Teran, and Eftychios Sifakis. 2011. Efficient Elasticity for Character Skinning with Contact and Collisions. In ACM SIGGRAPH 2011 Papers (Vancouver, British Columbia, Canada) (SIGGRAPH '11). Association for Computing Machinery, New York, NY, USA, Article 37.
42. Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2019. Occupancy Networks: Learning 3D Reconstruction in Function Space. In Proc. of Computer Vision and Pattern Recognition (CVPR).
43. Marko Mihajlovic, Yan Zhang, Michael J Black, and Siyu Tang. 2021. LEAP: Learning Articulated Occupancy of People. In Proc. of Computer Vision and Pattern Recognition (CVPR).
44. Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In Proc. of European Conference on Computer Vision (ECCV).
45. Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Love-grove. 2019. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In Proc. of Computer Vision and Pattern Recognition (CVPR).
46. Chaitanya Patel, Zhouyingcheng Liao, and Gerard Pons-Moll. 2020. TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Style. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE.
47. Alex Pentland and John Williams. 1989. Good Vibrations: Modal Dynamics for Graphics and Animation. Computer Graphics 23, 3 (1989), 215--222.
48. Gerard Pons-Moll, Sergi Pujades, Sonny Hu, and Michael J. Black. 2017. ClothCap: Seamless 4D Clothing Capture and Retargeting. ACM Transactions on Graphics (Proc. of SIGGRAPH) 36, 4, Article 73 (2017), 15 pages.
49. Gerard Pons-Moll, Javier Romero, Naureen Mahmood, and Michael J. Black. 2015. Dyna: A Model of Dynamic Human Shape in Motion. ACM Trans. Graph. 34, 4, Article 120 (July 2015), 14 pages.
50. Cristian Romero, Dan Casas, Jesús Pérez, and Miguel Otaduy. 2021. Learning Contact Corrections for Handle-Based Subspace Dynamics. ACM Trans. Graph. 40, 4 (2021).
51. Cristian Romero, Miguel A. Otaduy, Dan Casas, and Jesus Perez. 2020. Modeling and Estimation of Nonlinear Skin Mechanics for Animated Avatars. Computer Graphics Forum (Proc. Eurographics) 39, 2 (2020).
52. Javier Romero, Dimitrios Tzionas, and Michael J. Black. 2017. Embodied Hands: Modeling and Capturing Hands and Bodies Together. ACM Trans. Graph. 36, 6, Article 245 (2017).
53. Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. 2019. PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization.
54. Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. 2020. PIFuHD: MultiLevel Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization. In Proc. of Computer Vision and Pattern Recognition (CVPR).
55. Shunsuke Saito, Jinlong Yang, Qianli Ma, and Michael J. Black. 2021. SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks. In Proc. of Computer Vision and Pattern Recognition (CVPR).
56. Igor Santesteban, Elena Garces, Miguel A. Otaduy, and Dan Casas. 2020. SoftSMPL: Data-driven Modeling of Nonlinear Soft-tissue Dynamics for Parametric Humans. Computer Graphics Forum 39, 2 (2020), 65--75.
57. Igor Santesteban, Miguel A. Otaduy, and Dan Casas. 2019. Learning-Based Animation of Clothing for Virtual Try-On. Computer Graphics Forum 38, 2 (2019), 355--366.
58. Igor Santesteban, Nils Thuerey, Miguel A Otaduy, and Dan Casas. 2021. Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021).
59. Siyuan Shen, Yin Yang, Tianjia Shao, He Wang, Chenfanfu Jiang, Lei Lan, and Kun Zhou. 2021. High-Order Differentiable Autoencoder for Nonlinear Model Reduction. ACM Trans. Graph. 40, 4, Article 68 (2021).
60. Eftychios Sifakis and Jernej Barbic. 2012. FEM Simulation of 3D Deformable Solids: A Practitioner's Guide to Theory, Discretization and Model Reduction. In ACM SIGGRAPH 2012 Courses. 20:1--20:50.
61. Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. 2020. Implicit Neural Representations with Periodic Activation Functions. In Proc. NeurIPS.
62. Breannan Smith, Fernando De Goes, and Theodore Kim. 2018. Stable Neo-Hookean Flesh Simulation. ACM Trans. Graph. 37, 2, Article 12 (2018).
63. Steven L. Song, Weiqi Shi, and Michael Reed. 2020. Accurate Face Rig Approximation with Deep Differential Subspace Reconstruction. ACM Trans. Graph. 39, 4 (2020).
64. Javier Tapia, Cristian Romero, Jesús Pérez, and Miguel A. Otaduy. 2021. Parametric Skeletons with Reduced Soft-Tissue Deformations. Computer Graphics Forum (2021).
65. Yun Teng, Mark Meyer, Tony DeRose, and Theodore Kim. 2015. Subspace Condensation: Full Space Adaptivity for Subspace Deformations. ACM Trans. Graph. 34, 4, Article 76 (July 2015), 9 pages.
66. Demetri Terzopoulos, John Platt, Alan Barr, and Kurt Fleischer. 1987. Elastically Deformable Models. In Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '87). Association for Computing Machinery, New York, NY, USA, 205--214.
67. Garvita Tiwari, Nikolaos Sarafianos, Tony Tung, and Gerard Pons-Moll. 2021. Neural-GIF: Neural generalized implicit functions for animating people in clothing. In Proc. of Computer Vision and Pattern Recognition (CVPR). 11708--11718.
68. Yu Wang, Alec Jacobson, Jernej Barbič, and Ladislav Kavan. 2015. Linear Subspace Design for Real-Time Shape Deformation. ACM Trans. Graph. 34, 4, Article 57 (2015).
69. Xiaofeng Wu, Rajaditya Mukherjee, and Huamin Wang. 2015. A Unified Approach for Subspace Simulation of Deformable Bodies in Multiple Domains. ACM Trans. Graph. 34, 6, Article 241 (2015).
70. Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. 2021. Neural Fields in Visual Computing and Beyond. arXiv:arXiv:2111.11426 https://neuralfields.cs.brown.edu/
71. Hongyi Xu and Jernej Barbič. 2016. Pose-Space Subspace Dynamics. ACM Trans. Graph. 35, 4, Article 35 (July 2016), 14 pages.
72. Meng Zhang, Tuanfeng Y. Wang, Duygu Ceylan, and Niloy J. Mitra. 2021. Dynamic Neural Garments. ACM Trans. Graph. 40, 6, Article 235 (2021).

Index Terms

• Computing methodologies

  • Computer graphics

    • Animation

      • Physical simulation