Learning skeletal articulations with neural blend shapes

Abstract

Animating a newly designed character using motion capture (mocap) data is a long standing problem in computer animation. A key consideration is the skeletal structure that should correspond to the available mocap data, and the shape deformation in the joint regions, which often requires a tailored, pose-specific refinement. In this work, we develop a neural technique for articulating 3D characters using enveloping with a pre-defined skeletal structure which produces high quality pose dependent deformations. Our framework learns to rig and skin characters with the same articulation structure (e.g., bipeds or quadrupeds), and builds the desired skeleton hierarchy into the network architecture. Furthermore, we propose neural blend shapes - a set of corrective pose-dependent shapes which improve the deformation quality in the joint regions in order to address the notorious artifacts resulting from standard rigging and skinning. Our system estimates neural blend shapes for input meshes with arbitrary connectivity, as well as weighting coefficients which are conditioned on the input joint rotations. Unlike recent deep learning techniques which supervise the network with ground-truth rigging and skinning parameters, our approach does not assume that the training data has a specific underlying deformation model. Instead, during training, the network observes deformed shapes and learns to infer the corresponding rig, skin and blend shapes using indirect supervision. During inference, we demonstrate that our network generalizes to unseen characters with arbitrary mesh connectivity, including unrigged characters built by 3D artists. Conforming to standard skeletal animation models enables direct plug-and-play in standard animation software, as well as game engines.

References

1. Kfir Aberman, Peizhuo Li, Olga Sorkine-Hornung, Dani Lischinski, Daniel Cohen-Or, and Baoquan Chen. 2020. Skeleton-Aware Networks for Deep Motion Retargeting. ACM Transactions on Graphics (TOG) 39, 4 (2020), 62.
2. Adobe Systems Inc. 2018. Mixamo. https://www.mixamo.com. https://www.mixamo.com Accessed: 2018-12-27.
3. Oscar Kin-Chung Au, Chiew-Lan Tai, Hung-Kuo Chu, Daniel Cohen-Or, and Tong-Yee Lee. 2008. Skeleton Extraction by Mesh Contraction. ACM Transactions on Graphics 27, 3 (Aug. 2008), 1--10.
4. Stephen W. Bailey, Dalton Omens, Paul Dilorenzo, and James F. O'Brien. 2020. Fast and Deep Facial Deformations. ACM Trans. Graph. 39, 4, Article 94 (July 2020), 15 pages.
5. Stephen W Bailey, Dave Otte, Paul Dilorenzo, and James F O'Brien. 2018. Fast and deep deformation approximations. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1--12.
6. Ilya Baran and Jovan Popović. 2007. Automatic rigging and animation of 3d characters. ACM Transactions on graphics (TOG) 26, 3 (2007), 72--es.
7. Gaurav Bharaj, Thorsten Thormählen, Hans-Peter Seidel, and Christian Theobalt. 2012. Automatically Rigging Multi-component Characters. Computer Graphics Forum 31, 2pt4 (2012), 755--764. : https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-8659.2012.03034.x.
8. Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll. 2019. Multi-Garment Net: Learning to Dress 3D People from Images. In IEEE International Conference on Computer Vision (ICCV). IEEE.
9. Mario Botsch, Leif Kobbelt, Mark Pauly, Pierre Alliez, and Bruno Lévy. 2010. Polygon mesh processing. CRC press.
10. Junjie Cao, Andrea Tagliasacchi, Matt Olson, Hao Zhang, and Zhinxun Su. 2010. Point Cloud Skeletons via Laplacian Based Contraction. In 2010 Shape Modeling International Conference. IEEE, Aix-en-Provence, France, 187--197.
11. Edilson De Aguiar, Christian Theobalt, Sebastian Thrun, and Hans-Peter Seidel. 2008. Automatic conversion of mesh animations into skeleton-based animations. In Computer Graphics Forum, Vol. 27. Wiley Online Library, 389--397.
12. Olivier Dionne and Martin de Lasa. 2013. Geodesic voxel binding for production character meshes. In Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA '13). Association for Computing Machinery, New York, NY, USA, 173--180.
13. Stefan Fröhlich and Mario Botsch. 2011. Example-driven deformations based on discrete shells. Computer graphics forum 30, 8 (2011), 2246--2257.
14. Michael Gleicher. 1998. Retargetting motion to new characters. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. 33--42.
15. Fabian Hahn, Sebastian Martin, Bernhard Thomaszewski, Robert Sumner, Stelian Coros, and Markus Gross. 2012. Rig-Space Physics. ACM Transactions on Graphics 31, 4 (July 2012), 72:1--72:8.
16. Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. 2019. MeshCNN: A Network with an Edge. ACM Trans. Graph. 38, 4, Article 90 (July 2019), 12 pages.
17. Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. 2020. Point2Mesh: A Self-Prior for Deformable Meshes. ACM Trans. Graph. 39, 4, Article 126 (July 2020), 12 pages.
18. Nils Hasler, Thorsten Thormählen, Bodo Rosenhahn, and Hans-Peter Seidel. 2010. Learning skeletons for shape and pose. In Proceedings of the 2010 ACM SIGGRAPH symposium on Interactive 3D Graphics and Games. 23--30.
19. Jim Hejl. 2004. Hardware skinning with quaternions. Game Programming Gems 4 (2004), 487--495.
20. Hugues Hoppe. 1996. Progressive meshes. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. 99--108.
21. Yixin Hu, Teseo Schneider, Bolun Wang, Denis Zorin, and Daniele Panozzo. 2020. Fast Tetrahedral Meshing in the Wild. ACM Trans. Graph. 39, 4, Article 117 (July 2020), 18 pages.
22. Alec Jacobson, Ilya Baran, Jovan Popovic, and Olga Sorkine. 2011. Bounded biharmonic weights for real-time deformation. ACM Trans. Graph. 30, 4 (2011), 78.
23. Doug L. James and Christopher D. Twigg. 2005. Skinning mesh animations. In ACM SIGGRAPH 2005 Papers (SIGGRAPH '05). Association for Computing Machinery, New York, NY, USA, 399--407.
24. Pushkar Joshi, Mark Meyer, Tony DeRose, Brian Green, and Tom Sanocki. 2007. Harmonic coordinates for character articulation. ACM Transactions on Graphics 26, 3 (July 2007), 71--es.
25. Tao Ju, Scott Schaefer, and Joe Warren. 2005. Mean value coordinates for closed triangular meshes. ACM Transactions on Graphics 24, 3 (Jul 2005), 561--566.
26. Ladislav Kavan, Steven Collins, Jiří Žára, and Carol O'Sullivan. 2007. Skinning with dual quaternions. In Proceedings of the 2007 symposium on Interactive 3D graphics and games. 39--46.
27. Ladislav Kavan and Olga Sorkine. 2012. Elasticity-Inspired Deformers for Character Articulation. ACM Trans. Graph. 31, 6 (2012), 196:1--196:8.
28. Ladislav Kavan and Jiří Žára. 2005. Spherical blend skinning: a real-time deformation of articulated models. In Proceedings of the 2005 symposium on Interactive 3D graphics and games. 9--16.
29. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
30. Paul G. Kry, Doug L. James, and Dinesh K. Pai. 2002. EigenSkin: real time large deformation character skinning in hardware. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation (SCA '02). Association for Computing Machinery, New York, NY, USA, 153--159.
31. Binh Huy Le and Zhigang Deng. 2014. Robust and accurate skeletal rigging from mesh sequences. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1--10.
32. Binh Huy Le and Jessica K Hodgins. 2016. Real-time skeletal skinning with optimized centers of rotation. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1--10.
33. John P Lewis, Matt Cordner, and Nickson Fong. 2000. Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. 165--172.
34. Tianxing Li, Rui Shi, and Takashi Kanai. 2020. DenseGATs: A Graph-Attention-Based Network for Nonlinear Character Deformation. In Symposium on Interactive 3D Graphics and Games. 1--9.
35. Lijuan Liu, Youyi Zheng, Di Tang, Yi Yuan, Changjie Fan, and Kun Zhou. 2019. Neuroskinning: Automatic skin binding for production characters with deep graph networks. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--12.
36. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. 2015. SMPL: A skinned multi-person linear model. ACM transactions on graphics (TOG) 34, 6 (2015), 1--16.
37. Nadia Magnenat-Thalmann, Richard Laperrire, and Daniel Thalmann. 1988. Joint-dependent local deformations for hand animation and object grasping. In In Proceedings on Graphics interface'88. Citeseer.
38. Bruce Merry, Patrick Marais, and James Gain. 2006. Animation space: A truly linear framework for character animation. ACM Transactions on Graphics (TOG) 25, 4 (2006), 1400--1423.
39. Christian Miller, Okan Arikan, and Don Fussell. 2010. Frankenrigs: building character rigs from multiple sources. In Proceedings of the 2010 ACM SIGGRAPH symposium on Interactive 3D Graphics and Games. 31--38.
40. Tomohiko Mukai and Shigeru Kuriyama. 2016. Efficient dynamic skinning with low-rank helper bone controllers. ACM Transactions on Graphics 35, 4 (July 2016), 36:1--36:11.
41. Tina O'Hailey. 2018. Rig it right! Maya animation rigging concepts. Routledge.
42. Ahmed A A Osman, Timo Bolkart, and Michael J. Black. 2020. STAR: A Spare Trained Articulated Human Body Regressor. In European Conference on Computer Vision (ECCV). https://star.is.tue.mpg.de
43. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024--8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
44. Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 652--660.
45. Autodesk research. 2020. Meshmixer. https://www.meshmixer.com/
46. Scott Schaefer and Can Yuksel. 2007. Example-based skeleton extraction. In Symposium on Geometry Processing. 153--162.
47. Jaewoo Seo, Geoffrey Irving, J. P. Lewis, and Junyong Noh. 2011. Compression and direct manipulation of complex blendshape models. In Proceedings of the 2011 SIGGRAPH Asia Conference (SA '11). Association for Computing Machinery, New York, NY, USA, 1--10.
48. Peter-Pike J. Sloan, Charles F. Rose, and Michael F. Cohen. 2001. Shape by example. In Proceedings of the 2001 symposium on Interactive 3D graphics (I3D '01). Association for Computing Machinery, New York, NY, USA, 135--143.
49. Steven L Song, Weiqi Shi, and Michael Reed. 2020. Accurate Face Rig Approximation with Deep Differential Subspace Reconstruction. arXiv preprint arXiv:2006.01746 (2020).
50. Robert W Sumner, Matthias Zwicker, Craig Gotsman, and Jovan Popović. 2005. Meshbased inverse kinematics. ACM transactions on graphics (TOG) 24, 3 (2005), 488--495.
51. Xiaohuan Corina Wang and Cary Phillips. 2002. Multi-weight enveloping: least-squares approximation techniques for skin animation. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation. 129--138.
52. Rich Wareham and Joan Lasenby. 2008. Bone Glow: An Improved Method for the Assignment of Weights for Mesh Deformation. In Articulated Motion and Deformable Objects (Lecture Notes in Computer Science), Francisco J. Perales and Robert B. Fisher (Eds.). Springer, Berlin, Heidelberg, 63--71.
53. Ofir Weber, Olga Sorkine, Yaron Lipman, and Craig Gotsman. 2007. Context-aware skeletal shape deformation. In Computer Graphics Forum, Vol. 26. Wiley Online Library, 265--274.
54. Zhan Xu, Yang Zhou, Evangelos Kalogerakis, Chris Landreth, and Karan Singh. 2020. RigNet: Neural Rigging for Articulated Characters. ACM Transactions on Graphics 39, 4 (July 2020).
55. Wang Yifan, Noam Aigerman, Vladimir G. Kim, Siddhartha Chaudhuri, and Olga Sorkine-Hornung. 2020. Neural Cages for Detail-Preserving 3D Deformations. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Seattle, WA, USA, 72--80.
56. Jiayi Eris Zhang, Seungbae Bang, David I. W. Levin, and Alec Jacobson. 2020. Complementary Dynamics. ACM Transactions on Graphics 39, 6 (Nov. 2020), 179:1--179:11.

Index Terms

• Computing methodologies

  • Computer graphics

    • Animation

  • Machine learning

    • Machine learning approaches