Abstract
We present a novel method to enrich existing vertex-based human body models by adding soft-tissue dynamics. Our model learns to predict per-vertex 3D offsets, referred to as dynamic blendshapes, that reproduce nonlinear mesh deformation effects as a function of pose information. This enables the synthesis of realistic 3D mesh animations, including soft-tissue effects, using just skeletal motion. At the core of our method there is a neural network regressor trained on high-quality 4D scans from which we extract pose, shape and soft-tissue information. Our regressor uses a novel nonlinear subspace, which we build using an autoencoder, to efficiently compact soft-tissue dynamics information. Once trained, our method can be plugged to existing vertex-based skinning methods with little computational overhead (<10ms), enabling real-time nonlinear dynamics. We qualitatively and quantitatively evaluate our method, and show compelling animations with soft-tissue effects, created using publicly available motion capture datasets.
- Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-scale Machine Learning. In Conference on Operating Systems Design and Implementation. 265--283. Google Scholar
Digital Library
- Brett Allen, Brian Curless, and Zoran Popović. 2002. Articulated body deformation from range scan data. In ACM Transactions on Graphics (TOG), Vol. 21. ACM, 612--619. Google Scholar
Digital Library
- Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. 2005. SCAPE: Shape Completion and Animation of People. ACM Transactions on Graphics (TOG) 24, 3 (2005), 408--416. Google Scholar
Digital Library
- Jernej Barbič and Doug L. James. 2005. Real-time subspace integration for St. Venant-Kirchhoff deformable models. ACM Transactions on Graphics (Proc. of SIGGRAPH) 24, 3 (2005), 982--990. Google Scholar
Digital Library
- Federica Bogo, Javier Romero, Matthew Loper, and Michael J Black. 2014. FAUST: Dataset and evaluation for 3D mesh registration. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3794--3801. Google Scholar
Digital Library
- Federica Bogo, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2017. Dynamic FAUST: Registering Human Bodies in Motion. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR).Google Scholar
- Chris Budd, Peng Huang, Martin Klaudiny, and Adrian Hilton. 2013. Global non-rigid alignment of surface sequences. International Journal of Computer Vision 102, 1-3 (2013), 256--270. Google Scholar
Digital Library
- Steve Capell, Seth Green, Brian Curless, Tom Duchamp, and Zoran Popović. 2002. Interactive skeleton-driven dynamic deformations. ACM Transactions on Graphics (Proc. of SIGGRAPH) 21, 3 (2002), 586--593. Google Scholar
Digital Library
- Dan Casas, Marco Volino, John Collomosse, and Adrian Hilton. 2014. 4D Video Textures for Interactive Character Appearance. Computer Graphics Forum (Proc. Eurographics) 33, 2 (2014), 371--380. Google Scholar
Digital Library
- Yinpeng Chen, Zicheng Liu, and Zhengyou Zhang. 2013. Tensor-based Human Body Modeling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 105--112. Google Scholar
Digital Library
- CMU. 2003. CMU: Carnegie-Mellon Mocap Database. In http://mocap.cs.cmu.edu.Google Scholar
- Edilson De Aguiar, Leonid Sigal, Adrien Treuille, and Jessica K Hodgins. 2010. Stable Spaces for Real-Time Clothing. ACM Transactions on Graphics (TOG) 29, 4 (2010), 106. Google Scholar
Digital Library
- Andrew Feng, Dan Casas, and Ari Shapiro. 2015. Avatar Reshaping and Automatic Rigging Using a Deformable Model. In ACM SIGGRAPH Conference on Motion in Games (MIG). 57--64. Google Scholar
Digital Library
- Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. 2015. Recurrent network models for human dynamics. In IEEE International Conference on Computer Vision (ICCV). 4346--4354. Google Scholar
Digital Library
- Ikhsanul Habibie, Daniel Holden, Jonathan Schwarz, Joe Yearsley, and Taku Komura. 2017. Recurrent Variational Autoencoder for Human Motion Synthesis. In BMVC17.Google Scholar
- Fabian Hahn, Sebastian Martin, Bernhard Thomaszewski, Robert W. Sumner, Stelian Coros, and Markus Gross. 2012. Rig-space physics. ACM Transactions on Graphics (Proc. SIGGRAPH) 31, 4 (2012). Google Scholar
Digital Library
- Fabian Hahn, Bernhard Thomaszewski, Stelian Coros, Robert W. Sumner, Forrester Cole, Mark Meyer, Tony DeRose, and Markus Gross. 2014. Subspace clothing simulation using adaptive bases. ACM Transactions on Graphics (TOG) 33, 4 (2014), 105. Google Scholar
Digital Library
- Fabian Hahn, Bernhard Thomaszewski, Stelian Coros, Robert W. Sumner, and Markus Gross. 2013. Efficient simulation of secondary motion in rig-space. In ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 165--171. Google Scholar
Digital Library
- Nils Hasler, Carsten Stoll, Martin Sunkel, Bodo Rosenhahn, and H-P Seidel. 2009. A statistical model of human pose and body shape. In Computer Graphics Forum (Proc. of Eurographics), Vol. 28. 337--346.Google Scholar
Cross Ref
- Geoffrey E. Hinton and Ruslan R Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science 313, 5786 (2006), 504--507.Google Scholar
- David A. Hirshberg, Matthew Loper, Eric Rachlin, and Michael J Black. 2012. Coregistration: Simultaneous alignment and modeling of articulated 3D shape. In European Conference on Computer Vision (ECCV). 242--255. Google Scholar
Digital Library
- Daniel Holden, Taku Komura, and Jun Saito. 2017. Phase-functioned Neural Networks for Character Control. ACM Transactions on Graphics (Proc. SIGGRAPH) 36, 4 (2017). Google Scholar
Digital Library
- Chun-Hao Huang, Benjamin Allain, Edmond Boyer, Jean-Sébastien Franco, Federico Tombari, Nassir Navab, and Slobodan Ilic. 2017. Tracking-by-detection of 3d human shapes: from surfaces to volumes. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2017).Google Scholar
- Alec Jacobson and Olga Sorkine. 2011. Stretchable and twistable bones for skeletal shape deformation. ACM Transactions on Graphics (TOG) 30, 6 (2011). Google Scholar
Digital Library
- Doug L. James and Christopher D Twigg. 2005. Skinning mesh animations. ACM Transactions on Graphics (TOG) 24, 3 (2005), 399--407. Google Scholar
Digital Library
- Petr Kadleček, Alexandru-Eugen Ichim, Tiantian Liu, Jaroslav Křivánek, and Ladislav Kavan. 2016. Reconstructing personalized anatomical models for physics-based body animation. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 35, 6 (2016), 213. Google Scholar
Digital Library
- Ladislav Kavan, Steven Collins, Jiři Žára, and Carol O'Sullivan. 2008. Geometric skinning with approximate dual quaternion blending. ACM Transactions on Graphics (TOG) 27, 4 (2008). Google Scholar
Digital Library
- Ladislav Kavan, Peter-Pike Sloan, and Carol O'Sullivan. 2010. Fast and Efficient Skinning of Animated Meshes. Computer Graphics Forum 29, 2 (2010), 327--336.Google Scholar
Cross Ref
- Meekyoung Kim, Gerard Pons-Moll, Sergi Pujades, Sungbae Bang, Jinwwok Kim, Michael Black, and Sung-Hee Lee. 2017. Data-Driven Physics for Human Soft Tissue Animation. ACM Transactions on Graphics, (Proc. SIGGRAPH) 36, 4 (2017). Google Scholar
Digital Library
- Theodore Kim and Doug L. James. 2012. Physics-based character skinning using multidomain subspace deformations. IEEE Transactions on Visualization and Computer Graphics 18, 8 (2012), 1228--1240. Google Scholar
Digital Library
- Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
- Diederik P. Kingma and Max Welling. 2013. Auto-Encoding Variational Bayes. arXiv preprint arXiv:1312.6114 (2013).Google Scholar
- Paul G. Kry, Doug L. James, and Dinesh K. Pai. 2002. Eigenskin: real time large deformation character skinning in hardware. In ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA). ACM, 153--159. Google Scholar
Digital Library
- Lubor Ladický, SoHyeon Jeong, Barbara Solenthaler, Marc Pollefeys, and Markus Gross. 2015. Data-driven Fluid Simulations Using Regression Forests. ACM Trans. Graph. 34, 6 (2015), 199:1--199:9. Google Scholar
Digital Library
- Christoph Lassner, Gerard Pons-Moll, and Peter V. Gehler. 2017. A Generative Model of People in Clothing. In IEEE International Conference on Computer Vision (ICCV).Google Scholar
- Binh Huy Le and Zhigang Deng. 2012. Smooth Skinning Decomposition with Rigid Bones. ACM Transactions on Graphics (TOG) 31, 6 (2012), 199. Google Scholar
Digital Library
- Binh Huy Le and Zhigang Deng. 2014. Robust and accurate skeletal rigging from mesh sequences. ACM Transactions on Graphics (TOG) 33, 4 (2014). Google Scholar
Digital Library
- Binh Huy Le and Jessica K Hodgins. 2016. Real-time skeletal skinning with optimized centers of rotation. ACM Transactions on Graphics (TOG) 35, 4 (2016), 37. Google Scholar
Digital Library
- John P. Lewis, Matt Cordner, and Nickson Fong. 2000. Pose Space Deformation: a unified approach to shape interpolation and skeleton-driven deformation. In Conference on Computer Graphics and Interactive Techniques. 165--172. Google Scholar
Digital Library
- Libin Liu, KangKang Yin, Bin Wang, and Baining Guo. 2013. Simulation and Control of Skeleton-driven Soft Body Characters. ACM Trans. Graph. 32, 6, Article 215 (Nov. 2013), 8 pages. Google Scholar
Digital Library
- Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2015. SMPL: A Skinned Multi-Person Linear Model. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 34, 6 (2015), 248:1--248:16. Google Scholar
Digital Library
- Nadia Magnenat-Thalmann, Richard Laperrire, and Daniel Thalmann. 1988. Joint-dependent local deformations for hand animation and object grasping. In Proceedings on Graphics Interface. Google Scholar
Digital Library
- Timothy Masters. 1993. Practical neural network recipes in C++. Morgan Kaufmann. Google Scholar
Digital Library
- Rajaditya Mukherjee, Xiaofeng Wu, and Huamin Wang. 2016. Incremental Deformation Subspace Reconstruction. Computer Graphics Forum 35, 7 (2016), 169--178. Google Scholar
Digital Library
- Leonid Pishchulin, Stefanie Wuhrer, Thomas Helten, Christian Theobalt, and Bernt Schiele. 2017. Building statistical shape spaces for 3D human modeling. Pattern Recognition 67 (2017), 276--286. Google Scholar
Digital Library
- Gerard Pons-Moll, Javier Romero, Naureen Mahmood, and Michael J Black. 2015. Dyna: A model of dynamic human shape in motion. ACM Transactions on Graphics, (Proc. SIGGRAPH) 34, 4 (2015). Google Scholar
Digital Library
- Nadia Robertini, Dan Casas, Helge Rhodin, Hans-Peter Seidel, and Christian Theobalt. 2016. Model-Based Outdoor Performance Capture. In International Conference on 3D Vision (3DV).Google Scholar
- Eftychios Sifakis, Igor Neverov, and Ronald Fedkiw. 2005. Automatic Determination of Facial Muscle Activations from Sparse Motion Capture Marker Data. ACM Trans. Graph. 24, 3 (July 2005), 417--425. Google Scholar
Digital Library
- Peter-Pike Sloan, Charles F. Rose III, and Michael F. Cohen. 2001. Shape by example. In Symposium on Interactive 3D Graphics (i3D). ACM, 135--143. Google Scholar
Digital Library
- Ayush Tewari, Michael Zollhofer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Christian Theobalt. 2017. MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction. In IEEE International Conference on Computer Vision (ICCV).Google Scholar
- Matthew Trumble, Andrew Gilbert, Charles Malleson, Adrian Hilton, and John Collomosse. 2017. Total Capture: 3D Human Pose Estimation Fusing Video and Inertial Sensors. In BMVC17.Google Scholar
- Rodolphe Vaillant, Loïc Barthe, Gaël Guennebaud, Marie-Paule Cani, Damien Rohmer, Brian Wyvill, Olivier Gourmel, and Mathias Paulin. 2013. Implicit skinning: real-time skin deformation with contact modeling. ACM Transactions on Graphics (TOG) 32, 4 (2013), 125. Google Scholar
Digital Library
- Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In ACM International Conference on Machine Learning (ICML). 1096--1103. Google Scholar
Digital Library
- Xiaohuan Corina Wang and Cary Phillips. 2002. Multi-weight enveloping: least-squares approximation techniques for skin animation. In ACM SIGGRAPH/Eurographics Symposium on Computer animation (SCA). 129--138. Google Scholar
Digital Library
- Hongyi Xu and Jernej Barbič. 2016. Pose-Space Subspace Dynamics. ACM Transactions on Graphics (Proc. SIGGRAPH) 35, 4 (2016). Google Scholar
Digital Library
- Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee. 2016. Attribute2image: Conditional image generation from visual attributes. In European Conference on Computer Vision (ECCV). 776--791.Google Scholar
Cross Ref
Index Terms
Learning Nonlinear Soft-Tissue Dynamics for Interactive Avatars
Recommendations
SMPL: a skinned multi-person linear model
We present a learned model of human body shape and pose-dependent shape variation that is more accurate than previous models and is compatible with existing graphics pipelines. Our Skinned Multi-Person Linear model (SMPL) is a skinned vertex-based model ...
Dynamic skinning for popping dance
ICEC'06: Proceedings of the 5th international conference on Entertainment ComputingThis paper presents an efficient technique to enhance the realism of character animation by adding muscle dynamics. Focusing on the isometric contraction of muscles, the proposed algorithm takes normal mesh and clenched mesh, and uses the disparity ...
State of the Art in Skinning Techniques for Articulated Deformable Characters
GRAPP 2016: Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications: Volume 1: GRAPPSkinning is an indispensable component of the content creation pipeline for character animation in the context
of feature films, video games, and in the special effects industry. Skinning techniques define the deformation
of the character skin for every ...






Comments