skip to main content
research-article

Learning Nonlinear Soft-Tissue Dynamics for Interactive Avatars

Published:25 July 2018Publication History
Skip Abstract Section

Abstract

We present a novel method to enrich existing vertex-based human body models by adding soft-tissue dynamics. Our model learns to predict per-vertex 3D offsets, referred to as dynamic blendshapes, that reproduce nonlinear mesh deformation effects as a function of pose information. This enables the synthesis of realistic 3D mesh animations, including soft-tissue effects, using just skeletal motion. At the core of our method there is a neural network regressor trained on high-quality 4D scans from which we extract pose, shape and soft-tissue information. Our regressor uses a novel nonlinear subspace, which we build using an autoencoder, to efficiently compact soft-tissue dynamics information. Once trained, our method can be plugged to existing vertex-based skinning methods with little computational overhead (<10ms), enabling real-time nonlinear dynamics. We qualitatively and quantitatively evaluate our method, and show compelling animations with soft-tissue effects, created using publicly available motion capture datasets.

References

  1. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-scale Machine Learning. In Conference on Operating Systems Design and Implementation. 265--283. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Brett Allen, Brian Curless, and Zoran Popović. 2002. Articulated body deformation from range scan data. In ACM Transactions on Graphics (TOG), Vol. 21. ACM, 612--619. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. 2005. SCAPE: Shape Completion and Animation of People. ACM Transactions on Graphics (TOG) 24, 3 (2005), 408--416. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Jernej Barbič and Doug L. James. 2005. Real-time subspace integration for St. Venant-Kirchhoff deformable models. ACM Transactions on Graphics (Proc. of SIGGRAPH) 24, 3 (2005), 982--990. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Federica Bogo, Javier Romero, Matthew Loper, and Michael J Black. 2014. FAUST: Dataset and evaluation for 3D mesh registration. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3794--3801. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Federica Bogo, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2017. Dynamic FAUST: Registering Human Bodies in Motion. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR).Google ScholarGoogle Scholar
  7. Chris Budd, Peng Huang, Martin Klaudiny, and Adrian Hilton. 2013. Global non-rigid alignment of surface sequences. International Journal of Computer Vision 102, 1-3 (2013), 256--270. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Steve Capell, Seth Green, Brian Curless, Tom Duchamp, and Zoran Popović. 2002. Interactive skeleton-driven dynamic deformations. ACM Transactions on Graphics (Proc. of SIGGRAPH) 21, 3 (2002), 586--593. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Dan Casas, Marco Volino, John Collomosse, and Adrian Hilton. 2014. 4D Video Textures for Interactive Character Appearance. Computer Graphics Forum (Proc. Eurographics) 33, 2 (2014), 371--380. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Yinpeng Chen, Zicheng Liu, and Zhengyou Zhang. 2013. Tensor-based Human Body Modeling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 105--112. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. CMU. 2003. CMU: Carnegie-Mellon Mocap Database. In http://mocap.cs.cmu.edu.Google ScholarGoogle Scholar
  12. Edilson De Aguiar, Leonid Sigal, Adrien Treuille, and Jessica K Hodgins. 2010. Stable Spaces for Real-Time Clothing. ACM Transactions on Graphics (TOG) 29, 4 (2010), 106. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Andrew Feng, Dan Casas, and Ari Shapiro. 2015. Avatar Reshaping and Automatic Rigging Using a Deformable Model. In ACM SIGGRAPH Conference on Motion in Games (MIG). 57--64. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. 2015. Recurrent network models for human dynamics. In IEEE International Conference on Computer Vision (ICCV). 4346--4354. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Ikhsanul Habibie, Daniel Holden, Jonathan Schwarz, Joe Yearsley, and Taku Komura. 2017. Recurrent Variational Autoencoder for Human Motion Synthesis. In BMVC17.Google ScholarGoogle Scholar
  16. Fabian Hahn, Sebastian Martin, Bernhard Thomaszewski, Robert W. Sumner, Stelian Coros, and Markus Gross. 2012. Rig-space physics. ACM Transactions on Graphics (Proc. SIGGRAPH) 31, 4 (2012). Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Fabian Hahn, Bernhard Thomaszewski, Stelian Coros, Robert W. Sumner, Forrester Cole, Mark Meyer, Tony DeRose, and Markus Gross. 2014. Subspace clothing simulation using adaptive bases. ACM Transactions on Graphics (TOG) 33, 4 (2014), 105. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Fabian Hahn, Bernhard Thomaszewski, Stelian Coros, Robert W. Sumner, and Markus Gross. 2013. Efficient simulation of secondary motion in rig-space. In ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 165--171. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Nils Hasler, Carsten Stoll, Martin Sunkel, Bodo Rosenhahn, and H-P Seidel. 2009. A statistical model of human pose and body shape. In Computer Graphics Forum (Proc. of Eurographics), Vol. 28. 337--346.Google ScholarGoogle ScholarCross RefCross Ref
  20. Geoffrey E. Hinton and Ruslan R Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science 313, 5786 (2006), 504--507.Google ScholarGoogle Scholar
  21. David A. Hirshberg, Matthew Loper, Eric Rachlin, and Michael J Black. 2012. Coregistration: Simultaneous alignment and modeling of articulated 3D shape. In European Conference on Computer Vision (ECCV). 242--255. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Daniel Holden, Taku Komura, and Jun Saito. 2017. Phase-functioned Neural Networks for Character Control. ACM Transactions on Graphics (Proc. SIGGRAPH) 36, 4 (2017). Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Chun-Hao Huang, Benjamin Allain, Edmond Boyer, Jean-Sébastien Franco, Federico Tombari, Nassir Navab, and Slobodan Ilic. 2017. Tracking-by-detection of 3d human shapes: from surfaces to volumes. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2017).Google ScholarGoogle Scholar
  24. Alec Jacobson and Olga Sorkine. 2011. Stretchable and twistable bones for skeletal shape deformation. ACM Transactions on Graphics (TOG) 30, 6 (2011). Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Doug L. James and Christopher D Twigg. 2005. Skinning mesh animations. ACM Transactions on Graphics (TOG) 24, 3 (2005), 399--407. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Petr Kadleček, Alexandru-Eugen Ichim, Tiantian Liu, Jaroslav Křivánek, and Ladislav Kavan. 2016. Reconstructing personalized anatomical models for physics-based body animation. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 35, 6 (2016), 213. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Ladislav Kavan, Steven Collins, Jiři Žára, and Carol O'Sullivan. 2008. Geometric skinning with approximate dual quaternion blending. ACM Transactions on Graphics (TOG) 27, 4 (2008). Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Ladislav Kavan, Peter-Pike Sloan, and Carol O'Sullivan. 2010. Fast and Efficient Skinning of Animated Meshes. Computer Graphics Forum 29, 2 (2010), 327--336.Google ScholarGoogle ScholarCross RefCross Ref
  29. Meekyoung Kim, Gerard Pons-Moll, Sergi Pujades, Sungbae Bang, Jinwwok Kim, Michael Black, and Sung-Hee Lee. 2017. Data-Driven Physics for Human Soft Tissue Animation. ACM Transactions on Graphics, (Proc. SIGGRAPH) 36, 4 (2017). Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Theodore Kim and Doug L. James. 2012. Physics-based character skinning using multidomain subspace deformations. IEEE Transactions on Visualization and Computer Graphics 18, 8 (2012), 1228--1240. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980 (2014).Google ScholarGoogle Scholar
  32. Diederik P. Kingma and Max Welling. 2013. Auto-Encoding Variational Bayes. arXiv preprint arXiv:1312.6114 (2013).Google ScholarGoogle Scholar
  33. Paul G. Kry, Doug L. James, and Dinesh K. Pai. 2002. Eigenskin: real time large deformation character skinning in hardware. In ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA). ACM, 153--159. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Lubor Ladický, SoHyeon Jeong, Barbara Solenthaler, Marc Pollefeys, and Markus Gross. 2015. Data-driven Fluid Simulations Using Regression Forests. ACM Trans. Graph. 34, 6 (2015), 199:1--199:9. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Christoph Lassner, Gerard Pons-Moll, and Peter V. Gehler. 2017. A Generative Model of People in Clothing. In IEEE International Conference on Computer Vision (ICCV).Google ScholarGoogle Scholar
  36. Binh Huy Le and Zhigang Deng. 2012. Smooth Skinning Decomposition with Rigid Bones. ACM Transactions on Graphics (TOG) 31, 6 (2012), 199. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Binh Huy Le and Zhigang Deng. 2014. Robust and accurate skeletal rigging from mesh sequences. ACM Transactions on Graphics (TOG) 33, 4 (2014). Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Binh Huy Le and Jessica K Hodgins. 2016. Real-time skeletal skinning with optimized centers of rotation. ACM Transactions on Graphics (TOG) 35, 4 (2016), 37. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. John P. Lewis, Matt Cordner, and Nickson Fong. 2000. Pose Space Deformation: a unified approach to shape interpolation and skeleton-driven deformation. In Conference on Computer Graphics and Interactive Techniques. 165--172. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Libin Liu, KangKang Yin, Bin Wang, and Baining Guo. 2013. Simulation and Control of Skeleton-driven Soft Body Characters. ACM Trans. Graph. 32, 6, Article 215 (Nov. 2013), 8 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2015. SMPL: A Skinned Multi-Person Linear Model. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 34, 6 (2015), 248:1--248:16. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Nadia Magnenat-Thalmann, Richard Laperrire, and Daniel Thalmann. 1988. Joint-dependent local deformations for hand animation and object grasping. In Proceedings on Graphics Interface. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Timothy Masters. 1993. Practical neural network recipes in C++. Morgan Kaufmann. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Rajaditya Mukherjee, Xiaofeng Wu, and Huamin Wang. 2016. Incremental Deformation Subspace Reconstruction. Computer Graphics Forum 35, 7 (2016), 169--178. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Leonid Pishchulin, Stefanie Wuhrer, Thomas Helten, Christian Theobalt, and Bernt Schiele. 2017. Building statistical shape spaces for 3D human modeling. Pattern Recognition 67 (2017), 276--286. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Gerard Pons-Moll, Javier Romero, Naureen Mahmood, and Michael J Black. 2015. Dyna: A model of dynamic human shape in motion. ACM Transactions on Graphics, (Proc. SIGGRAPH) 34, 4 (2015). Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Nadia Robertini, Dan Casas, Helge Rhodin, Hans-Peter Seidel, and Christian Theobalt. 2016. Model-Based Outdoor Performance Capture. In International Conference on 3D Vision (3DV).Google ScholarGoogle Scholar
  48. Eftychios Sifakis, Igor Neverov, and Ronald Fedkiw. 2005. Automatic Determination of Facial Muscle Activations from Sparse Motion Capture Marker Data. ACM Trans. Graph. 24, 3 (July 2005), 417--425. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Peter-Pike Sloan, Charles F. Rose III, and Michael F. Cohen. 2001. Shape by example. In Symposium on Interactive 3D Graphics (i3D). ACM, 135--143. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Ayush Tewari, Michael Zollhofer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Christian Theobalt. 2017. MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction. In IEEE International Conference on Computer Vision (ICCV).Google ScholarGoogle Scholar
  51. Matthew Trumble, Andrew Gilbert, Charles Malleson, Adrian Hilton, and John Collomosse. 2017. Total Capture: 3D Human Pose Estimation Fusing Video and Inertial Sensors. In BMVC17.Google ScholarGoogle Scholar
  52. Rodolphe Vaillant, Loïc Barthe, Gaël Guennebaud, Marie-Paule Cani, Damien Rohmer, Brian Wyvill, Olivier Gourmel, and Mathias Paulin. 2013. Implicit skinning: real-time skin deformation with contact modeling. ACM Transactions on Graphics (TOG) 32, 4 (2013), 125. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In ACM International Conference on Machine Learning (ICML). 1096--1103. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Xiaohuan Corina Wang and Cary Phillips. 2002. Multi-weight enveloping: least-squares approximation techniques for skin animation. In ACM SIGGRAPH/Eurographics Symposium on Computer animation (SCA). 129--138. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Hongyi Xu and Jernej Barbič. 2016. Pose-Space Subspace Dynamics. ACM Transactions on Graphics (Proc. SIGGRAPH) 35, 4 (2016). Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee. 2016. Attribute2image: Conditional image generation from visual attributes. In European Conference on Computer Vision (ECCV). 776--791.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Learning Nonlinear Soft-Tissue Dynamics for Interactive Avatars

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!