Scalable muscle-actuated human simulation and control

Abstract

Many anatomical factors, such as bone geometry and muscle condition, interact to affect human movements. This work aims to build a comprehensive musculoskeletal model and its control system that reproduces realistic human movements driven by muscle contraction dynamics. The variations in the anatomic model generate a spectrum of human movements ranging from typical to highly stylistic movements. To do so, we discuss scalable and reliable simulation of anatomical features, robust control of under-actuated dynamical systems based on deep reinforcement learning, and modeling of pose-dependent joint limits. The key technical contribution is a scalable, two-level imitation learning algorithm that can deal with a comprehensive full-body musculoskeletal model with 346 muscles. We demonstrate the predictive simulation of dynamic motor skills under anatomical conditions including bone deformity, muscle weakness, contracture, and the use of a prosthesis. We also simulate various pathological gaits and predictively visualize how orthopedic surgeries improve post-operative gaits.

References

1. Ijaz Akhter and Michael J. Black. 2015. Pose-conditioned joint angle limits for 3D human pose reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1446--1455.
2. Edith M Arnold, Samuel R Ward, Richard L Lieber, and Scott L Delp. 2010. A model of the lower limb for analysis of human movement. Annals of biomedical engineering 38, 2 (2010), 269--279.
3. Pierre-Luc Bacon, Jean Harb, and Doina Precup. 2017. The option-critic architecture. In Thirty-First AAAI Conference on Artificial Intelligence.
4. Glen Berseth, Cheng Xie, Paul Cernek, and Michiel Van de Panne. 2018. Progressive reinforcement learning with distillation for multi-skilled motion control. arXiv preprint arXiv.1802.04765 (2018).
5. Alexander Clegg, Wenhao Yu, Jie Tan, C Karen Liu, and Greg Turk. 2018. Learning to dress: Synthesizing human dressing motion via deep reinforcement learning. ACM Transactions on Graphics 37, 6 (2018).
6. Michael Damsgaard, John Rasmussen, Søren T. Christensen, Egidijus Surma, and Mark D. Zee. 2006. Analysis of musculoskeletal systems in the AnyBody Modeling System. Simulation Modelling Practice and Theory 14, 8 (2006), 1100 -- 1111.
7. Scott L. Delp, Frank C. Anderson, Anderson S. Arnold, Peter Loan, Ayman Habib, T. John, Eran Guendelman, and Darryl G. Thelen. 2007. OpenSim: Open-Source Software to Create and Analyze Dynamic Simulations of Movement. IEEE Transactions on Biomedical Engineering 54, 11 (2007), 1940--1950.
8. Danny Driess, Heiko Zimmermann, Simon Wolfen, Dan Suissa, Daniel Haeufle, Daniel Hennes, Marc Toussaint, and Syn Schmitt. 2018. Learning to Control Redundant Musculoskeletal Systems with Neural Networks and SQP: Exploiting Muscle Properties. In 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 6461--6468.
9. Thomas Geijtenbeek, Michiel van de Panne, and A. Frank van der Stappen. 2013. Flexible Muscle-Based Locomotion for Bipedal Creatures. ACM Transactions on Graphics 32, 6 (2013).
10. Daseong Han, Junyong Noh, Xiaogang Jin, Joseph S. Shin, and Sung Y. Shin. 2014. On-line real-time physics-based predictive motion control with balance recovery. In Computer Graphics Forum, Vol. 33. 245--254.
11. Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Ali Eslami, Martin Riedmiller, et al. 2017. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286 (2017).
12. Yifeng Jiang and C Karen Liu. 2018. Data-driven approach to simulating realistic human joint constraints. In 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 1098--1103.
13. Petr Kadleček, Alexandru-Eugen Ichim, Tiantian Liu, Jaroslav Křivánek, and Ladislav Kavan. 2016. Reconstructing personalized anatomical models for physics-based body animation. ACM Transactions on Graphics 35, 6, Article 213 (2016).
14. Łukasz Kidziński, Sharada Prasanna Mohanty, Carmichael Ong, Zhewei Huang, et al. 2018. Learning to Run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments. arXiv preprint arXiv:1804.00361 (2018).
15. Meekyoung Kim, Gerard Pons-Moll, Sergi Pujades, Seungbae Bang, Jinwook Kim, Michael J. Black, and Sung-Hee Lee. 2017. Data-driven Physics for Human Soft Tissue Animation. ACM Transactions on Graphics 36, 4, Article 54 (2017), 12 pages.
16. Taku Komura, Yoshihisa Shinagawa, and Tosiyasu L Kunii. 2000. Creating and retargetting motion by the musculoskeletal human body model. The Visual Computer 16, 5 (2000), 254--270.
17. Jehee Lee. 2008. Representing Rotations and Orientations in Geometric Computing. IEEE Computer Graphics and Applications 28, 2 (2008), 75--83.
18. Jeongseok Lee, Michael X. Grey, Sehoon Ha, Tobias Kunz, Sumit Jain, Yuting Ye, Siddhartha S. Srinivasa, Mike Stilman, and C. Karen Liu. 2018a. DART: Dynamic Animation and Robotics Toolkit. The Journal of Open Source Software 3, 22 (2018), 500.
19. Sunghee Lee, Eftychios Sifakis, and Demetri Terzopoulos. 2009. Comprehensive biomechanical modeling and simulation of the upper body. ACM Transactions on Graphics 28, 4, Article 99 (2009).
20. Seunghwan Lee, Ri Yu, Jungnam Park, Mridul Aanjaneya, Eftychios Sifakis, and Jehee Lee. 2018b. Dexterous Manipulation and Control with Volumetric Muscles. ACM Transactions on Graphics 37, 4, Article 57 (2018), 13 pages.
21. Sung Hee Lee and Demetri Terzopoulos. 2006. Heads Up!: Biomechanical Modeling and Neuromuscular Control of the Neck. ACM Transactions on Graphics 25, 3 (2006), 1188--1198.
22. Yoonsang Lee, Sungeun Kim, and Jehee Lee. 2010. Data-driven biped control. ACM Transactions on Graphics 29, 4 (2010), 129.
23. Yoonsang Lee, Kyungho Lee, Soon-Sun Kwon, Jiwon Jeong, Carol O'Sullivan, Moon Seok Park, and Jehee Lee. 2015. Push-Recovery Stability of Biped Locomotion. ACM Transactions on Graphics 34, 6 (2015).
24. Yoonsang Lee, Moon Seok Park, Taesoo Kwon, and Jehee Lee. 2014. Locomotion control for many-muscle humanoids. ACM Transactions on Graphics 33, 6, Article 218 (2014).
25. Andrew Levy, Robert Platt, and Kate Saenko. 2019. Hierarchical reinforcement learning with hindsight. In Proceedings of the International Conference on Learning Representations (ICLR).
26. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015).
27. Libin Liu and Jessica Hodgins. 2017. Learning to Schedule Control Fragments for Physics-Based Characters Using Deep Q-Learning. ACM Transactions on Graphics 36, 3 (2017).
28. Libin Liu and Jessica Hodgins. 2018. Learning Basketball Dribbling Skills Using Trajectory Optimization and Deep Reinforcement Learning. ACM Transactions on Graphics 37, 4 (2018).
29. John E. Lloyd, Ian Stavness, and Sidney Fels. 2012. ARTISYNTH: a fast interactive biomechanical modeling toolkit combining multibody and finite element simulation.
30. Josh Merel, Yuval Tassa, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. 2017. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv:1707.02201 (2017).
31. Brian Mirtich and John Canny. 1995. Impulse-based simulation of rigid bodies. In Proceedings of the 1995 symposium on Interactive 3D graphics. 181-ff.
32. Masaki Nakada, Tao Zhou, Honglin Chen, Tomer Weiss, and Demetri Terzopoulos. 2018. Deep Learning of Biomimetic Sensorimotor Control for Biomechanical Human Animation. ACM Transactions on Graphics 37, 4, Article 56 (2018), 15 pages.
33. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS-W.
34. Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. 2018a. Deep-Mimic: Example-guided Deep Reinforcement Learning of Physics-based Character Skills. ACM Transactions on Graphics 37, 4, Article 143 (2018).
35. Xue Bin Peng, Glen Berseth, Kangkang Yin, and Michiel Van De Panne. 2017. DeepLoco: Dynamic Locomotion Skills Using Hierarchical Deep Reinforcement Learning. ACM Transaction on Graphics 36, 4, Article 41 (2017).
36. Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. 2018b. SFV: Reinforcement Learning of Physical Skills from Videos. ACM Transactions on Graphics 37, 6, Article 178 (2018).
37. Xue Bin Peng and Michiel van de Panne. 2017. Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?. In Proceedings of the ACM SIGGRAPH / Eurographics Symposium on Computer Animation (SCA '17). Article 12, 13 pages.
38. Prashant Sachdeva, Shinjiro Sueda, Susanne Bradley, Mikhail Fain, and Dinesh K Pai. 2015. Biomechanical simulation and control of hands and tendinous systems. ACM Transactions on Graphics 34, 4, Article 42 (2015).
39. Shunsuke Saito, Zi-Ye Zhou, and Ladislav Kavan. 2015. Computational bodybuilding: Anatomically-based modeling of human bodies. ACM Transactions on Graphics 34, 4, Article 41 (2015).
40. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015. Trust region policy optimization. In International Conference on Machine Learning. 1889--1897.
41. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).
42. Eftychios Sifakis, Igor Neverov, and Ronald Fedkiw. 2005. Automatic determination of facial muscle activations from sparse motion capture marker data. ACM Transactions on Graphics 24, 3 (2005).
43. Kwang Won Sok, Manmyung Kim, and Jehee Lee. 2007. Simulating biped behaviors from human motion data. ACM Transactions on Graphics 26, 3 (2007).
44. Shinjiro Sueda, Andrew Kaufman, and Dinesh K. Pai. 2008. Musculotendon Simulation for Hand Animation. ACM Transactions on Graphics 27, 3, Article 83 (2008), 8 pages.
45. Darryl G Thelen et al. 2003. Adjustment of muscle mechanics model parameters to simulate dynamic contractions in older adults. Transactions-American Society Of Mechanical Engineers Journal Of Biomechanical Engineering 125, 1 (2003), 70--77.
46. Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. 2017. Feudal networks for hierarchical reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70. 3540--3549.
47. Andreas Wachter and Lorenz T. Biegler. 2006. On the Implementation of an Interior-point Filter Line-search Algorithm for Large-scale Nonlinear Programming. Math. Program. 106, 1 (2006), 25--57.
48. Jack M. Wang, Samuel R. Hamner, Scott L. Delp, and Vladlen Koltun. 2012. Optimizing Locomotion Controllers Using Biologically-based Actuators and Objectives. ACM Transaction on Graphics 31, 4, Article 25 (2012).
49. Ziyu Wang, Josh S Merel, Scott E Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. 2017. Robust imitation of diverse behaviors. In Advances in Neural Information Processing Systems. 5320--5329.
50. Jungdam Won, Jongho Park, Kwanyu Kim, and Jehee Lee. 2017. How to Train Your Dragon: Example-guided Control of Flapping Flight. ACM Transactions on Graphics 36, 6, Article 198 (2017), 13 pages.
51. Jungdam Won, Jungnam Park, and Jehee Lee. 2018. Aerobatics Control of Flying Creatures via Self-regulated Learning. In SIGGRAPH Asia 2018 Technical Papers (SIGGRAPH Asia '18). Article 181.
52. KangKang Yin, Kevin Loken, and Michiel van de Panne. 2007. SIMBICON: Simple Biped Locomotion Control. ACM Transaction on Graphics 26, 3, Article 105 (2007).
53. Wenhao Yu, Greg Turk, and C. Karen Liu. 2018. Learning Symmetric and Low-energy Locomotion. ACM Transactions on Graphics 37, 4, Article 144 (2018), 12 pages.
54. Felix E Zajac. 1989. Muscle and tendon Properties models scaling and application to biomechanics and motor. Critical reviews in biomedical engineering 17, 4 (1989), 359--411.
55. Felix E Zajac, Richard R Neptune, and Steven A Kautz. 2003. Biomechanics and muscle coordination of human walking: part II: lessons from dynamical simulations and clinical implications. Gait & posture 17, 1 (2003), 1--17.

Index Terms

• Computing methodologies

  • Computer graphics

    • Animation

      • Motion processing

      • Physical simulation