Deep learning of biomimetic sensorimotor control for biomechanical human animation

Abstract

We introduce a biomimetic framework for human sensorimotor control, which features a biomechanically simulated human musculoskeletal model actuated by numerous muscles, with eyes whose retinas have nonuniformly distributed photoreceptors. The virtual human's sensorimotor control system comprises 20 trained deep neural networks (DNNs), half constituting the neuromuscular motor subsystem, while the other half compose the visual sensory subsystem. Directly from the photoreceptor responses, 2 vision DNNs drive eye and head movements, while 8 vision DNNs extract visual information required to direct arm and leg actions. Ten DNNs achieve neuromuscular control---2 DNNs control the 216 neck muscles that actuate the cervicocephalic musculoskeletal complex to produce natural head movements, and 2 DNNs control each limb; i.e., the 29 muscles of each arm and 39 muscles of each leg. By synthesizing its own training data, our virtual human automatically learns efficient, online, active visuomotor control of its eyes, head, and limbs in order to perform nontrivial tasks involving the foveation and visual pursuit of target objects coupled with visually-guided limb-reaching actions to intercept the moving targets, as well as to carry out drawing and writing tasks.

References

1. J. Bergstra, O. Breuleux, F. Bastien, P. Lamblin, R. Pascanu, G. Desjardins, J. Turian, D. Warde-Farley, and Y. Bengio. 2010. Theano: A CPU and GPU math compiler in Python. In Proc. 9th Python in Science Conference. Austin, TX, 1--7.
2. A.L. Cruz Ruiz, C. Pontonnier, N. Pronost, and G. Dumont. 2017. Muscle-based control for character animation. Computer Graphics Forum 36, 6 (2017), 122--147.
3. M. F. Deering. 2005. A photon accurate model of the human eye. ACM Transactions on Graphics 24, 3 (2005), 649--658.
4. P. Faloutsos, M. van de Panne, and D. Terzopoulos. 2001. Composable controllers for physics-based character animation. In Proc. 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '01). Los Angeles, CA, 251--260.
5. Y. Fan, J. Litven, and D.K. Pai. 2014. Active volumetric musculoskeletal systems. ACM Transactions on Graphics 33, 4 (2014), 152.
6. R. Featherstone. 2014. Rigid Body Dynamics Algorithms. Springer, New York, NY.
7. T. Geijtenbeek, M. Van De Panne, and A.F. Van Der Stappen. 2013. Flexible muscle-based locomotion for bipedal creatures. ACM Transactions on Graphics 32, 6 (2013), 206.
8. I. Goodfellow, Y. Bengio, and A. Courville. 2016. Deep Learning. MIT Press, Cambridge, MA.
9. R. Grzeszczuk, D. Terzopoulos, and G. Hinton. 1998. NeuroAnimator: Fast neural network emulation and control of physics-based models. In Computer Graphics Proceedings, Annual Conference Series. Orlando, FL, 9--20. Proc. ACM SIGGRAPH 98.
10. K. He, X. Zhang, S. Ren, and J. Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proc. IEEE International Conference on Computer Vision. Santiago, Chile, 1026--1034.
11. J.K. Hodgins, W.L. Wooten, D.C. Brogan, and J.F. O'Brien. 1995. Animating human athletics. In Proc. ACM SIGGRAPH '95 Conference. Los Angeles, CA, 71--78.
12. D. Holden, T. Komura, and J. Saito. 2017. Phase-functioned neural networks for character control. ACM Transactions on Graphics 36, 4 (2017), 42.
13. W. Huang, M. Kapadia, and D. Terzopoulos. 2010. Full-body hybrid motor control for reaching. In Motion in Games (Lecture Notes in Computer Science, Vol. 6459). Springer-Verlag, Berlin, 36--47.
14. A.-E. Ichim, P. Kadleček, L. Kavan, and M. Pauly. 2017. Phace: Physics-based face modeling and animation. ACM Transactions on Graphics 36, 4 (2017), 153.
15. P. Kadleček, A.-E. Ichim, T. Liu, J. Křivánek, and L. Kavan. 2016. Reconstructing personalized anatomical models for physics-based body animation. ACM Transactions on Graphics 35, 6 (2016), 213.
16. K. Kähler, J. Haber, H. Yamauchi, and H.-P. Seidel. 2002. Head shop: Generating animated head models with anatomical structure. In Proc. 2002 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. San Antonio, TX, 55--63.
17. D. Kingma and J. Ba. 2014. Adam: A method for stochastic optimization. Technical Report. arXiv preprint arXiv:1412.6980.
18. S.-H. Lee, E. Sifakis, and D. Terzopoulos. 2009. Comprehensive biomechanical modeling and simulation of the upper body. ACM Transactions on Graphics 28, 4 (2009), 99:1--17.
19. S.-H. Lee and D. Terzopoulos. 2006. Heads Up! Biomechanical modeling and neuromuscular control of the neck. ACM Transactions on Graphics 23, 212 (2006), 1188--1198. Proc. ACM SIGGRAPH 2006.
20. Y. Lee, M.S. Park, T. Kwon, and J. Lee. 2014. Locomotion control for many-muscle humanoids. ACM Transactions on Graphics 33, 6 (2014), 218.
21. Y. Lee, D. Terzopoulos, and K. Waters. 1995. Realistic modeling for facial animation. In Computer Graphics Proceedings, Annual Conference Series (Proc. ACM SIGGRAPH 95). Los Angleles, CA, 55--62.
22. M. Lesmana, A. Landgren, P.-E. Forssén, and D.K. Pai. 2014. Active gaze stabilization. In Proc. Indian Conference on Computer Vision, Graphics, and Image Processing. Bangalore, India, Article 81, 8 pages.
23. M. Lesmana and D.K. Pai. 2011. A biologically inspired controller for fast eye movements. In IEEE International Conference on Robotics and Automation (ICRA). IEEE, Shanghai, China, 3670--3675.
24. L. Liu and J. Hodgins. 2017. Learning to schedule control fragments for physics-based characters using deep Q-learning. ACM Transactions on Graphics 36, 3 (2017), 29.
25. M. Nakada, H. Chen, and D. Terzopoulos. 2018. Deep learning of biomimetic visual perception for virtual humans. In Proc. ACM Symposium on Applied Perception (SAP '18). Vancouver, BC, 1--8.
26. M. Nakada and D. Terzopoulos. 2015. Deep learning of neuromuscular control for biomechanical human animation. In Advances in Visual Computing (Lecture Notes in Computer Science, Vol. 9474). Springer, Berlin, 339--348. Proc. International Symposium on Visual Computing, Las Vegas, NV, December 2015.
27. X.B. Peng, G. Berseth, K. Yin, and M. Van De Panne. 2017. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Transactions on Graphics 36, 4 (2017), 41.
28. T.F. Rabie and D. Terzopoulos. 2000. Active perception in virtual humans. In Proc. Vision Interface 2000. Montreal, Canada, 16--22.
29. P. Sachdeva, S. Sueda, S. Bradley, M. Fain, and D.K. Pai. 2015. Biomechanical simulation and control of hands and tendinous systems. ACM Transactions on Graphics 34, 4 (2015), 42.
30. E.L. Schwartz. 1977. Spatial mapping in the primate sensory projection: Analytic structure and relevance to perception. Biological Cybernetics 25, 4 (1977), 181--194.
31. W. Si, S.-H. Lee, E. Sifakis, and D. Terzopoulos. 2014. Realistic biomechanical simulation and control of human swimming. ACM Transactions on Graphics 34, 1, Article 10 (Nov. 2014), 15 pages.
32. E. Sifakis, I. Neverov, and R. Fedkiw. 2005. Automatic determination of facial muscle activations from sparse motion capture marker data. ACM Transactions on Graphics 1, 212 (2005), 417--425.
33. S. Sueda, A. Kaufman, and D.K. Pai. 2008. Musculotendon simulation for hand animation. ACM Transactions on Graphics 27, 3 (Aug. 2008), 83.
34. D. Terzopoulos and T.F. Rabie. 1995. Animat vision: Active vision with artificial animals. In Proc. Fifth International Conference on Computer Vision (ICCV '95). Cambridge, MA, 840--845.
35. D. Terzopoulos and K. Waters. 1990. Physically-based facial modelling, analysis, and animation. Computer Animation and Virtual Worlds 1, 2 (1990), 73--80.
36. J.M. Wang, S.R. Hamner, S.L. Delp, and V. Koltun. 2012. Optimizing locomotion controllers using biologically-based actuators and objectives. ACM Transactions on Graphics 31, 4, Article 25 (2012), 11 pages.
37. Q. Wei, S. Sueda, and D.K. Pai. 2010. Biomechanical simulation of human eye movement. In Biomedical Simulation (Lecture Notes in Computer Science), Vol. 5958. Springer-Verlag, Berlin, 108--118.
38. S.H. Yeo, M. Lesmana, D.R. Neog, and D.K. Pai. 2012. Eyecatch: Simulating visuomotor coordination for object interception. ACM Transactions on Graphics 31, 4 (2012), 1--10.

Index Terms

• Computing methodologies

  • Computer graphics

    • Animation

      • Physical simulation

  • Machine learning

    • Machine learning approaches

      • Bio-inspired approaches

        • Artificial life

      • Neural networks