Linear Bellman combination for control of character animation

Abstract

Controllers are necessary for physically-based synthesis of character animation. However, creating controllers requires either manual tuning or expensive computer optimization. We introduce linear Bellman combination as a method for reusing existing controllers. Given a set of controllers for related tasks, this combination creates a controller that performs a new task. It naturally weights the contribution of each component controller by its relevance to the current state and goal of the system. We demonstrate that linear Bellman combination outperforms naive combination often succeeding where naive combination fails. Furthermore, this combination is provably optimal for a new task if the component controllers are also optimal for related tasks. We demonstrate the applicability of linear Bellman combination to interactive character control of stepping motions and acrobatic maneuvers.

References

1. Abbeel, P., and Ng, A. Y. 2004. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine learning (ICML), ACM, vol. 69, 1:1--1:8.
2. Atkeson, C. G., and Morimoto, J. 2002. Nonparametric representation of policies and value functions: A trajectory-based approach. In Advances in Neural Information Processing Systems (NIPS), vol. 15, 1611--1618.
3. Atkeson, C. G. 1994. Using local trajectory optimizers to speed up global optimization in dynamic programming. In Advances in Neural Information Processing Systems (NIPS), vol. 6, 663--670.
4. Barbič, J., and Popović, J. 2008. Real-time control of physically based simulations using gentle forces. ACM Transactions on Graphics 27, 5, 163:1--163:10.
5. Bellman, R. E. 1957. Dynamic Programming. Princeton University Press, Princeton, NJ.
6. Bergou, M., Mathur, S., Wardetzky, M., and Grinspun, E. 2007. Tracks: toward directable thin shells. ACM Transactions on Graphics 26, 3, 50:1--50:10.
7. Bertsekas, D. P. 2007. Dynamic Programming and Optimal Control, 3 ed., vol. I. Athena Scientific, Nashua, NH.
8. Brotman, L. S., and Netravali, A. N. 1988. Motion interpolation by optimal control. In Computer Graphics (Proceedings of SIGGRAPH 88), 309--315.
9. Burridge, R. R., Rizzi, A. A., and Koditschek, D. E. 1999. Sequential composition of dynamically desterous robot behaviours. International Journal of Robotics Research 18, 6, 534--555.
10. Coros, S., Beaudoin, P., Yin, K., and van de Panne, M. 2008. Synthesis of constrained walking skills. ACM Transactions on Graphics 27, 5, 113:1--113:9.
11. da Silva, M., Abe, Y., and Popović, J. 2008. Simulation of human motion data using short-horizon model-predictive control. Computer Graphics Forum 27, 2, 371--380.
12. da Silva, M., Abe, Y., and Popović, J. 2008. Interactive simulation of stylized human locomotion. ACM Transactions on Graphics 27, 3, 82:1--82:10.
13. Erez, T., and Smart, W. 2007. Bipedal walking on rough terrain using manifold control. International Conference on Intelligent Robots and Systems (IROS), 1539--1544.
14. Faloutsos, P., van de Panne, M., and Terzopoulos, D. 2001. Composable controllers for physics-based character animation. In Proceedings of ACM SIGGRAPH 2001, Annual Conference Series, 251--260.
15. Fattal, R., and Lischinski, D. 2004. Target-driven smoke animation. ACM Transactions on Graphics 23, 3, 441--448.
16. Fleming, W. H. 1978. Exit probabilities and optimal stochastic control. Applied Mathematics and Optimization 4, 329--346.
17. Hodgins, J. K., and Pollard, N. S. 1997. Adapting simulated behaviors for new characters. In Proceedings of SIGGRAPH 97, Annual Conference Series, 153--162.
18. Hodgins, J. K., Wooten, W. L., Brogan, D. C., and O'Brien, J. F. 1995. Animating human athletics. In Proceedings of ACM SIGGRAPH 95, Annual Conference Series, 71--78.
19. Holland, C. 1977. A new energy characterization of the smallest eigenvalue fo the schrödinger equation. Communications on Pure and Applied Mathematics 30, 755--765.
20. Jacobson, D., and Mayne, D. 1970. Differential Dynamic Programming, 1st ed. Elsevier, New York.
21. Kappen, H. J. 2005. Linear theory for control of nonlinear stochastic systems. Physical Review Letters 95, 20, 200--204.
22. Laszlo, J. F., van de Panne, M., and Fiume, E. L. 1996. Limit cycle control and its application to the animation of balancing and walking. In Proceedings of SIGGRAPH 96, Annual Conference Series, 155--162.
23. Liu, C. K., Hertzmann, A., and Popović, Z. 2005. Learning physics-based motion style with nonlinear inverse optimization. ACM Transactions on Graphics 24, 3, 1071--1081.
24. McCann, J., and Pollard, N. 2007. Responsive characters from motion fragments. ACM Transactions on Graphics 26, 3, 6:1--6:7.
25. McNamara, A., Treuille, A., Popović, Z., and Stam, J. 2004. Fluid control using the adjoint method. ACM Transactions on Graphics 23, 3, 449--456.
26. Milam, M. B. 2003. Real-Time Optimal Trajectory Generation for Constrained Dynamical Systems. PhD thesis, Caltech.
27. Ngo, J. T., and Marks, J. 1993. Spacetime constraints revisited. In Proceedings of ACM SIGGRAPH 2000, Annual Conference Series, 343--350.
28. Oksendal, B. K. 2002. Stochastic Differential Equations: An Introduction with Applications. Springer, New York, NY.
29. Pollard, N. S., and Behmaram-Mosavat, F. 2000. Force-based motion editing for locomotion tasks. In In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 663--669.
30. Popović, Z., and Witkin, A. P. 1999. Physically based motion transformation. In Computer Graphics (Proceedings of SIGGRAPH 99), ACM SIGGRAPH, Annual Conference Series, 11--20.
31. Safonova, A., Hodgins, J., and Pollard, N. 2004. Synthesizing physically realistic human motion in low-dimensional, behavior-specific spaces. ACM Transactions on Graphics 23, 3, 514--521.
32. Sok, K. W., Kim, M., and Lee, J. 2007. Simulating biped behaviors from human motion data. ACM Transactions on Graphics 26, 3, 107:1--107:9.
33. Sulejmanpasić, A., and Popović, J. 2005. Adaptation of performed ballistic motion. ACM Transactions on Graphics 24, 1, 165--179.
34. Sutton, R. S., and Barto, A. G. 1998. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA.
35. Tassa, Y., Erez, T., and Smart, W. 2008. Receding horizon differential dynamic programming. In Advances in Neural Information Processing Systems (NIPS), vol. 20, 1465--1472.
36. Todorov, E. 2006. Bayesian Brain: Probabilistic Approaches to Neural Coding. MIT Press, Cambridge, MA, ch. 12, 269--298.
37. Todorov, E. 2006. Linearly-solvable markov decision problems. Advances in Neural Information Processing Systems (NIPS) 19, 1369--1376.
38. Todorov, E. 2008. Efficient computation of optimal actions. http://www.cogsci.ucsd.edu/~todorov/papers/framework.pdf. Unpublished manuscript, March.
39. Todorov, E. 2009. Compositionality of optimal control laws. http://www.cogsci.ucsd.edu/~todorov/papers/primitives.pdf. Unpublished manuscript, January 15.
40. Treuille, A., Lee, Y., and Popović, Z. 2007. Near-optimal character animation with continuous control. ACM Transactions on Graphics 26, 3, 7:1--7:7.
41. van de Panne, M., Kim, R., and Fiume, E. 1994. Synthesizing parameterized motions. In Eurographics Workshop on Simulation and Animation.
42. Witkin, A., and Kass, M. 1988. Spacetime constraints. In Computer Graphics (Proceedings of SIGGRAPH 88), vol. 22, 159--168.
43. Wooten, W. L., and Hodgins, J. K. 2000. Simulating leaping, tumbling, landing and balancing humans. International Conference on Robotics and Automation (ICRA), 656--662.
44. Yin, K., Loken, K., and van de Panne, M. 2007. SIMBICON: Simple biped locomotion control. ACM Transactions on Graphics 26, 3, 105:1--105:10.
45. Yin, K., Coros, S., Beaudoin, P., and van de Panne, M. 2008. Continuation methods for adapting simulated skills. ACM Transactions on Graphics 27, 3, 81:1--81:7.
46. Zordan, V. B., and Hodgins, J. K. 2002. Motion capture-driven simulations that hit and react. In Symposium on Computer Animation (SCA), 89--96.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

        • 3D imaging

    • Control methods

      • Computational control theory

  • Computer graphics

    • Animation