Video-based 3D motion capture through biped control

Abstract

Marker-less motion capture is a challenging problem, particularly when only monocular video is available. We estimate human motion from monocular video by recovering three-dimensional controllers capable of implicitly simulating the observed human behavior and replaying this behavior in other environments and under physical perturbations. Our approach employs a state-space biped controller with a balance feedback mechanism that encodes control as a sequence of simple control tasks. Transitions among these tasks are triggered on time and on proprioceptive events (e.g., contact). Inference takes the form of optimal control where we optimize a high-dimensional vector of control parameters and the structure of the controller based on an objective function that compares the resulting simulated motion with input observations. We illustrate our approach by automatically estimating controllers for a variety of motions directly from monocular video. We show that the estimation of controller structure through incremental optimization and refinement leads to controllers that are more stable and that better approximate the reference motion. We demonstrate our approach by capturing sequences of walking, jumping, and gymnastics.

References

1. Baraff, D. 1996. Linear-time dynamics using Lagrange multipliers. In ACM SIGGRAPH.
2. Bhat, K. S., Seitz, S. M., Popovic, J., and Khosla, P. 2002. Computing the physical parameters of rigid-body motion from video. In ECCV, 551--566.
3. Bo, L., and Sminchisescu, C. 2010. Twin Gaussian processes for structured prediction. IJCV 87, 1--2.
4. Brubaker, M. A., and Fleet, D. J. 2008. The Kneed Walker for human pose tracking. In IEEE CVPR.
5. Brubaker, M. A., Fleet, D. J., and Hertzmann, A. 2007. Physics-based person tracking using simplified lower body dynamics. In IEEE CVPR.
6. Brubaker, M. A., Sigal, L., and Fleet, D. J. 2009. Estimating contact dynamics. In ICCV.
7. Chu, D., Shapiro, A., Allen, B., and Faloutsos, P. 2007. A dynamic controller toolkit. In ACM SIGGRAPH Video Game Symposium (Sandbox), 21--26.
8. Cline, M. 2002. Rigid Body Simulation with Contact and Constraints. Master's thesis, The University of British Columbia.
9. Coros, S., Beaudoin, P., and van de Panne, M. 2009. Robust task-based control policies for physics-based characters. In ACM Transactions on Graphics, vol. 28.
10. Crisis. 2006. http://crisis.sourceforge.net/.
11. Dempster, W. T. 1955. Space requirements of the seated operator: Geometrical, kinematic, and mechanical aspects of the body with special reference to the limbs. Tech. rep., Wright-Patterson Air Force Base 55--159.
12. Elgammal, A., Harwood, D., and Davis, L. 2000. Non-parametric model for background subtraction. In ECCV.
13. Gall, J., Stoll, C., de Aguiar, E., Theobalt, C., Rosenhahn, B., and Seidel, H.-P. 2009. Motion capture using joint skeleton tracking and surface estimation. In IEEE CVPR, 1746--1753.
14. Hansen, N. 2006. The CMA evolution strategy: A comparing review. Towards a New Evolutionary Computation. Advances on Estimation of Distribution Algorithms., 75--102.
15. Hodgins, J. K., Wooten, W. L., Brogan, D. C., and O'Brien, J. F. 1995. Animating human athletics. In ACM SIGGRAPH, 71--78.
16. Kwon, T., and Hodgins, J. 2010. Control systems for human running using an inverted pendulum model and a reference motion capture sequence. In ACM SIGGRAPH/Eurographics Symposium on Computer Animation.
17. Lee, Y., Kim, S., and Lee, J. 2010. Data-driven biped control. ACM Transactions on Graphics 29, 4.
18. Liu, C. K., Hertzmann, A., and Popović, Z. 2005. Learning physics-based motion style with nonlinear inverse optimization. ACM Transactions on Graphics 24, 3.
19. Liu, L., Yin, K., van de Panne, M., Shao, T., and Xu, W. 2010. Sampling-based contact-rich motion control. ACM Transactions on Graphics 29, 4.
20. Metaxas, D., and Terzopoulos, D. 1993. Shape and non-rigid motion estimation through physics-based synthesis. IEEE Transactions on PAMI 15, 6, 580--591.
21. Muico, U., Lee, Y., Popovic, J., and Popovic, Z. 2009. Contact-aware nonlinear control of dynamic characters. ACM Transactions on Graphics 28, 3.
22. Ngo, T., and Marks, J. 1993. Spacetime constraints revisited. In ACM SIGGRAPH.
23. ODE. 2006. http://www.ode.org/.
24. Pang, J. S., and Facchinei, F. 2003. Finite-dimensional variational inequalities and complementarity problems (i). Springer.
25. Schaal, S., and Schweighofer, N. 2005. Computational motor control in humans and robots. Cur. Op. in Neurobio., 6.
26. Sigal, L., Balan, A., and Black, M. J. 2010. Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. International Journal of Computer Vision 87, 1--2, 4--27.
27. Silva, M. D., Abe, Y., and Popovic, J. 2008. Interactive simulation of stylized human locomotion. ACM Transactions on Graphics 27, 3.
28. Sminchisescu, C., Kanaujia, A., and Metaxas, D. 2007. Bm3e: Discriminative density propagation for visual tracking. IEEE Transactions on PAMI 29, 11.
29. Tsai, Y.-Y., Cheng, K. B., Lin, W.-C., Lee, J., and Lee, T.-Y. 2010. Real-time physics-based 3d biped character animation using an inverted pendulum model. IEEE Transactions on Visualization and Computer Graphics 16, 2, 325--337.
30. Urtasun, R., J. Fleet, D., and Fua, P. 2006. Gaussian process dynamical models for 3d people tracking. In IEEE CVPR.
31. Vondrak, M., Sigal, L., and Jenkins, O. C. 2008. Physical simulation for probabilistic motion tracking. In IEEE CVPR.
32. Wang, J., Fleet, D. J., and Hertzmann, A. 2009. Optimizing walking controllers. ACM Transactions on Graphics 28, 5.
33. Wang, J., Fleet, D. J., and Hertzmann, A. 2010. Optimizing walking controllers for uncertain inputs and environments. ACM Transactions on Graphics 29, 4.
34. Wei, X., and Chai, J. 2010. Videomocap: Modeling physically realistic human motion from monocular video sequences. ACM Transactions on Graphics 29, 4.
35. Wren, C. R., and Pentland, A. 1998. Dynamic models of human motion. In Automatic Face and Gesture Recognition.
36. Yin, K., Loken, K., and van de Panne, M. 2007. SIMBICON: Simple biped locomotion control. ACM Transactions on Graphics 26, 3.
37. Yin, K., Coros, S., Beaudoin, P., and van de Panne, M. 2008. Continuation methods for adapting simulated skills. ACM Transactions on Graphics 27, 3.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Tracking

      • Computer vision tasks

        • Scene understanding

  • Computer graphics

    • Animation