Anindita Ghosh
Philipp Slusallek
ABSTRACT
We present a learning-based method for generating animated 3D pose sequences depicting multiple sequential or superimposed actions provided in long, compositional sentences. We propose a hierarchical two-stream sequential model to explore a finer joint-level mapping between natural language sentences and the corresponding 3D pose sequences of the motions. We learn two manifold representations of the motion –- one each for the upper body and the lower body movements. We evaluate our proposed model on the publicly available KIT Motion-Language Dataset containing 3D pose data with human-annotated sentences. Experimental results show that our model advances the state-of-the-art on text-based motion synthesis in objective evaluations by a margin of 50%.
Supplemental Material
Available for Download
- Chaitanya Ahuja and Louis-Philippe Morency. 2019. Language2Pose: Natural Language Grounded Pose Forecasting. In 2019 International Conference on 3D Vision (3DV). 719–728. https://doi.org/10.1109/3DV.2019.00084Google Scholar
Cross Ref
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805(2018).Google Scholar
- Eva Hanser, Paul Mc Kevitt, Tom Lunney, and Joan Condell. 2009. Scenemaker: Intelligent multimodal visualisation of natural language scripts. In Irish Conference on Artificial Intelligence and Cognitive Science. Springer, 144–153.Google Scholar
- Angela S Lin, Lemeng Wu, Rodolfo Corona, Kevin Tai, Qixing Huang, and Raymond J Mooney. 2018. Generating animated videos of human activities from natural language descriptions. Visually Grounded Interaction and Language Workshop, NeurIPS (2018), 2.Google Scholar
- Matthias Plappert, Christian Mandery, and Tamim Asfour. 2016. The KIT Motion-Language Dataset. Big Data 4, 4 (dec 2016), 236–252. https://doi.org/10.1089/big.2016.0028Google Scholar
Index Terms
Text-Based Motion Synthesis with a Hierarchical Two-Stream RNN
Recommendations
Hybrid motion graph for character motion synthesis
Objective: This paper proposes a novel framework of Hybrid Motion Graph (HMG) for creating character animations, which enhances the graph-based structural control by motion field representations for efficient motion synthesis of diverse and interactive ...
Rhythmic-motion synthesis based on motion-beat analysis
Real-time animation of human-like characters is an active research area in computer graphics. The conventional approaches have, however, hardly dealt with the rhythmic patterns of motions, which are essential in handling rhythmic motions such as dancing ...
Rhythmic-motion synthesis based on motion-beat analysis
SIGGRAPH '03: ACM SIGGRAPH 2003 PapersReal-time animation of human-like characters is an active research area in computer graphics. The conventional approaches have, however, hardly dealt with the rhythmic patterns of motions, which are essential in handling rhythmic motions such as dancing ...
Comments