skip to main content
research-article

Real-Time Style Modelling of Human Locomotion via Feature-Wise Transformations and Local Motion Phases

Authors Info & Claims
Published:04 May 2022Publication History
Skip Abstract Section

Abstract

Controlling the manner in which a character moves in a real-time animation system is a challenging task with useful applications. Existing style transfer systems require access to a reference content motion clip, however, in real-time systems the future motion content is unknown and liable to change with user input. In this work we present a style modelling system that uses an animation synthesis network to model motion content based on local motion phases. An additional style modulation network uses feature-wise transformations to modulate style in real-time. To evaluate our method, we create and release a new style modelling dataset, 100STYLE, containing over 4 million frames of stylised locomotion data in 100 different styles that present a number of challenges for existing systems. To model these styles, we extend the local phase calculation with a contact-free formulation. In comparison to other methods for real-time style modelling, we show our system is more robust and efficient in its style representation while improving motion quality.

Skip Supplemental Material Section

Supplemental Material

References

  1. Kfir Aberman, Peizhuo Li, Dani Lischinski, Olga Sorkine-Hornung, Daniel Cohen-Or, and Baoquan Chen. 2020a. Skeleton-aware networks for deep motion retargeting. ACM Transactions on Graphics (TOG) 39, 4 (2020), 62--1.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Kfir Aberman, Yijia Weng, Dani Lischinski, Daniel Cohen-Or, and Baoquan Chen. 2020b. Unpaired motion style transfer from video to animation. ACM Transactions on Graphics (TOG) 39, 4 (2020), 64.Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Amir Atapour-Abarghouei and Toby P Breckon. 2018. Real-time monocular depth estimation using synthetic data with domain adaptation via image style transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2800--2810.Google ScholarGoogle ScholarCross RefCross Ref
  4. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016).Google ScholarGoogle Scholar
  5. Alex Bird and Christopher K I Williams. 2019. Customizing sequence generation with multi-task dynamical systems. arXiv preprint arXiv:1910.05026 (2019).Google ScholarGoogle Scholar
  6. Judith Bütepage, Michael Black, Danica Kragic, and Hedvig Kjellström. 2017. Deep representation learning for human motion prediction and classification. arXiv preprint arXiv:1702.07486 (2017).Google ScholarGoogle Scholar
  7. Simon Clavet. 2016. Motion matching and the road to next-gen animation. In GDC.Google ScholarGoogle Scholar
  8. Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2015. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 (2015).Google ScholarGoogle Scholar
  9. Yuzhu Dong, Andreas Aristidou, Ariel Shamir, Moshe Mahler, and Eakta Jain. 2020. Adult2child: Motion style transfer using CycleGANs. In Motion, Interaction and Games (MIG '20). Association for Computing Machinery. https://doi.org/10.1145/3424636.3426909Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Han Du, Erik Herrmann, Janis Sprenger, Klaus Fischer, and Philipp Slusallek. 2019. Stylistic locomotion modeling and synthesis using variational generative models. In Motion, Interaction and Games (MIG '19). Association for Computing Machinery. https://doi.org/10.1145/3359566.3360083Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio. 2018. Feature-wise transformations. Distill (2018). https://doi.org/10.23915/distill.00011Google ScholarGoogle Scholar
  12. Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. 2017. A learned representation for artistic style. In ICLR.Google ScholarGoogle Scholar
  13. Cian Eastwood, Ian Mason, Christopher K.I. Williams, and Bernhard Schölkopf. 2021. Source-free adaptation to measurement shift via bottom-up feature restoration. arXiv preprint arXiv:2107.05446 (2021).Google ScholarGoogle Scholar
  14. Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. 2015. Recurrent network models for human dynamics. In Proceedings of the IEEE International Conference on Computer Vision. 4346--4354.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Levi Fussell, Kevin Bergamin, and Daniel Holden. 2021. SuperTrack: Motion tracking for physically simulated characters using supervised learning. ACM Transactions on Graphics 40 (12 2021), 1--13. https://doi.org/10.1145/3478513.3480527Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2015. A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576 (2015).Google ScholarGoogle Scholar
  17. Ralph Gross and Jianbo Shi. 2001. The CMU motion of body (MoBo) database. Technical Report.Google ScholarGoogle Scholar
  18. David Ha and Jürgen Schmidhuber. 2018. World models. arXiv preprint arXiv:1803.10122 (2018).Google ScholarGoogle Scholar
  19. Gustav Eje Henter, Simon Alexanderson, and Jonas Beskow. 2020. MoGlow: Probabilistic and controllable motion synthesis using normalising flows. ACM Transactions on Graphics 39, 4 (2020), 236:1-236:14. https://doi.org/10.1145/3414685.3417836Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Daniel Holden, Ikhsanul Habibie, Ikuo Kusajima, and Taku Komura. 2017a. Fast neural style transfer for motion data. IEEE computer graphics and applications 37, 4 (2017), 42--49.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Daniel Holden, Oussama Kanoun, Maksym Perepichka, and Tiberiu Popa. 2020. Learned motion matching. ACM Transactions on Graphics (TOG) 39, 4 (2020), 53--1.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Daniel Holden, Taku Komura, and Jun Saito. 2017b. Phase-functioned neural networks for character control. ACM Transactions on Graphics (TOG) 36, 4 (2017), 42.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Daniel Holden, Jun Saito, and Taku Komura. 2016. A deep learning framework for character motion synthesis and editing. ACM Trans. Graph. 35, 4 (2016), 138:1--138:11.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Yunzhong Hou and Liang Zheng. 2020. Source free domain adaptation with image translation. arXiv preprint arXiv:2008.07514 (2020).Google ScholarGoogle Scholar
  25. Eugene Hsu, Kari Pulli, and Jovan Popović. 2005. Style translation for human motion. In ACM Transactions on Graphics (TOG), Vol. 24. ACM, 1082--1089.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Xun Huang and Serge Belongie. 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In Computer Vision (ICCV), 2017 IEEE International Conference on.Google ScholarGoogle ScholarCross RefCross Ref
  27. Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. 2018. Multimodal unsupervised image-to-image translation. arXiv preprint arXiv:1804.04732 (2018).Google ScholarGoogle Scholar
  28. Leslie Ikemoto, Okan Arikan, and David Forsyth. 2009. Generalizing motion edits with Gaussian processes. ACM Trans. Graph. 28, 1 (Feb. 2009). https://doi.org/10.1145/1477926.1477927Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. 2014. Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence 36, 7 (Jul 2014), 1325--1339.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Xu Ji, Razvan Pascanu, Devon Hjelm, Balaji Lakshminarayanan, and Andrea Vedaldi. 2021. Test sample accuracy scales with training sample density in neural networks. arXiv preprint arXiv:2106.08365 (2021).Google ScholarGoogle Scholar
  31. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google ScholarGoogle Scholar
  32. Lucas Kovar, Michael Gleicher, and Frédéric Pighin. 2002. Motion Graphs. ACM Trans. Graph. 21, 3 (July 2002). https://doi.org/10.1145/566654.566605Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Kyungho Lee, Seyoung Lee, and Jehee Lee. 2018. Interactive character animation by learning multi-objective control. ACM Trans. Graph. 37, 6 (Dec. 2018). https://doi.org/10.1145/3272127.3275071Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Zimo Li, Yi Zhou, Shuangjiu Xiao, Chong He, and Hao Li. 2018. Auto-conditioned LSTM network for extended complex human motion synthesis. In ICLR.Google ScholarGoogle Scholar
  35. Hung Yu Ling, Fabio Zinno, George Cheng, and Michiel van de Panne. 2020. Character controllers using motion VAEs. 39, 4 (2020).Google ScholarGoogle Scholar
  36. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9, Nov (2008), 2579--2605.Google ScholarGoogle Scholar
  37. Julieta Martinez, Michael J. Black, and Javier Romero. 2017. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarGoogle ScholarCross RefCross Ref
  38. Ian Mason, Sebastian Starke, He Zhang, Hakan Bilen, and Taku Komura. 2018. Few-shot learning of homogeneous human locomotion styles. Computer Graphics Forum 37, 7 (2018), 143--153.Google ScholarGoogle ScholarCross RefCross Ref
  39. Tomohiko Mukai and Shigeru Kuriyama. 2005. Geostatistical motion interpolation. In ACM Transactions on Graphics (TOG), Vol. 24. ACM, 1062--1070.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Soomin Park, Deok-Kyeong Jang, and Sung-Hee Lee. 2021. Diverse motion stylization for multiple style domains via spatial-temporal graph-based generative model. Proceedings of the ACM on Computer Graphics and Interactive Techniques 4, 3 (2021), 1--17.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. 2018. DeepMimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph. 37, 4 (July 2018). https://doi.org/10.1145/3197517.3201311Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel van de Panne. 2017. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Transactions on Graphics (TOG) 36, 4 (2017), 41.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. 2018. Film: Visual reasoning with a general conditioning layer. In Thirty-Second AAAI Conference on Artificial Intelligence.Google ScholarGoogle ScholarCross RefCross Ref
  44. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems. 506--516.Google ScholarGoogle Scholar
  45. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2018. Efficient parametrization of multi-domain deep neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarGoogle ScholarCross RefCross Ref
  46. Charles Rose, Michael F Cohen, and Bobby Bodenheimer. 1998. Verbs and adverbs: Multidimensional motion interpolation. IEEE Computer Graphics and Applications 18, 5 (1998), 32--40.Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Alla Safonova, Jessica K. Hodgins, and Nancy S. Pollard. 2004. Synthesizing physically realistic human motion in low-dimensional, behavior-specific spaces. ACM Trans. Graph. 23, 3 (Aug. 2004), 514--521. https://doi.org/10.1145/1015706.1015754Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Harrison Jesse Smith, Chen Cao, Michael Neff, and Yingying Wang. 2019. Efficient neural networks for real-time motion style transfer. Proc. ACM Comput. Graph. Interact. Tech. 2, 2 (July 2019). https://doi.org/10.1145/3340254Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15, 1 (2014), 1929--1958.Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Sebastian Starke, He Zhang, Taku Komura, and Jun Saito. 2019. Neural state machine for character-scene interactions. ACM Trans. Graph. 38, 6 (Nov. 2019). https://doi.org/10.1145/3355089.3356505Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Sebastian Starke, Yiwei Zhao, Taku Komura, and Kazi Zaman. 2020. Local motion phases for learning multi-contact character movements. ACM Transactions on Graphics (TOG) 39, 4 (2020), 54--1.Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Amos J Storkey. 2009. When training and test sets are different: characterising learning transfer. In Dataset Shift in Machine Learning. MIT Press, 3--28.Google ScholarGoogle Scholar
  53. Graham W Taylor and Geoffrey E Hinton. 2009. Factored conditional restricted Boltzmann machines for modeling motion style. In Proceedings of the 26th annual international conference on machine learning. ACM, 1025--1032.Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. 2021. Self-supervised learning with data augmentations provably isolates content from style. In Advances in Neural Information Processing Systems.Google ScholarGoogle Scholar
  55. Jue Wang, Steven M. Drucker, Maneesh Agrawala, and Michael F. Cohen. 2006. The cartoon animation filter. ACM Trans. Graph. 25, 3 (July 2006). https://doi.org/10.1145/1141911.1142010Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. Yu-Hui Wen, Zhipeng Yang, Hongbo Fu, Lin Gao, Yanan Sun, and Yong-Jin Liu. 2021. Autoregressive stylized motion synthesis with generative flow. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 13612--13621.Google ScholarGoogle ScholarCross RefCross Ref
  57. Shihong Xia, Congyi Wang, Jinxiang Chai, and Jessica Hodgins. 2015. Realtime style transfer for unlabeled heterogeneous human motion. ACM Trans. Graph. 34, 4 (2015), 119:1--119:10.Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. KangKang Yin, Kevin Loken, and Michiel van de Panne. 2007. SIMBICON: Simple Biped Locomotion Control. ACM Trans. Graph. 26, 3 (July 2007). https://doi.org/10.1145/1276377.1276509Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Wenhao Yu, Greg Turk, and C. Karen Liu. 2018. Learning symmetric and low-energy locomotion. ACM Trans. Graph. 37, 4 (July 2018). https://doi.org/10.1145/3197517.3201397Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. M Ersin Yumer and Niloy J Mitra. 2016. Spectral style transfer for human motion between independent actions. ACM Transactions on Graphics (TOG) 35, 4 (2016), 137.Google ScholarGoogle Scholar
  61. He Zhang, Sebastian Starke, Taku Komura, and Jun Saito. 2018. Mode-adaptive neural networks for quadruped motion control. ACM Transactions on Graphics (TOG) 37, 4 (2018), 145.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Real-Time Style Modelling of Human Locomotion via Feature-Wise Transformations and Local Motion Phases

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image Proceedings of the ACM on Computer Graphics and Interactive Techniques
          Proceedings of the ACM on Computer Graphics and Interactive Techniques  Volume 5, Issue 1
          May 2022
          252 pages
          EISSN:2577-6193
          DOI:10.1145/3535313
          Issue’s Table of Contents

          Copyright © 2022 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 4 May 2022
          Published in pacmcgit Volume 5, Issue 1

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Research
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!