ABSTRACT

VR games offer new freedom for players to interact naturally using motion. This makes it harder to design games that react to player motions convincingly. We present a framework for VR sword fighting experiences against a virtual character that simplifies the necessary technical work to achieve a convincing simulation. The framework facilitates VR design by abstracting from difficult details on the lower "physical" level of interaction, using data-driven models to automate both the identification of user actions and the synthesis of character animations. Designers are able to specify the character's behaviour on a higher "semantic" level using parameterised building blocks, which allow for control over the experience while minimising manual development work. We conducted a technical evaluation, a questionnaire study and an interactive user study. Our results suggest that the framework produces more realistic and engaging interactions than simple hand-crafted interaction logic, while supporting a controllable and understandable behaviour design.
References
- Maryam Asadi-Aghbolaghi, Albert Clapés, Marco Bellantonio, Hugo Jair Escalante, Víctor Ponce-López, Xavier Baró, Isabelle Guyon, Shohreh Kasaei, and Sergio Escalera. 2017. Deep Learning for Action and Gesture Recognition in Image Sequences: A Survey. In Gesture Recognition. Springer, 539--578.Google Scholar
- Jeremy N. Bailenson, Jim Blascovich, Andrew C. Beall, and Jack M. Loomis. 2003. Interpersonal Distance in Immersive Virtual Environments. Personality and Social Psychology Bulletin 29, 7 (2003), 819--833.Google Scholar
Cross Ref
- Dimitrios Batras, Judith Guez, Jean-François Jégo, and Marie-Hélène Tramus. 2016. A Virtual Reality Agent-Based Platform for Improvisation Between Real and Virtual Actors Using Gestures. In Proceedings of the 2016 Virtual Reality International Conference (VRIC '16). ACM, 34:1--34:4.Google Scholar
Digital Library
- Michael Bratman. 1987. Intention, Plans, and Practical Reason. Harvard University Press, Cambridge, Mass.Google Scholar
- Jan Broersen, Mehdi Dastani, Joris Hulstijn, Zisheng Huang, and Leendert van der Torre. 2001. The BOID Architecture: Conflicts Between Beliefs, Obligations, Intentions and Desires. In Proceedings of the 4th International Conference on Autonomous Agents (AGENTS '01). ACM, 9--16.Google Scholar
- Armin Bruderlin and Lance Williams. 1995. Motion Signal Processing. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '95). ACM, 97--104.Google Scholar
Digital Library
- Michael Büttner. 2013. Reinforcement Learning Based Character Locomotion in Hitman: Absolution. In Game Developers Conference 2013 (GDC 2013).Google Scholar
- Edwin Catmull and Raphael Rom. 1974. A Class of Local Interpolating Splines. In Computer Aided Geometric Design. Academic Press, 317--326.Google Scholar
- Hong Cheng, Lu Yang, and Zicheng Liu. 2016. Survey on 3D Hand Gesture Recognition. IEEE Transactions on Circuits and Systems for Video Technology 26, 9 (2016), 1659--1673.Google Scholar
Digital Library
- Loïc Ciccone, Martin Guay, Maurizio Nitti, and Robert W. Sumner. 2017. Authoring Motion Cycles. In Proceedings of the ACM SIGGRAPH / Eurographics Symposium on Computer Animation (SCA '17). ACM, 8:1--8:9.Google Scholar
- Simon Clavet. 2016. Motion Matching and The Road to Next-Gen Animation. In Game Developers Conference 2016 (GDC 2016).Google Scholar
- Javier Dehesa and Ninja Theory Ltd. 2020. Dataset for "Touché: Data-Driven Interactive Sword Fighting in Virtual Reality". University of Bath Research Data Archive, Bath, UK. https://doi.org/10.15125/BATH-00754.Google Scholar
- Javier Dehesa, Andrew Vidler, Christof Lutteroth, and Julian Padget. 2019. Towards Data-Driven Sword Fighting Experiences in VR. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA '19). LBW2117:1--LBW2117:6.Google Scholar
- David DeVault, Ron Artstein, Grace Benn, Teresa Dey, Ed Fast, Alesia Gainer, Kallirroi Georgila, Jon Gratch, Arno Hartholt, Margaux Lhommet, Gale Lucas, Stacy Marsella, Fabrizio Morbini, Angela Nazarian, Stefan Scherer, Giota Stratou, Apar Suri, David Traum, Rachel Wood, Yuyu Xu, Albert Rizzo, and Louis-Philippe Morency. 2014. SimSensei Kiosk: A Virtual Human Interviewer for Healthcare Decision Support. In Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS '14). International Foundation for Autonomous Agents and Multiagent Systems, 1061--1068.Google Scholar
Digital Library
- Mahmoud Elmezain, Ayoub Al-Hamadi, Jörg Appenrodt, and Bernd Michaelis. 2008. A Hidden Markov Model-Based Continuous Gesture Recognition System for Hand Motion Trajectory. In Proceedings of the 19th International Conference on Pattern Recognition (ICPR 2008). IEEE, 1--4.Google Scholar
Cross Ref
- Kutluhan Erol, James A. Hendler, and Dana S. Nau. 1994. UMCP: A Sound and Complete Procedure for Hierarchical Task-Network Planning. In Proceedings of the 2nd International Conference on Artificial Intelligence Planning Systems (AIPS '14), Kristian J. Hammond (Ed.). AAAI, 249--254.Google Scholar
- Sergio Escalera, Xavier Baró, Jordi Gonzàlez, Miguel A. Bautista, Meysam Madadi, Miguel Reyes, Víctor Ponce-López, Hugo J. Escalante, Jamie Shotton, and Isabelle Guyon. 2014. ChaLearn Looking at People Challenge 2014: Dataset and Results. In Workshops at the 13th European Conference on Computer Vision (ECCV 2014) (Lecture Notes in Computer Science). Springer, 459--473.Google Scholar
- Sean Ryan Fanello, Ilaria Gori, Giorgio Metta, and Francesca Odone. 2017. Keep It Simple and Sparse: Real-Time Action Recognition. In Gesture Recognition. Springer, 303--328.Google Scholar
- Richard E. Fikes and Nils J. Nilsson. 1971. Strips: A New Approach to the Application of Theorem Proving to Problem Solving. Artificial Intelligence 2, 3 (1971), 189--208.Google Scholar
Digital Library
- Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. 2015. Recurrent Network Models for Human Dynamics. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV '15). 4346--4354.Google Scholar
Digital Library
- Michael P. Georgeff and François Felix Ingrand. 1989. Decision-Making in an Embedded Reasoning System. In Proceedings of the 11th International Joint Conference on Artificial Intelligence. Morgan Kaufmann, 972--978.Google Scholar
- Michael Gleicher. 2001. Motion Path Editing. In Proceedings of the 2001 Symposium on Interactive 3D Graphics (I3D '01). ACM, 195--202.Google Scholar
Digital Library
- Rachel Heck and Michael Gleicher. 2007. Parametric Motion Graphs. In Proceedings of the 2007 Symposium on Interactive 3D Graphics and Games (I3D '07). ACM, 129--136.Google Scholar
Digital Library
- Edmond S. L. Ho, Jacky C. P. Chan, Taku Komura, and Howard Leung. 2013. Interactive Partner Control in Close Interactions for Real-Time Applications. ACM Trans. Multimedia Comput. Commun. Appl. 9, 3 (July 2013), 21:1--21:19.Google Scholar
Digital Library
- Edmond S. L. Ho and Taku Komura. 2011. A Finite State Machine Based on Topology Coordinates for Wrestling Games. Computer Animation and Virtual Worlds 22, 5 (Sept. 2011), 435--443.Google Scholar
Digital Library
- Daniel Holden. 2018. Robust Solving of Optical Motion Capture Data by Denoising. ACM Transactions on Graphics 37, 4 (2018), 165:1--165:12.Google Scholar
Digital Library
- Daniel Holden, Taku Komura, and Jun Saito. 2017. Phase-Functioned Neural Networks for Character Control. ACM Transactions on Graphics 36, 4 (2017), 42:1--42:13.Google Scholar
Digital Library
- Lucio Ieronutti and Luca Chittaro. 2007. Employing Virtual Humans for Education and Training in X3D/VRML Worlds. Computers & Education 49, 1 (2007), 93--109.Google Scholar
Digital Library
- Charlene Jennett, Anna L. Cox, Paul Cairns, Samira Dhoparee, Andrew Epps, Tim Tijs, and Alison Walton. 2008. Measuring and Defining the Experience of Immersion in Games. International Journal of Human-Computer Studies 66, 9 (2008), 641--661.Google Scholar
Digital Library
- Sanna Kallio, Juha Kela, and Jani Mäntyjärvi. 2003. Online Gesture Recognition System for Mobile Interaction. In Proceedings of the 2003 IEEE International Conference on Systems, Man and Cybernetics (SMC '03), Vol. 3. IEEE, 2070--2076.Google Scholar
Cross Ref
- Michelle R. Kandalaft, Nyaz Didehbani, Daniel C. Krawczyk, Tandra T. Allen, and Sandra B. Chapman. 2013. Virtual Reality Social Cognition Training for Young Adults with High-Functioning Autism. Journal of Autism and Developmental Disorders 43, 1 (2013), 34--44.Google Scholar
Cross Ref
- Cem Keskin, Ali Taylan Cemgil, and Lale Akarun. 2011. DTW Based Clustering to Improve Hand Gesture Recognition. In Human Behavior Understanding. Springer, 72--81.Google Scholar
- Sangki Kim, Gunhyuk Park, Sunghoon Yim, Seungmoon Choi, and Seungjin Choi. 2009. Gesture-Recognizing Hand-Held Interface with Vibrotactile Feedback for 3D Interaction. IEEE Transactions on Consumer Electronics 55, 3 (2009), 1169--1177.Google Scholar
Digital Library
- Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015).Google Scholar
- Jan Kolkmeier, Jered Vroon, and Dirk Heylen. 2016. Interacting with Virtual Agents in Shared Space: Single and Joint Effects of Gaze and Proxemics. In Proceedings of the 16th International Conference on Intelligent Virtual Agents (IVA 2016). Springer, 1--14.Google Scholar
Cross Ref
- Iuliia Kotseruba and John K. Tsotsos. 2016. A Review of 40 Years of Cognitive Architecture Research: Core Cognitive Abilities and Practical Applications. arXiv:1610.08602 [cs] (2016).Google Scholar
- Lucas Kovar and Michael Gleicher. 2004. Automated Extraction and Parameterization of Motions in Large Data Sets. ACM Transactions on Graphics 23, 3 (2004), 559.Google Scholar
Digital Library
- Lucas Kovar, Michael Gleicher, and Frédéric Pighin. 2002. Motion Graphs. In Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '02). ACM, 473--482.Google Scholar
- John E. Laird, Allen Newell, and Paul S. Rosenbloom. 1987. SOAR: An Architecture for General Intelligence. Artificial Intelligence 33, 1 (1987), 1--64.Google Scholar
Digital Library
- Jehee Lee and Kang Hoon Lee. 2006. Precomputing Avatar Behavior from Human Motion Data. Graphical Models 68, 2 (March 2006), 158--174.Google Scholar
Digital Library
- Yongjoon Lee, Kevin Wampler, Gilbert Bernstein, Jovan Popovic, and Zoran Popovi´ c. 2010. Motion Fields for Interactive Character Locomotion. In ACM SIGGRAPH Asia 2010 Papers (SIGGRAPH ASIA '10). ACM, 138:1--138:8.Google Scholar
- Rung-Huei Liang and Ming Ouhyoung. 1998. A Real-Time Continuous Gesture Recognition System for Sign Language. In Proceedings of the 3rd IEEE International Conference on Automatic Face and Gesture Recognition (FG '98). IEEE, 558--567.Google Scholar
Cross Ref
- Joan Llobera, Bernhard Spanlang, Giulio Ruffini, and Mel Slater. 2010. Proxemics with Multiple Dynamic Characters in an Immersive Virtual Environment. ACM Trans. Appl. Percept. 8, 1 (2010), 3:1--3:12.Google Scholar
Digital Library
- Liang Lu, Lingpeng Kong, Chris Dyer, Noah A. Smith, and Steve Renals. 2016. Segmental Recurrent Neural Networks for End-to-End Speech Recognition. arXiv:1603.00223 [cs] (2016).Google Scholar
- Jani Mäntyjärvi, Juha Kela, Panu Korpipää, and Sanna Kallio. 2004. Enabling Fast and Effortless Customisation in Accelerometer Based Gesture Interaction. In Proceedings of the 3rd International Conference on Mobile and Ubiquitous Multimedia (MUM '04). ACM, 25--31.Google Scholar
Digital Library
- Aline Menin, Rafael Torchelsen, and Luciana Nedel. 2018. An Analysis of VR Technology Used in Immersive Simulations with a Serious Game Perspective. IEEE Computer Graphics and Applications 38, 2 (2018), 57--73.Google Scholar
Digital Library
- Sushmita Mitra and Tinku Acharya. 2007. Gesture Recognition: A Survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 37, 3 (2007), 311--324.Google Scholar
Digital Library
- Mark Mizuguchi, John Buchanan, and Tom Calvert. 2001. Data Driven Motion Transitions for Interactive Games. In Eurographics 2001 - Short Presentations. Eurographics Association.Google Scholar
- Pavlo Molchanov, Xiaodong Yang, Shalini Gupta, Kihwan Kim, Stephen Tyree, and Jan Kautz. 2016. Online Detection and Classification of Dynamic Hand Gestures with Recurrent 3D Convolutional Neural Networks. In Proceedings of the 2016 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '16). IEEE, 4207--4215.Google Scholar
Cross Ref
- Vinod Nair and Geoffrey E. Hinton. 2010. Rectified Linear Units Improve Restricted Boltzmann Machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning (ICML '10). Omnipress, 807--814.Google Scholar
- Natalia Neverova, Christian Wolf, Graham W. Taylor, and Florian Nebout. 2014. Multi-Scale Deep Learning for Gesture Detection and Localization. In Workshops at the 13th European Conference on Computer Vision (ECCV 2014). Springer, 474--490.Google Scholar
- Tsukasa Noma, Liwei Zhao, and Norman I. Badler. 2000. Design of a Virtual Human Presenter. IEEE Computer Graphics and Applications 20, 4 (July 2000), 79--85.Google Scholar
Digital Library
- Felipe Pepe. 2019. The CRPG Book: A Guide to Computer Role-Playing Games. Bitmap Books, Bath, UK. OCLC: 1114936629.Google Scholar
- Ken Perlin. 1995. Real Time Responsive Animation with Personality. IEEE Transactions on Visualization and Computer Graphics 1, 1 (March 1995), 5--15.Google Scholar
Digital Library
- Giuseppe Raffa, Jinwon Lee, Lama Nachman, and Junehwa Song. 2010. Don't Slow Me down: Bringing Energy Efficiency to Continuous Gesture Recognition. In Proceedings of the 2010 International Symposium on Wearable Computers (ISWC). IEEE, 1--8.Google Scholar
Cross Ref
- Anand S. Rao and Michael P. Georgeff. 1991. Modeling Rational Agents within a BDI-Architecture. In Proceedings of the 2nd International Conference on Principles of Knowledge Representation and Reasoning (KR'91). Morgan Kaufmann, 473--484.Google Scholar
Digital Library
- Miguel Reyes, Gabriel Domínguez, and Sergio Escalera. 2011. Feature-weighting in Dynamic Timewarping for Gesture Recognition in Depth Data. In Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops). IEEE, 1182--1188.Google Scholar
Cross Ref
- Jeff Rickel and W. Lewis Johnson. 1999. Animated Agents for Procedural Training in Virtual Reality: Perception, Cognition, and Motor Control. Applied Artificial Intelligence 13, 4--5 (1999), 343--382.Google Scholar
Cross Ref
- Richard M. Ryan. 1982. Control and Information in the Intrapersonal Sphere: An Extension of Cognitive Evaluation Theory. Journal of Personality and Social Psychology 43, 3 (1982), 450--461.Google Scholar
Cross Ref
- Mark Sagar. 2015. BabyX. In ACM SIGGRAPH 2015 Computer Animation Festival (SIGGRAPH '15). ACM, 184--184.Google Scholar
- Thomas Schlömer, Benjamin Poppinga, Niels Henze, and Susanne Boll. 2008. Gesture Recognition with a Wii Controller. In Proceedings of the 2nd International Conference on Tangible and Embedded Interaction (TEI '08). ACM, 11--14.Google Scholar
Digital Library
- Sebastian Starke, He Zhang, Taku Komura, and Jun Saito. 2019. Neural State Machine for Character-Scene Interactions. ACM Transactions on Graphics 38, 6 (2019), 209:1--209:14.Google Scholar
Digital Library
- Nick Taubert, Martin Löffler, Nicolas Ludolph, Andrea Christensen, Dominik Endres, and Martin A. Giese. 2013. A Virtual Reality Setup for Controllable, Stylized Real-Time Interactions Between Humans and Avatars with Sparse Gaussian Process Dynamical Models. In Proceedings of the ACM Symposium on Applied Perception (SAP '13). ACM, 41--44.Google Scholar
- Graham W. Taylor, Geoffrey E Hinton, and Sam T. Roweis. 2007. Modeling Human Motion Using Binary Latent Variables. In Advances in Neural Information Processing Systems 19. MIT Press, 1345--1352.Google Scholar
- Adrien Treuille, Yongjoon Lee, and Zoran Popovic. 2007. Near-Optimal Character Animation with Continuous Control. ACM Transactions on Graphics 36, 3 (2007), 7:1--7:7.Google Scholar
- Eleni Tsironi, Pablo Barros, and Stefan Wermter. 2016. Gesture Recognition with a Convolutional Long Short-Term Memory Recurrent Neural Network. In Proceedings of the 24th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2016). Ciaco i6doc.com, 213--218.Google Scholar
- David Vogt, Steve Grehl, Erik Berger, Heni Ben Amor, and Bernhard Jung. 2014. A Data-Driven Method for Real-Time Character Animation in Human-Agent Interaction. In Proceedings of the 14th International Conference on Intelligent Virtual Agents (IVA 2014). Springer, 463--476.Google Scholar
Cross Ref
- Sy Bor Wang, Ariadna Quattoni, Louis-Philippe Morency, David Demirdjian, and Trevor Darrell. 2006. Hidden Conditional Random Fields for Gesture Recognition. In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '06), Vol. 2. IEEE, 1521--1527.Google Scholar
- John Weissmann and Ralf Salomon. 1999. Gesture Recognition for Virtual Reality Applications Using Data Gloves and Neural Networks. In Proceedings of the International Joint Conference on Neural Networks (IJCNN '99), Vol. 3. IEEE, 2043--2046.Google Scholar
Cross Ref
- Andrew Witkin and Zoran Popovic. 1995. Motion Warping. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '95). ACM, 105--108.Google Scholar
- Bob G. Witmer and Michael J. Singer. 1998. Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence: Teleoperators and Virtual Environments 7, 3 (1998), 225--240.Google Scholar
Digital Library
- Sebastien C. Wong, Adam Gatt, Victor Stamatescu, and Mark D. McDonnell. 2016. Understanding Data Augmentation for Classification: When to Warp?. In Proceedings of the 2016 International Conference on Digital Image Computing: Techniques and Applications (DICTA '16). IEEE, 1--6.Google Scholar
- Deyou Xu. 2006. A Neural Network Approach for Hand Gesture Recognition in Virtual Reality Driving Training System of SPG. In Proceedings of the 18th International Conference on Pattern Recognition (ICPR '06), Vol. 3. IEEE, 519--522.Google Scholar
- Junji Yamato, Jun Ohya, and Kenichiro Ishii. 1992. Recognizing Human Action in Time-Sequential Images Using Hidden Markov Model. In Proceedings of the 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '92). IEEE, 379--385.Google Scholar
Cross Ref
- Ying Yin and Randall Davis. 2014. Real-Time Continuous Gesture Recognition for Natural Human-Computer Interaction. In Proceedings of the 2014 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC '14). 113--120.Google Scholar
Cross Ref
- Fisher Yu and Vladlen Koltun. 2015. Multi-Scale Context Aggregation by Dilated Convolutions. In Proceedings of the 4rd International Conference on Learning Representations (ICLR 2016).Google Scholar
- He Zhang, Sebastian Starke, Taku Komura, and Jun Saito. 2018. Mode-Adaptive Neural Networks for Quadruped Motion Control. ACM Transactions on Graphics 37, 4 (2018), 145:1--145:11.Google Scholar
Digital Library
Supplemental Material
Index Terms
Touché: Data-Driven Interactive Sword Fighting in Virtual Reality





Comments