Abstract

This paper presents a novel approach to automated playtesting for the prediction of human player behavior and experience. We have previously demonstrated that Deep Reinforcement Learning (DRL) game-playing agents can predict both game difficulty and player engagement, operationalized as average pass and churn rates. We improve this approach by enhancing DRL with Monte Carlo Tree Search (MCTS). We also motivate an enhanced selection strategy for predictor features, based on the observation that an AI agent's best-case performance can yield stronger correlations with human data than the agent's average performance. Both additions consistently improve the prediction accuracy, and the DRL-enhanced MCTS outperforms both DRL and vanilla MCTS in the hardest levels. We conclude that player modelling via automated playtesting can benefit from combining DRL and MCTS. Moreover, it can be worthwhile to investigate a subset of repeated best AI agent runs, if AI gameplay does not yield good predictions on average.
- Aghyad Mohammad Albaghajati and Moataz Aly Kamaleldin Ahmed. 2020. Video Game Automated Testing Approaches: An Assessment Framework. IEEE Transactions on Games (2020). https://doi.org/10.1109/TG.2020.3032796Google Scholar
- Sinan Ariyurek, Aysu Betin-Can, and Elif Surer. 2019. Automated Video Game Testing Using Synthetic and Human-Like Agents. IEEE Transactions on Games (2019). https://doi.org/10.1109/TG.2019.2947597Google Scholar
Cross Ref
- Sinan Ariyurek, Aysu Betin-Can, and Elif Surer. 2020. Enhancing the monte carlo tree search algorithm for video game testing. In 2020 IEEE Conference on Games (CoG). IEEE, Osaka, Japan, 25--32. https://doi.org/10.1109/CoG47356.2020.9231670Google Scholar
Cross Ref
- Richard Bellman. 1957. A Markovian Decision Process. Journal of Mathematics and Mechanics , Vol. 6, 5 (1957), 679--684.Google Scholar
- Joakim Bergdahl, Camilo Gordillo, Konrad Tollmar, and Linus Gisslén. 2020. Augmenting automated game testing with deep reinforcement learning. In 2020 IEEE Conference on Games (CoG). IEEE, Osaka, Japan, 600--603. https://doi.org/10.1109/CoG47356.2020.9231552Google Scholar
Cross Ref
- Valerio Bonometti, Charles Ringer, Mark Hall, Alex R Wade, and Anders Drachen. 2019. Modelling Early User-Game Interactions for Joint Estimation of Survival Time and Churn Probability. In 2019 IEEE Conference on Games (CoG). IEEE, London, United Kingdom, 1--8. https://doi.org/10.1109/CIG.2019.8848038Google Scholar
- Julia Ayumi Bopp, Klaus Opwis, and Elisa D. Mekler. 2018. ?An Odd Kind of Pleasure": Differentiating Emotional Challenge in Digital Games .Association for Computing Machinery, New York, NY, USA, 1--12. https://doi.org/10.1145/3173574.3173615Google Scholar
- E. A. Boyle, T. M. Connolly, T. Hainey, and J. M. Boyle. 2012. Engagement in Digital Entertainment Games: A Systematic Review . Computers in Human Behavior , Vol. 28, 3 (2012), 771--780. https://doi.org/10.1016/j.chb.2011.11.020Google Scholar
Digital Library
- Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. 2012. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games , Vol. 4, 1 (2012), 1--43. https://doi.org/10.1109/TCIAIG.2012.2186810Google Scholar
- Kenneth Chang, Batu Aytemiz, and Adam M Smith. 2019. Reveal-more: Amplifying human effort in quality assurance testing using automated exploration. In 2019 IEEE Conference on Games (CoG). IEEE, London, United Kingdom, 1--8. https://doi.org/10.1109/CIG.2019.8848091Google Scholar
Digital Library
- Noshaba Cheema, Laura A. Frey-Law, Kourosh Naderi, Jaakko Lehtinen, Philipp Slusallek, and Perttu H"am"al"ainen. 2020. Predicting Mid-Air Interaction Movements and Fatigue Using Deep Reinforcement Learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/10.1145/3313831.3376701Google Scholar
Digital Library
- Jenova Chen. 2007. Flow in Games (and Everything Else). Commun. ACM , Vol. 50, 4 (April 2007), 31--34. https://doi.org/10.1145/1232743.1232769Google Scholar
Digital Library
- Tom Cole, Paul Cairns, and Marco Gillies. 2015. Emotional and Functional Challenge in Core and Avant-Garde Games. In Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play (London, United Kingdom) (CHI PLAY '15). Association for Computing Machinery, New York, NY, USA, 121--126. https://doi.org/10.1145/2793107.2793147Google Scholar
Digital Library
- Anna Cox, Paul Cairns, Pari Shah, and Michael Carroll. 2012. Not Doing but Thinking: The Role of Challenge in the Gaming Experience. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI '12). Association for Computing Machinery, New York, NY, USA, 79--88. https://doi.org/10.1145/2207676.2207689Google Scholar
Digital Library
- Simon Demediuk, Marco Tamassia, Xiaodong Li, and William L. Raffe. 2019. Challenging AI: Evaluating the Effect of MCTS-Driven Dynamic Difficulty Adjustment on Player Enjoyment. In Proceedings of the Australasian Computer Science Week Multiconference (Sydney, NSW, Australia) (ACSW 2019). Association for Computing Machinery, New York, NY, USA, Article 43, bibinfonumpages7 pages. https://doi.org/10.1145/3290688.3290748Google Scholar
Digital Library
- Alena Denisova, Paul Cairns, Christian Guckelsberger, and David Zendle. 2020. Measuring perceived challenge in digital games: Development & validation of the challenge originating from recent gameplay interaction scale (CORGIS). International Journal of Human-Computer Studies , Vol. 137 (2020), 102383. https://doi.org/10.1016/j.ijhcs.2019.102383Google Scholar
Digital Library
- Alena Denisova, Christian Guckelsberger, and David Zendle. 2017. Challenge in Digital Games: Towards Developing a Measurement Tool .Association for Computing Machinery, New York, NY, USA, 2511--2519. https://doi.org/10.1145/3027063.3053209Google Scholar
- Rovio Entertainment. 2018. Angry Birds Dream Blast . Game [Android, iOS].Google Scholar
- Samuel J Gershman, Eric J Horvitz, and Joshua B Tenenbaum. 2015. Computational rationality: A converging paradigm for intelligence in brains, minds, and machines. Science , Vol. 349, 6245 (2015), 273--278. https://doi.org/10.1126/science.aac6076Google Scholar
- Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. 2016. Deep learning. Vol. 1. MIT press Cambridge.Google Scholar
Digital Library
- Christian Guckelsberger, Christoph Salge, Jeremy Gow, and Paul Cairns. 2017. Predicting Player Experience without the Player.: An Exploratory Study. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (Amsterdam, The Netherlands) (CHI PLAY '17). Association for Computing Machinery, New York, NY, USA, 305--315. https://doi.org/10.1145/3116595.3116631Google Scholar
Digital Library
- Stefan Freyr Gudmundsson, Philipp Eisen, Erik Poromaa, Alex Nodet, Sami Purmonen, Bartlomiej Kozakowski, Richard Meurling, and Lele Cao. 2018. Human-like playtesting with deep learning. In 2018 IEEE Conference on Computational Intelligence and Games (CIG). IEEE, Maastricht, 1--8. https://doi.org/10.1109/CIG.2018.8490442Google Scholar
Digital Library
- Cristina Guerrero-Romero, Annie Louis, and Diego Perez-Liebana. 2017. Beyond playing to win: Diversifying heuristics for GVGAI. In 2017 IEEE Conference on Computational Intelligence and Games (CIG). IEEE, New York, NY, 118--125. https://doi.org/10.1109/CIG.2017.8080424Google Scholar
Digital Library
- Daniel Hernandez, Charles Takashi Toyin Gbadamosi, James Goodman, and James Alfred Walker. 2020. Metagame Autobalancing for Competitive Multiplayer Games. In 2020 IEEE Conference on Games (CoG) . IEEE, Osaka, Japan, 275--282. https://doi.org/10.1109/CoG47356.2020.9231762Google Scholar
- Christoffer Holmgård, Michael Cerny Green, Antonios Liapis, and Julian Togelius. 2018. Automated playtesting with procedural personas through MCTS with evolved heuristics. IEEE Transactions on Games , Vol. 11, 4 (2018), 352--362. https://doi.org/10.1109/TG.2018.2808198Google Scholar
- Aaron Isaksen, Drew Wallace, Adam Finkelstein, and Andy Nealen. 2017. Simulating strategy and dexterity for puzzle games. In 2017 IEEE Conference on Computational Intelligence and Games (CIG). IEEE, New York, NY, 142--149. https://doi.org/10.1109/CIG.2017.8080427Google Scholar
Digital Library
- Antti Kangasr"a"asiö , Kumaripaba Athukorala, Andrew Howes, Jukka Corander, Samuel Kaski, and Antti Oulasvirta. 2017. Inferring Cognitive Models from Data Using Approximate Bayesian Computation .Association for Computing Machinery, New York, NY, USA, 1295--1306. https://doi.org/10.1145/3025453.3025576Google Scholar
- Oleksandra Keehl and Adam M Smith. 2018. Monster carlo: an mcts-based framework for machine playtesting unity games. In 2018 IEEE Conference on Computational Intelligence and Games (CIG). IEEE, Maastricht, 1--8. https://doi.org/10.1109/CIG.2018.8490363Google Scholar
Digital Library
- Levente Kocsis and Csaba Szepesvári. 2006. Bandit Based Monte-Carlo Planning. In Proceedings of the 17th European Conference on Machine Learning (Berlin, Germany) (ECML'06). Springer-Verlag, Berlin, Heidelberg, 282--293. https://doi.org/10.1007/11871842_29Google Scholar
Digital Library
- Jeppe Theiss Kristensen, Arturo Valdivia, and Paolo Burelli. 2020. Estimating Player Completion Rate in Mobile Puzzle Games Using Reinforcement Learning. In 2020 IEEE Conference on Games (CoG). IEEE, Osaka, Japan, 636--639. https://doi.org/10.1109/CoG47356.2020.9231581Google Scholar
- Richard L Lewis, Andrew Howes, and Satinder Singh. 2014. Computational rationality: Linking mechanism and behavior through bounded utility maximization. Topics in cognitive science , Vol. 6, 2 (2014), 279--311. https://doi.org/10.1111/tops.12086Google Scholar
- Carlos Ling, Konrad Tollmar, and Linus Gisslén. 2020. Using Deep Convolutional Neural Networks to Detect Rendered Glitches in Video Games. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 16. 66--73. https://ojs.aaai.org/index.php/AIIDE/article/view/7409Google Scholar
- Derek Lomas, Kishan Patel, Jodi L. Forlizzi, and Kenneth R. Koedinger. 2013. Optimizing Challenge in an Educational Game Using Large-Scale Design Experiments .Association for Computing Machinery, New York, NY, USA, 89--98. https://doi.org/10.1145/2470654.2470668Google Scholar
- Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous Methods for Deep Reinforcement Learning. In Proceedings of The 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 48), , Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, USA, 1928--1937. http://proceedings.mlr.press/v48/mniha16.htmlGoogle Scholar
- Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et almbox. 2015. Human-level control through deep reinforcement learning. nature , Vol. 518, 7540 (2015), 529--533. https://doi.org/10.1038/nature14236Google Scholar
- Jeanne Nakamura and Mihaly Csikszentmihalyi. 2014. The concept of flow. In Flow and the foundations of positive psychology. Springer, 239--263. https://doi.org/10.1007/978--94-017--9088--8_16Google Scholar
- Thorbjørn S Nielsen, Gabriella AB Barros, Julian Togelius, and Mark J Nelson. 2015. General video game evaluation using relative algorithm performance profiles. In European Conference on the Applications of Evolutionary Computation. Springer, 369--380. https://doi.org/10.1007/978--3--319--16549--3_30Google Scholar
Cross Ref
- Heather L O'Brien and Elaine G Toms. 2008. What Is User Engagement? a Conceptual Framework for Defining User Engagement With Technology . Journal of the American Society for Information Science and Technology , Vol. 59, 6 (2008), 938--955. https://doi.org/10.1002/asi.20801Google Scholar
Digital Library
- Antti Oulasvirta. 2019. It's Time to Rediscover HCI Models. Interactions , Vol. 26, 4 (June 2019), 52--56. https://doi.org/10.1145/3330340Google Scholar
Digital Library
- Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. 2017. Curiosity-Driven Exploration by Self-Supervised Prediction. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 (Sydney, NSW, Australia) (ICML'17). JMLR.org, 2778--2787.Google Scholar
Cross Ref
- Diego Perez-Liebana, Spyridon Samothrakis, Julian Togelius, Tom Schaul, and Simon Lucas. 2016. General Video Game AI: Competition, Challenges and Opportunities. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 30. AAAI press, 4335--4337.Google Scholar
- Johannes Pfau, Antonios Liapis, Georg Volkmar, Georgios N Yannakakis, and Rainer Malaka. 2020. Dungeons & replicants: automated game balancing via deep player behavior modeling. In 2020 IEEE Conference on Games (CoG). IEEE, 431--438. https://doi.org/10.1109/CoG47356.2020.9231958Google Scholar
Cross Ref
- Erik Ragnar Poromaa. 2017. Crushing candy crush: predicting human success rate in a mobile game using Monte-Carlo tree search.Google Scholar
- Martin L Puterman. 2014. Markov Decision Processes: Discrete Stochastic Dynamic Programming .Wiley. https://doi.org/10.1002/9780470316887Google Scholar
Digital Library
- David Reguera, Pol Colomer-de Simón, Iván Encinas, Manel Sort, Jan Wedekind, and Marián Boguná. 2019. The Physics of Fun: Quantifying Human Engagement into Playful Activities. arXiv preprint arXiv:1911.01864 (2019).Google Scholar
- Shaghayegh Roohi, Asko Relas, Jari Takatalo, Henri Heiskanen, and Perttu H"am"al"ainen. 2020. Predicting Game Difficulty and Churn Without Players. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (Virtual Event, Canada) (CHI PLAY '20). Association for Computing Machinery, New York, NY, USA, 585--593. https://doi.org/10.1145/3410404.3414235Google Scholar
Digital Library
- Karsten Rothmeier, Nicolas Pflanzl, Joschka Hüllmann, and Mike Preuss. 2020. Prediction of Player Churn and Disengagement Based on User Activity Data of a Freemium Online Strategy Game. IEEE Transactions on Games (2020). https://doi.org/10.1109/TG.2020.2992282Google Scholar
Cross Ref
- Richard M Ryan and Edward L Deci. 2000. Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions. Contemporary educational psychology , Vol. 25, 1 (2000), 54--67. https://doi.org/10.1006/ceps.1999.1020Google Scholar
- Richard M Ryan, C Scott Rigby, and Andrew Przybylski. 2006. The Motivational Pull of Video Games: A Self-determination Theory Approach. Motivation and emotion , Vol. 30, 4 (2006), 344--360. https://doi.org/10.1007/s11031-006--9051--8Google Scholar
- Jesse Schell. 2008. The Art of Game Design: A Book of Lenses .Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.Google Scholar
Digital Library
- John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).Google Scholar
- Ruimin Shen, Yan Zheng, Jianye Hao, Zhaopeng Meng, Yingfeng Chen, Changjie Fan, and Yang Liu. 2020. Generating Behavior-Diverse Game AIs with Evolutionary Multi-Objective Deep Reinforcement Learning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20 , , Christian Bessiere (Ed.). International Joint Conferences on Artificial Intelligence Organization, 3371--3377. https://doi.org/10.24963/ijcai.2020/466 Main track.Google Scholar
Cross Ref
- Yuchul Shin, Jaewon Kim, Kyohoon Jin, and Young Bin Kim. 2020. Playtesting in Match 3 Game Using Strategic Plays via Reinforcement Learning. IEEE Access , Vol. 8 (2020), 51593--51600. https://doi.org/10.1109/ACCESS.2020.2980380Google Scholar
Cross Ref
- David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et almbox. 2016. Mastering the game of Go with deep neural networks and tree search. nature , Vol. 529, 7587 (2016), 484--489. https://doi.org/10.1038/nature16961Google Scholar
- Nathan Sorenson, Philippe Pasquier, and Steve DiPaola. 2011. A generic approach to challenge modeling for the procedural creation of video game levels. IEEE Transactions on Computational Intelligence and AI in Games , Vol. 3, 3 (2011), 229--244. https://doi.org/10.1109/TCIAIG.2011.2161310Google Scholar
Cross Ref
- Samantha Stahlke, Atiya Nova, and Pejman Mirza-Babaei. 2020. Artificial Players in the Design Process: Developing an Automated Testing Tool for Game Level and World Design. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play. 267--280. https://doi.org/10.1145/3410404.3414249Google Scholar
Digital Library
- Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction .MIT press.Google Scholar
Digital Library
- Penelope Sweetser and Peta Wyeth. 2005. GameFlow: A Model for Evaluating Player Enjoyment in Games. Computers in Entertainment (CIE) , Vol. 3, 3 (July 2005), 3. https://doi.org/10.1145/1077246.1077253Google Scholar
Digital Library
- Julian Togelius and Jurgen Schmidhuber. 2008. An experiment in automatic game design. In 2008 IEEE Symposium On Computational Intelligence and Games. IEEE, Perth, WA, Australia, 111--118. https://doi.org/10.1109/CIG.2008.5035629Google Scholar
Cross Ref
- April Tyack and Elisa D. Mekler. 2020. Self-Determination Theory in HCI Games Research: Current Uses and Open Questions. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1--22. https://doi.org/10.1145/3313831.3376723Google Scholar
Digital Library
- Marc Van Kreveld, Maarten Löffler, and Paul Mutser. 2015. Automated puzzle difficulty estimation. In 2015 IEEE Conference on Computational Intelligence and Games (CIG). IEEE, Tainan, Taiwan, 415--422. https://doi.org/10.1109/CIG.2015.7317913Google Scholar
Cross Ref
- Georgios N. Yannakakis and John Hallam. 2007. Towards Optimizing Entertainment in Computer Games. Applied Artificial Intelligence , Vol. 21, 10 (Nov. 2007), 933--971. https://doi.org/10.1080/08839510701527580Google Scholar
Digital Library
- Georgios N Yannakakis and Julian Togelius. 2011. Experience-driven procedural content generation. IEEE Transactions on Affective Computing , Vol. 2, 3 (2011), 147--161. https://doi.org/10.1109/T-AFFC.2011.6Google Scholar
Digital Library
- Georgios N Yannakakis and Julian Togelius. 2018. Artificial intelligence and games . Vol. 2. Springer. https://doi.org/10.1007/978--3--319--63519--4Google Scholar
Digital Library
Index Terms
Predicting Game Difficulty and Engagement Using AI Players
Recommendations
Predicting Game Difficulty and Churn Without Players
CHI PLAY '20: Proceedings of the Annual Symposium on Computer-Human Interaction in PlayWe propose a novel simulation model that is able to predict the per-level churn and pass rates of Angry Birds Dream Blast, a popular mobile free-to-play game. Our primary contribution is to combine AI gameplay using Deep Reinforcement Learning (DRL) ...
Evolutionary algorithm for game difficulty control
CIMMACS'05: Proceedings of the 4th WSEAS international conference on Computational intelligence, man-machine systems and cyberneticsIn this paper we propose an evolutionary algorithm against player (EAP) for game that controls the difficulty of a game based on the player's propensity and proficiency fundamental using the Genetic Algorithm (GA). This paper describes how we use the GA ...
Predicting player disengagement in online games
CGW14: Proceedings of the Third International Conference on Computer GamesGame engagement, as one of the most fundamental objectives for game designers to achieve, has become an attractive industrial and academic topic. An important direction in this area is to construct a model to predict how long a player could be engaged ...






Comments