skip to main content
research-article

Predicting Game Difficulty and Engagement Using AI Players

Published:06 October 2021Publication History
Skip Abstract Section

Abstract

This paper presents a novel approach to automated playtesting for the prediction of human player behavior and experience. We have previously demonstrated that Deep Reinforcement Learning (DRL) game-playing agents can predict both game difficulty and player engagement, operationalized as average pass and churn rates. We improve this approach by enhancing DRL with Monte Carlo Tree Search (MCTS). We also motivate an enhanced selection strategy for predictor features, based on the observation that an AI agent's best-case performance can yield stronger correlations with human data than the agent's average performance. Both additions consistently improve the prediction accuracy, and the DRL-enhanced MCTS outperforms both DRL and vanilla MCTS in the hardest levels. We conclude that player modelling via automated playtesting can benefit from combining DRL and MCTS. Moreover, it can be worthwhile to investigate a subset of repeated best AI agent runs, if AI gameplay does not yield good predictions on average.

References

  1. Aghyad Mohammad Albaghajati and Moataz Aly Kamaleldin Ahmed. 2020. Video Game Automated Testing Approaches: An Assessment Framework. IEEE Transactions on Games (2020). https://doi.org/10.1109/TG.2020.3032796Google ScholarGoogle Scholar
  2. Sinan Ariyurek, Aysu Betin-Can, and Elif Surer. 2019. Automated Video Game Testing Using Synthetic and Human-Like Agents. IEEE Transactions on Games (2019). https://doi.org/10.1109/TG.2019.2947597Google ScholarGoogle ScholarCross RefCross Ref
  3. Sinan Ariyurek, Aysu Betin-Can, and Elif Surer. 2020. Enhancing the monte carlo tree search algorithm for video game testing. In 2020 IEEE Conference on Games (CoG). IEEE, Osaka, Japan, 25--32. https://doi.org/10.1109/CoG47356.2020.9231670Google ScholarGoogle ScholarCross RefCross Ref
  4. Richard Bellman. 1957. A Markovian Decision Process. Journal of Mathematics and Mechanics , Vol. 6, 5 (1957), 679--684.Google ScholarGoogle Scholar
  5. Joakim Bergdahl, Camilo Gordillo, Konrad Tollmar, and Linus Gisslén. 2020. Augmenting automated game testing with deep reinforcement learning. In 2020 IEEE Conference on Games (CoG). IEEE, Osaka, Japan, 600--603. https://doi.org/10.1109/CoG47356.2020.9231552Google ScholarGoogle ScholarCross RefCross Ref
  6. Valerio Bonometti, Charles Ringer, Mark Hall, Alex R Wade, and Anders Drachen. 2019. Modelling Early User-Game Interactions for Joint Estimation of Survival Time and Churn Probability. In 2019 IEEE Conference on Games (CoG). IEEE, London, United Kingdom, 1--8. https://doi.org/10.1109/CIG.2019.8848038Google ScholarGoogle Scholar
  7. Julia Ayumi Bopp, Klaus Opwis, and Elisa D. Mekler. 2018. ?An Odd Kind of Pleasure": Differentiating Emotional Challenge in Digital Games .Association for Computing Machinery, New York, NY, USA, 1--12. https://doi.org/10.1145/3173574.3173615Google ScholarGoogle Scholar
  8. E. A. Boyle, T. M. Connolly, T. Hainey, and J. M. Boyle. 2012. Engagement in Digital Entertainment Games: A Systematic Review . Computers in Human Behavior , Vol. 28, 3 (2012), 771--780. https://doi.org/10.1016/j.chb.2011.11.020Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. 2012. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games , Vol. 4, 1 (2012), 1--43. https://doi.org/10.1109/TCIAIG.2012.2186810Google ScholarGoogle Scholar
  10. Kenneth Chang, Batu Aytemiz, and Adam M Smith. 2019. Reveal-more: Amplifying human effort in quality assurance testing using automated exploration. In 2019 IEEE Conference on Games (CoG). IEEE, London, United Kingdom, 1--8. https://doi.org/10.1109/CIG.2019.8848091Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Noshaba Cheema, Laura A. Frey-Law, Kourosh Naderi, Jaakko Lehtinen, Philipp Slusallek, and Perttu H"am"al"ainen. 2020. Predicting Mid-Air Interaction Movements and Fatigue Using Deep Reinforcement Learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/10.1145/3313831.3376701Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Jenova Chen. 2007. Flow in Games (and Everything Else). Commun. ACM , Vol. 50, 4 (April 2007), 31--34. https://doi.org/10.1145/1232743.1232769Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Tom Cole, Paul Cairns, and Marco Gillies. 2015. Emotional and Functional Challenge in Core and Avant-Garde Games. In Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play (London, United Kingdom) (CHI PLAY '15). Association for Computing Machinery, New York, NY, USA, 121--126. https://doi.org/10.1145/2793107.2793147Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Anna Cox, Paul Cairns, Pari Shah, and Michael Carroll. 2012. Not Doing but Thinking: The Role of Challenge in the Gaming Experience. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI '12). Association for Computing Machinery, New York, NY, USA, 79--88. https://doi.org/10.1145/2207676.2207689Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Simon Demediuk, Marco Tamassia, Xiaodong Li, and William L. Raffe. 2019. Challenging AI: Evaluating the Effect of MCTS-Driven Dynamic Difficulty Adjustment on Player Enjoyment. In Proceedings of the Australasian Computer Science Week Multiconference (Sydney, NSW, Australia) (ACSW 2019). Association for Computing Machinery, New York, NY, USA, Article 43, bibinfonumpages7 pages. https://doi.org/10.1145/3290688.3290748Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Alena Denisova, Paul Cairns, Christian Guckelsberger, and David Zendle. 2020. Measuring perceived challenge in digital games: Development & validation of the challenge originating from recent gameplay interaction scale (CORGIS). International Journal of Human-Computer Studies , Vol. 137 (2020), 102383. https://doi.org/10.1016/j.ijhcs.2019.102383Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Alena Denisova, Christian Guckelsberger, and David Zendle. 2017. Challenge in Digital Games: Towards Developing a Measurement Tool .Association for Computing Machinery, New York, NY, USA, 2511--2519. https://doi.org/10.1145/3027063.3053209Google ScholarGoogle Scholar
  18. Rovio Entertainment. 2018. Angry Birds Dream Blast . Game [Android, iOS].Google ScholarGoogle Scholar
  19. Samuel J Gershman, Eric J Horvitz, and Joshua B Tenenbaum. 2015. Computational rationality: A converging paradigm for intelligence in brains, minds, and machines. Science , Vol. 349, 6245 (2015), 273--278. https://doi.org/10.1126/science.aac6076Google ScholarGoogle Scholar
  20. Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. 2016. Deep learning. Vol. 1. MIT press Cambridge.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Christian Guckelsberger, Christoph Salge, Jeremy Gow, and Paul Cairns. 2017. Predicting Player Experience without the Player.: An Exploratory Study. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (Amsterdam, The Netherlands) (CHI PLAY '17). Association for Computing Machinery, New York, NY, USA, 305--315. https://doi.org/10.1145/3116595.3116631Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Stefan Freyr Gudmundsson, Philipp Eisen, Erik Poromaa, Alex Nodet, Sami Purmonen, Bartlomiej Kozakowski, Richard Meurling, and Lele Cao. 2018. Human-like playtesting with deep learning. In 2018 IEEE Conference on Computational Intelligence and Games (CIG). IEEE, Maastricht, 1--8. https://doi.org/10.1109/CIG.2018.8490442Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Cristina Guerrero-Romero, Annie Louis, and Diego Perez-Liebana. 2017. Beyond playing to win: Diversifying heuristics for GVGAI. In 2017 IEEE Conference on Computational Intelligence and Games (CIG). IEEE, New York, NY, 118--125. https://doi.org/10.1109/CIG.2017.8080424Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Daniel Hernandez, Charles Takashi Toyin Gbadamosi, James Goodman, and James Alfred Walker. 2020. Metagame Autobalancing for Competitive Multiplayer Games. In 2020 IEEE Conference on Games (CoG) . IEEE, Osaka, Japan, 275--282. https://doi.org/10.1109/CoG47356.2020.9231762Google ScholarGoogle Scholar
  25. Christoffer Holmgård, Michael Cerny Green, Antonios Liapis, and Julian Togelius. 2018. Automated playtesting with procedural personas through MCTS with evolved heuristics. IEEE Transactions on Games , Vol. 11, 4 (2018), 352--362. https://doi.org/10.1109/TG.2018.2808198Google ScholarGoogle Scholar
  26. Aaron Isaksen, Drew Wallace, Adam Finkelstein, and Andy Nealen. 2017. Simulating strategy and dexterity for puzzle games. In 2017 IEEE Conference on Computational Intelligence and Games (CIG). IEEE, New York, NY, 142--149. https://doi.org/10.1109/CIG.2017.8080427Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Antti Kangasr"a"asiö , Kumaripaba Athukorala, Andrew Howes, Jukka Corander, Samuel Kaski, and Antti Oulasvirta. 2017. Inferring Cognitive Models from Data Using Approximate Bayesian Computation .Association for Computing Machinery, New York, NY, USA, 1295--1306. https://doi.org/10.1145/3025453.3025576Google ScholarGoogle Scholar
  28. Oleksandra Keehl and Adam M Smith. 2018. Monster carlo: an mcts-based framework for machine playtesting unity games. In 2018 IEEE Conference on Computational Intelligence and Games (CIG). IEEE, Maastricht, 1--8. https://doi.org/10.1109/CIG.2018.8490363Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Levente Kocsis and Csaba Szepesvári. 2006. Bandit Based Monte-Carlo Planning. In Proceedings of the 17th European Conference on Machine Learning (Berlin, Germany) (ECML'06). Springer-Verlag, Berlin, Heidelberg, 282--293. https://doi.org/10.1007/11871842_29Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Jeppe Theiss Kristensen, Arturo Valdivia, and Paolo Burelli. 2020. Estimating Player Completion Rate in Mobile Puzzle Games Using Reinforcement Learning. In 2020 IEEE Conference on Games (CoG). IEEE, Osaka, Japan, 636--639. https://doi.org/10.1109/CoG47356.2020.9231581Google ScholarGoogle Scholar
  31. Richard L Lewis, Andrew Howes, and Satinder Singh. 2014. Computational rationality: Linking mechanism and behavior through bounded utility maximization. Topics in cognitive science , Vol. 6, 2 (2014), 279--311. https://doi.org/10.1111/tops.12086Google ScholarGoogle Scholar
  32. Carlos Ling, Konrad Tollmar, and Linus Gisslén. 2020. Using Deep Convolutional Neural Networks to Detect Rendered Glitches in Video Games. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 16. 66--73. https://ojs.aaai.org/index.php/AIIDE/article/view/7409Google ScholarGoogle Scholar
  33. Derek Lomas, Kishan Patel, Jodi L. Forlizzi, and Kenneth R. Koedinger. 2013. Optimizing Challenge in an Educational Game Using Large-Scale Design Experiments .Association for Computing Machinery, New York, NY, USA, 89--98. https://doi.org/10.1145/2470654.2470668Google ScholarGoogle Scholar
  34. Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous Methods for Deep Reinforcement Learning. In Proceedings of The 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 48), , Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, USA, 1928--1937. http://proceedings.mlr.press/v48/mniha16.htmlGoogle ScholarGoogle Scholar
  35. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et almbox. 2015. Human-level control through deep reinforcement learning. nature , Vol. 518, 7540 (2015), 529--533. https://doi.org/10.1038/nature14236Google ScholarGoogle Scholar
  36. Jeanne Nakamura and Mihaly Csikszentmihalyi. 2014. The concept of flow. In Flow and the foundations of positive psychology. Springer, 239--263. https://doi.org/10.1007/978--94-017--9088--8_16Google ScholarGoogle Scholar
  37. Thorbjørn S Nielsen, Gabriella AB Barros, Julian Togelius, and Mark J Nelson. 2015. General video game evaluation using relative algorithm performance profiles. In European Conference on the Applications of Evolutionary Computation. Springer, 369--380. https://doi.org/10.1007/978--3--319--16549--3_30Google ScholarGoogle ScholarCross RefCross Ref
  38. Heather L O'Brien and Elaine G Toms. 2008. What Is User Engagement? a Conceptual Framework for Defining User Engagement With Technology . Journal of the American Society for Information Science and Technology , Vol. 59, 6 (2008), 938--955. https://doi.org/10.1002/asi.20801Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Antti Oulasvirta. 2019. It's Time to Rediscover HCI Models. Interactions , Vol. 26, 4 (June 2019), 52--56. https://doi.org/10.1145/3330340Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. 2017. Curiosity-Driven Exploration by Self-Supervised Prediction. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 (Sydney, NSW, Australia) (ICML'17). JMLR.org, 2778--2787.Google ScholarGoogle ScholarCross RefCross Ref
  41. Diego Perez-Liebana, Spyridon Samothrakis, Julian Togelius, Tom Schaul, and Simon Lucas. 2016. General Video Game AI: Competition, Challenges and Opportunities. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 30. AAAI press, 4335--4337.Google ScholarGoogle Scholar
  42. Johannes Pfau, Antonios Liapis, Georg Volkmar, Georgios N Yannakakis, and Rainer Malaka. 2020. Dungeons & replicants: automated game balancing via deep player behavior modeling. In 2020 IEEE Conference on Games (CoG). IEEE, 431--438. https://doi.org/10.1109/CoG47356.2020.9231958Google ScholarGoogle ScholarCross RefCross Ref
  43. Erik Ragnar Poromaa. 2017. Crushing candy crush: predicting human success rate in a mobile game using Monte-Carlo tree search.Google ScholarGoogle Scholar
  44. Martin L Puterman. 2014. Markov Decision Processes: Discrete Stochastic Dynamic Programming .Wiley. https://doi.org/10.1002/9780470316887Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. David Reguera, Pol Colomer-de Simón, Iván Encinas, Manel Sort, Jan Wedekind, and Marián Boguná. 2019. The Physics of Fun: Quantifying Human Engagement into Playful Activities. arXiv preprint arXiv:1911.01864 (2019).Google ScholarGoogle Scholar
  46. Shaghayegh Roohi, Asko Relas, Jari Takatalo, Henri Heiskanen, and Perttu H"am"al"ainen. 2020. Predicting Game Difficulty and Churn Without Players. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (Virtual Event, Canada) (CHI PLAY '20). Association for Computing Machinery, New York, NY, USA, 585--593. https://doi.org/10.1145/3410404.3414235Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Karsten Rothmeier, Nicolas Pflanzl, Joschka Hüllmann, and Mike Preuss. 2020. Prediction of Player Churn and Disengagement Based on User Activity Data of a Freemium Online Strategy Game. IEEE Transactions on Games (2020). https://doi.org/10.1109/TG.2020.2992282Google ScholarGoogle ScholarCross RefCross Ref
  48. Richard M Ryan and Edward L Deci. 2000. Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions. Contemporary educational psychology , Vol. 25, 1 (2000), 54--67. https://doi.org/10.1006/ceps.1999.1020Google ScholarGoogle Scholar
  49. Richard M Ryan, C Scott Rigby, and Andrew Przybylski. 2006. The Motivational Pull of Video Games: A Self-determination Theory Approach. Motivation and emotion , Vol. 30, 4 (2006), 344--360. https://doi.org/10.1007/s11031-006--9051--8Google ScholarGoogle Scholar
  50. Jesse Schell. 2008. The Art of Game Design: A Book of Lenses .Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).Google ScholarGoogle Scholar
  52. Ruimin Shen, Yan Zheng, Jianye Hao, Zhaopeng Meng, Yingfeng Chen, Changjie Fan, and Yang Liu. 2020. Generating Behavior-Diverse Game AIs with Evolutionary Multi-Objective Deep Reinforcement Learning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20 , , Christian Bessiere (Ed.). International Joint Conferences on Artificial Intelligence Organization, 3371--3377. https://doi.org/10.24963/ijcai.2020/466 Main track.Google ScholarGoogle ScholarCross RefCross Ref
  53. Yuchul Shin, Jaewon Kim, Kyohoon Jin, and Young Bin Kim. 2020. Playtesting in Match 3 Game Using Strategic Plays via Reinforcement Learning. IEEE Access , Vol. 8 (2020), 51593--51600. https://doi.org/10.1109/ACCESS.2020.2980380Google ScholarGoogle ScholarCross RefCross Ref
  54. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et almbox. 2016. Mastering the game of Go with deep neural networks and tree search. nature , Vol. 529, 7587 (2016), 484--489. https://doi.org/10.1038/nature16961Google ScholarGoogle Scholar
  55. Nathan Sorenson, Philippe Pasquier, and Steve DiPaola. 2011. A generic approach to challenge modeling for the procedural creation of video game levels. IEEE Transactions on Computational Intelligence and AI in Games , Vol. 3, 3 (2011), 229--244. https://doi.org/10.1109/TCIAIG.2011.2161310Google ScholarGoogle ScholarCross RefCross Ref
  56. Samantha Stahlke, Atiya Nova, and Pejman Mirza-Babaei. 2020. Artificial Players in the Design Process: Developing an Automated Testing Tool for Game Level and World Design. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play. 267--280. https://doi.org/10.1145/3410404.3414249Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction .MIT press.Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Penelope Sweetser and Peta Wyeth. 2005. GameFlow: A Model for Evaluating Player Enjoyment in Games. Computers in Entertainment (CIE) , Vol. 3, 3 (July 2005), 3. https://doi.org/10.1145/1077246.1077253Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Julian Togelius and Jurgen Schmidhuber. 2008. An experiment in automatic game design. In 2008 IEEE Symposium On Computational Intelligence and Games. IEEE, Perth, WA, Australia, 111--118. https://doi.org/10.1109/CIG.2008.5035629Google ScholarGoogle ScholarCross RefCross Ref
  60. April Tyack and Elisa D. Mekler. 2020. Self-Determination Theory in HCI Games Research: Current Uses and Open Questions. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1--22. https://doi.org/10.1145/3313831.3376723Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Marc Van Kreveld, Maarten Löffler, and Paul Mutser. 2015. Automated puzzle difficulty estimation. In 2015 IEEE Conference on Computational Intelligence and Games (CIG). IEEE, Tainan, Taiwan, 415--422. https://doi.org/10.1109/CIG.2015.7317913Google ScholarGoogle ScholarCross RefCross Ref
  62. Georgios N. Yannakakis and John Hallam. 2007. Towards Optimizing Entertainment in Computer Games. Applied Artificial Intelligence , Vol. 21, 10 (Nov. 2007), 933--971. https://doi.org/10.1080/08839510701527580Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. Georgios N Yannakakis and Julian Togelius. 2011. Experience-driven procedural content generation. IEEE Transactions on Affective Computing , Vol. 2, 3 (2011), 147--161. https://doi.org/10.1109/T-AFFC.2011.6Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. Georgios N Yannakakis and Julian Togelius. 2018. Artificial intelligence and games . Vol. 2. Springer. https://doi.org/10.1007/978--3--319--63519--4Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Predicting Game Difficulty and Engagement Using AI Players

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!