skip to main content
research-article

Investigating the Performance of Various Deep Neural Networks-based Approaches Designed to Identify Game Events in Gameplay Footage

Published:04 May 2022Publication History
Skip Abstract Section

Abstract

Video games, in addition to representing an extremely relevant field of entertainment and market, have been widely used as a case study in artificial intelligence for representing a problem with a high degree of complexity. In such studies, the investigation of approaches that endow player agents with the ability to retrieve relevant information from game scenes stands out, since such information can be very useful to improve their learning ability. This work proposes and analyses new deep learning-based models to identify game events occurring in Super Mario Bros gameplay footage. The architecture of each model is composed of a feature extractor convolutional neural network (CNN) and a classifier neural network (NN). The extracting CNN aims to produce a feature-based representation for game scenes and submit it to the classifier, so that the latter can identify the game event present in each scene. The models differ from each other according to the following elements: the type of the CNN; the type of the NN classifier; and the type of the game scene representation at the CNN input, being either single frames, or chunks, which are n-sequential frames (in this paper 6 frames were used per chunk) grouped into a single input. The main contribution of this article is to demonstrate the greater performance reached by the models which combines the chunk representation for the game scenes with the resources of the classifier recurrent neural networks (RNN).

References

  1. N. Aloysius and M. Geetha. 2017. A review on deep convolutional neural networks. In 2017 International Conference on Communication and Signal Processing (ICCSP). 0588--0592. https://doi.org/10.1109/ICCSP.2017.8286426Google ScholarGoogle ScholarCross RefCross Ref
  2. Leonard A. Annetta. 2008. Video Games in Education: Why They Should Be Used and How They Are Being Used. Theory Into Practice (2008). https://doi.org/10.1080/00405840802153940Google ScholarGoogle Scholar
  3. Elizabeth Boyle, Thomas M. Connolly, and Thomas Hainey. 2011. The role of psychology in understanding the impact of computer games. Entertainment Computing 2, 2 (2011), 69--74. https://doi.org/10.1016/j.entcom.2010.12.002 Serious Games Development and Applications.Google ScholarGoogle ScholarCross RefCross Ref
  4. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. arXiv:1406.1078 [cs.CL]Google ScholarGoogle Scholar
  5. Jiyang Gao, Zhenheng Yang, and Ram Nevatia. 2017. RED: Reinforced Encoder-Decoder Networks for Action Anticipation. arXiv:1707.04818 http://arxiv.org/abs/1707.04818Google ScholarGoogle Scholar
  6. Roeland De Geest, Efstratios Gavves, Amir Ghodrati, Zhenyang Li, Cees Snoek, and Tinne Tuytelaars. 2016a. Online Action Detection. arXiv:1604.06506 [cs.CV]Google ScholarGoogle Scholar
  7. Roeland De Geest, Efstratios Gavves, Amir Ghodrati, Zhenyang Li, Cees Snoek, and Tinne Tuytelaars. 2016b. Online Action Detection. CoRR abs/1604.06506. arXiv:1604.06506 http://arxiv.org/abs/1604.06506Google ScholarGoogle Scholar
  8. Global Data. 2021. Video games market set to become a 300bn-plus industry by 2025. https://www.globaldata.com/video-games-market-set-to-become-a-300bn-plus-industry-by-2025.Google ScholarGoogle Scholar
  9. Matthew Guzdial, Boyang Li, and Mark O. Riedl. 2017. Game Engine Learning from Video. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (Melbourne, Australia) (IJCAI'17). AAAI Press, 3707--3713.Google ScholarGoogle Scholar
  10. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarGoogle ScholarCross RefCross Ref
  11. Sepp Hochreiter. 1998. The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 6 (04 1998), 107--116. https://doi.org/10.1142/S0218488598000094Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (1997), 1735--1780. https://doi.org/10.1162/neco.1997.9.8.1735 arXiv:https://doi.org/10.1162/neco.1997.9.8.1735Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. J J Hopfield. 1982. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences 79 (1982), 2554--2558. https://doi.org/10.1073/pnas.79.8.2554Google ScholarGoogle ScholarCross RefCross Ref
  14. V. Janarthanan. 2012. Serious Video Games: Games for Education and Health. In 2012 Ninth International Conference on Information Technology - New Generations. https://doi.org/10.1109/ITNG.2012.79Google ScholarGoogle Scholar
  15. Michael I. Jordan. 1997. Chapter 25 - Serial Order: A Parallel Distributed Processing Approach. 121 (1997), 471--495. https://doi.org/10.1016/S0166-4115(97)80111-2Google ScholarGoogle Scholar
  16. S. Karakovskiy and J. Togelius. 2012. The Mario AI Benchmark and Competitions. IEEE Transactions on Computational Intelligence and AI in Games 4, 1 (2012), 55--67. https://doi.org/10.1109/TCIAIG.2012.2188528Google ScholarGoogle ScholarCross RefCross Ref
  17. Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. 2014. Large-Scale Video Classification with Convolutional Neural Networks. In 2014 IEEE Conference on Computer Vision and Pattern Recognition. 1725--1732. https://doi.org/10.1109/CVPR.2014.223Google ScholarGoogle Scholar
  18. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.Google ScholarGoogle Scholar
  19. Michail D. Kozlov and Mark K. Johansen. 2010. Real Behavior in Virtual Environments: Psychology Experiments in a Simple Virtual-Reality Paradigm Using Video Games. Cyberpsychology, Behavior, and Social Networking (2010). https://doi.org/10.1089/cyber.2009.0310Google ScholarGoogle Scholar
  20. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 1097--1105.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Kangwook Lee, Hoon Kim, and Changho Suh. 2017. Crash To Not Crash: Playing Video Games To Predict Vehicle Collisions. In ICML 2017.Google ScholarGoogle Scholar
  22. Zijin Luo, Matthew Guzdial, Nicholas Liao, and Mark Riedl. 2018. Player Experience Extraction from Gameplay Video. CoRR abs/1809.06201 (2018). arXiv:1809.06201Google ScholarGoogle Scholar
  23. Zijin Luo, Matthew Guzdial, and Mark Riedl. 2019. Making CNNs for Video Parsing Accessible. CoRR abs/1906.11877 (2019). arXiv:1906.11877Google ScholarGoogle Scholar
  24. M. Ravanbakhsh, M. Nabi, E. Sangineto, L. Marcenaro, C. Regazzoni, and N. Sebe. 2017. Abnormal event detection in videos using generative adversarial nets. In 2017 IEEE International Conference on Image Processing (ICIP). 1577--1581. https://doi.org/10.1109/ICIP.2017.8296547Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Johanna Roettl and Ralf Terlutter. 2018. The same video game in 2D, 3D or virtual reality - How does technology impact game evaluation and brand placements? PLOS ONE 13, 7 (07 2018), 1--24. https://doi.org/10.1371/journal.pone.0200724Google ScholarGoogle Scholar
  26. Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. Inverted Residuals and Linear Bottlenecks: Mobile Networks for Classification, Detection and Segmentation. CoRR abs/1801.04381 (2018). arXiv:1801.04381Google ScholarGoogle Scholar
  27. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. 2018. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362, 6419 (2018), 1140--1144. https://doi.org/10.1126/science.aar6404Google ScholarGoogle Scholar
  28. Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv 1409.1556 (09 2014).Google ScholarGoogle Scholar
  29. Khurram Soomro, Amir Zamir, and Mubarak Shah. 2012. UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. CoRR (12 2012).Google ScholarGoogle Scholar
  30. Adam Summerville, Sam Snodgrass, Matthew Guzdial, Christoffer Holmgård, Amy K. Hoover, Aaron Isaksen, Andy Nealen, and Julian Togelius. 2017. Procedural Content Generation via Machine Learning (PCGML). CoRR abs/1702.00539 (2017). arXiv:1702.00539 http://arxiv.org/abs/1702.00539Google ScholarGoogle Scholar
  31. Jeremy Heng Meng Wong and Mark John Francis Gales. 2016. Sequence Student-Teacher Training of Deep Neural Networks. In INTERSPEECH. https://doi.org/10.21437/Interspeech.2016-911Google ScholarGoogle Scholar
  32. Mingze Xu, Mingfei Gao, Yi-Ting Chen, Larry S. Davis, and David J. Crandall. 2018. Temporal Recurrent Networks for Online Action Detection. arXiv:1811.07391 http://arxiv.org/abs/1811.07391Google ScholarGoogle Scholar
  33. Manzhu Yu, Myra Bambacus, Guido Cervone, Keith Clarke, Daniel Duffy, Qunying Huang, Jing Li, Wenwen Li, Zhenlong Li, Qian Liu, Bernd Resch, Jingchao Yang, and Chaowei Yang. 2020. Spatiotemporal event detection: a review. International Journal of Digital Earth 13, 12 (2020), 1339--1365. https://doi.org/10.1080/17538947.2020.1738569 arXiv:https://doi.org/10.1080/17538947.2020.1738569Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Investigating the Performance of Various Deep Neural Networks-based Approaches Designed to Identify Game Events in Gameplay Footage

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Article Metrics

        • Downloads (Last 12 months)35
        • Downloads (Last 6 weeks)1

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!