skip to main content
research-article

For Honor, for Toxicity: Detecting Toxic Behavior through Gameplay

Published:06 October 2021Publication History
Skip Abstract Section

Abstract

Is it possible to detect toxicity in games just by observing in-game behavior? If so, what are the behavioral factors that will help machine learning to discover the unknown relationship between gameplay and toxic behavior? In this initial study, we examine whether it is possible to predict toxicity in the MOBA gameFor Honor by observing in-game behavior for players that have been labeled as toxic (i.e. players that have been sanctioned by Ubisoft community managers). We test our hypothesis of detecting toxicity through gameplay with a dataset of almost 1,800 sanctioned players, and comparing these sanctioned players with unsanctioned players. Sanctioned players are defined by their toxic action type (offensive behavior vs. unfair advantage) and degree of severity (warned vs. banned). Our findings, based on supervised learning with random forests, suggest that it is not only possible to behaviorally distinguish sanctioned from unsanctioned players based on selected features of gameplay; it is also possible to predict both the sanction severity (warned vs. banned) and the sanction type (offensive behavior vs. unfair advantage). In particular, all random forest models predict toxicity, its severity, and type, with an accuracy of at least 82%, on average, on unseen players. This research shows that observing in-game behavior can support the work of community managers in moderating and possibly containing the burden of toxic behavior.

References

  1. 2016. For Honor Code Of Conduct. https://forums.ubisoft.com/showthread.php/1490092-Code-of-Conduct. Accessed: 20-02--10.Google ScholarGoogle Scholar
  2. 2018. Dev Blog: Toxicity. ubisoft.com/en-us/game/rainbow-six/siege/news-updates/4mPLa195TWiXREafM4T4oC. Accessed: 20-02--10.Google ScholarGoogle Scholar
  3. Lucy A. Sparrow, Martin Gibbs, and Michael Arnold. 2021. The Ethics of Multiplayer Game Design and Community Management: Industry Perspectives and Challenges. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1--13.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Sonam Adinolf and Selen Turkay. 2018. Toxic behaviors in Esports games: player perceptions and coping strategies. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts. 365--372.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Anti Defamation League. 2020. Free to Play? Hate, Harassment and Positive Social Experience in Online Games 2020. https://www.adl.org/free-to-play-2020. Accessed: 20-02--10.Google ScholarGoogle Scholar
  6. Mary Elizabeth Ballard and Kelly MarieWelch. 2017. Virtual warfare: Cyberbullying and cyber-victimization in MMOG play. Games and culture 12, 5 (2017), 466--491.Google ScholarGoogle ScholarCross RefCross Ref
  7. Shaowen Bardzell. 2010. Feminist HCI: taking stock and outlining an agenda for design. In Proceedings of the SIGCHI conference on human factors in computing systems. 1301--1310.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Christopher Barlett and Sarah M Coyne. 2014. A meta-analysis of sex differences in cyber-bullying behavior: The moderating role of age. Aggressive behavior 40, 5 (2014), 474--488.Google ScholarGoogle Scholar
  9. Nicole A Beres, Julian Frommel, Elizabeth Reid, Regan L Mandryk, and Madison Klarkowski. 2021. Don't You Know That You're Toxic: Normalization of Toxicity in Online Gaming. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1--15.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Jeremy Blackburn and Haewoon Kwak. 2014. STFU NOOB! predicting crowdsourced decisions on toxic behavior in online games. In Proceedings of the 23rd international conference on World wide web. 877--888.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Chris Bruzzo. 2020. EA and the war against toxicity. https://www.gamesindustry.biz/articles/2020-06--23-ea-and-thewar- against-toxicity. Accessed: 20-02--10.Google ScholarGoogle Scholar
  12. Lindsey A Cary, Jordan Axt, and Alison L Chasteen. 2020. The interplay of individual differences, norms, and group identification in predicting prejudiced behavior in online video game interactions. Journal of Applied Social Psychology 50, 11 (2020), 623--637.Google ScholarGoogle ScholarCross RefCross Ref
  13. Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002. SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research 16 (2002), 321--357.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Valve Corporation. 20013. League Of Legends.Google ScholarGoogle Scholar
  15. Amanda C Cote. 2017. "I Can Defend Myself" Women's Strategies for Coping With Harassment While Gaming Online. Games and Culture 12, 2 (2017), 136--155.Google ScholarGoogle ScholarCross RefCross Ref
  16. Pascal Debroek. 2020. The Impact of Toxicity on Retention and LTV. https://www.gamasutra.com/blogs/PascalDebroek/ 20201125/374272/The_Impact_of_Toxicity_on_Retention_and_LTV.php. Accessed: 20-02--10.Google ScholarGoogle Scholar
  17. Blizzard Entertainment. 2015. Overwatch.Google ScholarGoogle Scholar
  18. Niklas Ericsson and Hampus Bergström. 2020. How toxicity differ between male and female players in competitive Overwatch.Google ScholarGoogle Scholar
  19. David R Ewoldsen, Cassie A Eno, BradleyMOkdie, John A Velez, Rosanna E Guadagno, and Jamie DeCoster. 2012. Effect of playing violent video games cooperatively or competitively on subsequent cooperative behavior. Cyberpsychology, Behavior, and Social Networking 15, 5 (2012), 277--280.Google ScholarGoogle ScholarCross RefCross Ref
  20. Fair Play Alliance. 2021. https://fairplayalliance.org/Google ScholarGoogle Scholar
  21. Chek Yang Foo and Elina MI Koivisto. 2004. Defining grief play in MMORPGs: player and developer perceptions. In Proceedings of the 2004 ACM SIGCHI International Conference on Advances in computer entertainment technology. 245--250.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Jesse Fox and Wai Yen Tang. 2017. Women's experiences with general and sexual harassment in online video games: Rumination, organizational responsiveness, withdrawal, and coping strategies. New Media & Society 19, 8 (2017), 1290--1307. https://doi.org/10.1177/1461444816635778 arXiv:https://doi.org/10.1177/1461444816635778Google ScholarGoogle Scholar
  23. Julian Frommel, Valentin Sagl, Ansgar E Depping, Colby Johanson, Matthew K Miller, and Regan L Mandryk. 2020. Recognizing Affiliation: Using Behavioural Traces to Predict the Quality of Social Interactions in Online Games. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--16.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Riot Games. 2009. League Of Legends.Google ScholarGoogle Scholar
  25. Spiros V Georgakopoulos, Sotiris K Tasoulis, Aristidis G Vrahatis, and Vassilis P Plagianakos. 2019. Convolutional neural networks for twitter text toxicity analysis. In INNS Big Data and Deep Learning conference. Springer, 370--379.Google ScholarGoogle Scholar
  26. Ayushi Ghosh. 2021. Analyzing Toxicity in Online Gaming Communities. Turkish Journal of Computer and Mathematics Education (TURCOMAT) 12, 10 (2021), 4448--4455.Google ScholarGoogle Scholar
  27. Kishonna L Gray. 2012. Deviant bodies, stigmatized identities, and racist acts: Examining the experiences of African- American gamers in Xbox Live. New Review of Hypermedia and Multimedia 18, 4 (2012), 261--276.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Kishonna L Gray. 2012. Intersecting oppressions and online communities: Examining the experiences of women of color in Xbox Live. Information, Communication & Society 15, 3 (2012), 411--428.Google ScholarGoogle ScholarCross RefCross Ref
  29. Kishonna L Gray. 2014. Race, gender, and deviance in Xbox live: Theoretical perspectives from the virtual margins. Routledge.Google ScholarGoogle Scholar
  30. Emily Jane Hayday, Holly Collison, and Geoffery Z. Kohe. 2020. Landscapes of tension, tribalism and toxicity: configuring a spatial politics of esport communities. Leisure Studies 0, 0 (2020), 1--15. https://doi.org/10.1080/02614367. 2020.1808049 arXiv:https://doi.org/10.1080/02614367.2020.1808049Google ScholarGoogle Scholar
  31. Haibo He, Yang Bai, Edwardo A Garcia, and Shutao Li. 2008. ADASYN: Adaptive synthetic sampling approach for imbalanced learning. In 2008 IEEE international joint conference on neural networks (IEEE world congress on computational intelligence). IEEE, 1322--1328.Google ScholarGoogle Scholar
  32. Zorah Hilvert-Bruce and James T Neill. 2020. I'm just trolling: The role of normative beliefs in aggressive behaviour in online gaming. Computers in Human Behavior 102 (2020), 303--311.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Jessica M Jerabeck and Christopher J Ferguson. 2013. The influence of solitary and cooperative violent video game play on aggressive and prosocial behavior. Computers in Human Behavior 29, 6 (2013), 2573--2578.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Daniel Johnson, Lennart E. Nacke, and Peta Wyeth. 2015. All about That Base: Differing Player Experiences in Video Game Genres and the Unique Case of MOBA Games. Association for Computing Machinery, New York, NY, USA, 2265--2274. https://doi.org/10.1145/2702123.2702447Google ScholarGoogle Scholar
  35. Daniel Johnson, Xiang Zhao, Katherine M White, and Varuni Wickramasinghe. 2021. Need satisfaction, passion, empathy and helping behaviour in videogame play. Computers in Human Behavior 122 (2021), 106817.Google ScholarGoogle ScholarCross RefCross Ref
  36. Jeff Kaplan. [n.d.]. Our progress so far. https://us.forums.blizzard.com/en/overwatch/t/our-progress-so-far/159046. Accessed: 20-02--10.Google ScholarGoogle Scholar
  37. Bastian Kordyaka, Katharina Jahn, and Bjoern Niehaves. 2020. Towards a unified theory of toxic behavior in video games. Internet Research 30, 4 (2020), 1081--1102. https://doi.org/10.1108/INTR-08--2019-0343Google ScholarGoogle ScholarCross RefCross Ref
  38. Lindah Kotut and D Scott McCrickard. 2020. Amplifying the Griot: Design Fiction for Development as an Inclusivity Lens. Technical Report. EasyChair.Google ScholarGoogle Scholar
  39. Yubo Kou. 2020. Toxic Behaviors in Team-Based Competitive Gaming: The Case of League of Legends. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play. 81--92.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Yubo Kou and Xinning Gui. 2021. Flag and Flaggability in Automated Moderation: The Case of Reporting Toxic Behavior in an Online Game Community. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1--12.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Yubo Kou and Bonnie Nardi. 2013. Regulating anti-social behavior on the Internet: The example of League of Legends. In iConference 2013 Proceedings. iSchools, 616--622. https://doi.org/doi:10.9776/13289Google ScholarGoogle Scholar
  42. Rachel Kowert. 2020. Dark Participation in Games. Frontiers in Psychology 11 (2020), 2969. https://doi.org/10.3389/ fpsyg.2020.598947Google ScholarGoogle ScholarCross RefCross Ref
  43. Jeffrey H. Kuznekoff and Lindsey M. Rose. 2013. Communication in multiplayer gaming: Examining player responses to gender cues. New Media & Society 15, 4 (2013), 541--556. https://doi.org/10.1177/1461444812458271 arXiv:https://doi.org/10.1177/1461444812458271Google ScholarGoogle Scholar
  44. Haewoon Kwak, Jeremy Blackburn, and Seungyeop Han. 2015. Exploring Cyberbullying and Other Toxic Behavior in Team Competition Online Games. Association for Computing Machinery, New York, NY, USA, 3739--3748. https: //doi.org/10.1145/2702123.2702529Google ScholarGoogle Scholar
  45. Maud Lemercier-Dugarin, Lucia Romo, Charles Tijus, and Oulmann Zerhouni. 2021. "Who Are the Cyka Blyat?" How Empathy, Impulsivity, and Motivations to Play Predict Aggressive Behaviors in Multiplayer Online Games. Cyberpsychology, Behavior, and Social Networking 24, 1 (2021), 63--69.Google ScholarGoogle ScholarCross RefCross Ref
  46. Roger J Lewis. 2000. An introduction to classification and regression tree (CART) analysis. In Annual meeting of the society for academic emergency medicine in San Francisco, California, Vol. 14.Google ScholarGoogle Scholar
  47. Rachael Luck. 2003. Dialogue in participatory design. Design studies 24, 6 (2003), 523--535.Google ScholarGoogle Scholar
  48. Daniel Madden, Yuxuan Liu, Haowei Yu, Mustafa Feyyaz Sonbudak, Giovanni M Troiano, and Casper Harteveld. 2021. "Why Are You Playing Games? You Are a Girl!": Exploring Gender Biases in Esports. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1--15.Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Brendan Maher. 2016. Can a video game company tame toxic behaviour? Nature 531, 7596 (2016), 568--571.Google ScholarGoogle Scholar
  50. E Makuch. 2014. MOBAs are "notoriously toxic," says Strife developer. Retrieved July 6, 2015.Google ScholarGoogle Scholar
  51. Lavinia McLean and Mark D. Griffiths. 2019. Female Gamers' Experience of Online Harassment and Social Support in Online Gaming: A Qualitative Study. International Journal of Mental Health and Addiction 17, 4 (2019), 970--994. https://doi.org/10.1007/s11469-018--9962-0Google ScholarGoogle ScholarCross RefCross Ref
  52. "David Melhart, Antonios Liapis, and Georgios N. Yannakakis". "2021". "The Affect Game AnnotatIoN (AGAIN) Dataset". "IEEE Transactions on Affective Computing" ("2021"), "1--14". "in review".Google ScholarGoogle Scholar
  53. Marçal Mora-Cantallops and Miguel-Ángel Sicilia. 2018. MOBA games: A literature review. Entertainment computing 26 (2018), 128--138.Google ScholarGoogle Scholar
  54. Shane Murnion, William J Buchanan, Adrian Smales, and Gordon Russell. 2018. Machine learning and semantic analysis of in-game chat for cyberbullying. Computers & Security 76 (2018), 197--213.Google ScholarGoogle ScholarCross RefCross Ref
  55. M. Märtens, S. Shen, A. Iosup, and F. Kuipers. 2015. Toxicity detection in multiplayer online games. In 2015 International Workshop on Network and Systems Support for Games (NetGames). 1--6. https://doi.org/10.1109/NetGames.2015.7382991Google ScholarGoogle ScholarCross RefCross Ref
  56. Larissa Vivian Nägele, Merja Ryöppy, and Danielle Wilde. 2018. PDFI: Participatory design fiction with vulnerable users. In Proceedings of the 10th Nordic Conference on Human-Computer Interaction. 819--831.Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Joaquim AM Neto, Kazuki M Yokoyama, and Karin Becker. 2017. Studying toxic behavior influence and player chat in an online video game. In Proceedings of the International Conference on Web Intelligence. 26--33.Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Adewale Obadimu, Esther Mead, Muhammad Nihal Hussain, and Nitin Agarwal. 2019. Identifying toxicity within youtube video comment. In International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation. Springer, 214--223.Google ScholarGoogle ScholarCross RefCross Ref
  59. Stephanie M Ortiz. 2019. "You can say I got desensitized to it": How men of color cope with everyday racism in online gaming. Sociological Perspectives 62, 4 (2019), 572--588.Google ScholarGoogle ScholarCross RefCross Ref
  60. Hunter L Paul, Nicholas David Bowman, and Jaime Banks. 2015. The enjoyment of griefing in online games. Journal of Gaming & Virtual Worlds 7, 3 (2015), 243--258.Google ScholarGoogle ScholarCross RefCross Ref
  61. Ari Schlesinger, Kenton P O'Hara, and Alex S Taylor. 2018. Let's talk about race: Identity, chatbots, and AI. In Proceedings of the 2018 chi conference on human factors in computing systems. 1--14.Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Konstantinos Sechidis, Grigorios Tsoumakas, and Ioannis Vlahavas. 2011. On the stratification of multi-label data. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 145--158.Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. Sercan Sengün, Joni Salminen, Soon-gyo Jung, Peter Mawhorter, and Bernard J Jansen. 2019. Analyzing Hate Speech Toward Players from the MENA in League of Legends. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1--6.Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. Sercan Sengün, Joni Salminen, Peter Mawhorter, Soon-gyo Jung, and Bernard Jansen. 2019. Exploring the relationship between game content and culture-based toxicity: a case study of league of legends and MENA players. In Proceedings of the 30th ACM Conference on Hypertext and Social Media. 87--95.Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. Cuihua Shen, Qiusi Sun, Taeyoung Kim, Grace Wolff, Rabindra Ratan, and Dmitri Williams. 2020. Viral vitriol: Predictors and contagion of online toxicity in World of Tanks. Computers in Human Behavior 108 (2020), 106343. https://doi.org/10.1016/j.chb.2020.106343Google ScholarGoogle ScholarCross RefCross Ref
  66. Kenneth B Shores, Yilin He, Kristina L Swanenburg, Robert Kraut, and John Riedl. 2014. The identification of deviance and its impact on retention in a multiplayer game. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 1356--1365.Google ScholarGoogle ScholarDigital LibraryDigital Library
  67. Noah Smith. 2019. Racism, misogyny, death threats: Why can't the booming video-game industry curb toxicity? https://www.washingtonpost.com/technology/2019/02/26/racism-misogyny-death-threats-why-cant-boomingvideo- game-industry-curb-toxicity/. Accessed: 20-02--10.Google ScholarGoogle Scholar
  68. Lucy Sparrow, Martin Gibbs, and Michael Arnold. 2019. Apathetic villagers and the trolls who love them: Player amorality in online multiplayer games. In Proceedings of the 31st Australian Conference on Human-Computer-Interaction. 447--451.Google ScholarGoogle ScholarDigital LibraryDigital Library
  69. Wessel Stoop, Florian Kunneman, Antal van den Bosch, and Ben Miller. 2019. Detecting harassment in real-time as conversations develop. In Proceedings of the Third Workshop on Abusive Language Online. 19--24.Google ScholarGoogle Scholar
  70. Wai Yen Tang, Felix Reer, and Thorsten Quandt. 2020. Investigating sexual harassment in online video games: How personality and context factors are related to toxic sexual behaviors against fellow players. Aggressive Behavior 46, 1 (2020), 127--135.Google ScholarGoogle ScholarCross RefCross Ref
  71. Haydn Taylor. 2020. Valve introduces new measures to shut down team fortress 2 racist bot problem. https://www.gamesindustry.biz/articles/2020-06--18-valve-introduces-new-measures-to-shut-down-teamfortress- 2s-racist-bot-problem. Accessed: 20-02--10.Google ScholarGoogle Scholar
  72. Joseph J Thompson, Betty HM Leung, Mark R Blair, and Maite Taboada. 2017. Sentiment analysis of player chat messaging in the video game StarCraft 2: Extending a lexicon-based model. Knowledge-Based Systems 137 (2017), 149--162.Google ScholarGoogle ScholarDigital LibraryDigital Library
  73. Arjen Traas. 2017. The Impact of Toxic Behavior on Match Outcomes in DotA. Ph.D. Dissertation. Tilburg University.Google ScholarGoogle Scholar
  74. Selen Türkay, Jessica Formosa, Sonam Adinolf, Robert Cuthbert, and Roger Altizer. 2020. See No Evil, Hear No Evil, Speak No Evil: How Collegiate Players Define, Experience and Cope with Toxicity. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/10.1145/3313831.3376191Google ScholarGoogle ScholarDigital LibraryDigital Library
  75. Ubisoft. 2016. For Honor.Google ScholarGoogle Scholar
  76. Rebekah Valentine. 2020. Infinity Ward pledges to do more to moderate racist content in its games. https://www.gamesindustry.biz/articles/2020-06-03-infinity-ward-pledges-to-do-more-to-moderate-racistcontent- in-its-games. Accessed: 20-02--10.Google ScholarGoogle Scholar
  77. Mark Verschoor. 2016. Eating seeds as a pastime activity: Predicting toxicity in online game chat using in-game events. Master's thesis. Tilburg University.Google ScholarGoogle Scholar
  78. Julia Crouse Waddell and Wei Peng. 2014. Does it matter with whom you slay? The effects of competition, cooperation and relationship type among video game players. Computers in Human Behavior 38 (2014), 331--338.Google ScholarGoogle ScholarDigital LibraryDigital Library
  79. Wargaming. 2010. World of Tanks.Google ScholarGoogle Scholar
  80. Tony Xiao. 2019. Confronting Toxicity in Gaming: Going Beyond ?Mute". https://www.nytimes.com/2019/06/06/ learning/confronting-toxicity-in-gaming-going-beyond-mute.html. Accessed: 20-02--10.Google ScholarGoogle Scholar
  81. Georgios N Yannakakis and Julian Togelius. 2018. Artificial intelligence and games. Vol. 2. Springer.Google ScholarGoogle ScholarDigital LibraryDigital Library
  82. James Zou and Londa Schiebinger. 2018. AI can be sexist and racist-it's time to make it fair. Nature (2018), 324--326. https://doi.org/10.1038/d41586-018-05707--8Google ScholarGoogle Scholar

Index Terms

  1. For Honor, for Toxicity: Detecting Toxic Behavior through Gameplay

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Article Metrics

      • Downloads (Last 12 months)240
      • Downloads (Last 6 weeks)26

      Other Metrics

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!