Abstract

For citizen science games (CSGs) to be successful in advancing scientific research, they must effectively train players. Designing tutorials for training can be aided through developing a skill chain of required skills and their dependencies, but skill chain development is an intensive process. In this work, we hypothesized that free recall may be a simpler yet effective method of directly eliciting skill chains. We elicited 23 skill chains from players and developers and augmented our reflexive thematic analysis with 11 semi-structured interviews in order to determine how players and developers conceptualize skill trees and whether free recall can be used as an alternative to more resource-intensive cognitive task analyses. We provide three main contributions: (1) a comparison of skill chain conceptualizations between players and developers and across prior literature; (2) insights to the process of free recall in eliciting CSG skill chains; and (3) a preliminary toolkit of CSG skill-based design recommendations based on our findings. We conclude CSG developers should: give the big picture up front; embrace social learning and paratext use; reinforce the intended structure of knowledge; situate learning within applicable, meaningful contexts; design for discovery and self-reflection; and encourage practice and learning beyond the tutorial. Free recall was ineffective for determining a traditional skill chain but was able to elicit the core gameplay loops, tutorial overviews, and some expert insights.
- Lada A Adamic. 2000. Zipf, power-laws, and pareto-a ranking tutorial. Xerox Palo Alto Research Center, Palo Alto, CA (2000).Google Scholar
- Vincent Aleven and Kenneth R Koedinger. 2013. Knowledge component (KC) approaches to learner modeling. Design Recommendations for Intelligent Tutoring Systems 1 (2013), 165--182.Google Scholar
- Erik Andersen, Eleanor O'Rourke, Yun-En Liu, Rich Snider, Jeff Lowdermilk, David Truong, Seth Cooper, and Zoran Popovic. 2012. The impact of tutorials on games of varying complexity. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 59--68.Google Scholar
Digital Library
- Sylvester Arnab, Theodore Lim, Maira B Carvalho, Francesco Bellotti, Sara De Freitas, Sandy Louchart, Neil Suttie, Riccardo Berta, and Alessandro De Gloria. 2015. Mapping learning and game mechanics for serious games analysis. British Journal of Educational Technology 46, 2 (2015), 391--411.Google Scholar
Cross Ref
- Batu Aytemiz, Isaac Karth, Jesse Harder, Adam M Smith, and Jim Whitehead. 2018. Talin: A Framework for Dynamic Tutorials based on the Skill Atoms Theory. In Fourteenth Artificial Intelligence and Interactive Digital Entertainment Conference.Google Scholar
- Sander CJ Bakkes, Pieter HM Spronck, and Giel van Lankveld. 2012. Player behavioural modelling for video games. Entertainment Computing 3, 3 (2012), 71--79.Google Scholar
Cross Ref
- Aaron Bauer and Zoran Popović. 2017. Collaborative problem solving in an open-ended scientific discovery game. Proceedings of the ACM on human-computer interaction 1, CSCW (2017), 1--21.Google Scholar
Digital Library
- Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77--101.Google Scholar
- Virginia Braun and Victoria Clarke. 2019. Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health 11, 4 (2019), 589--597.Google Scholar
Digital Library
- Virginia Braun and Victoria Clarke. 2021. Can I use TA? Should I use TA? Should I not use TA? Comparing reflexive thematic analysis and other pattern-based qualitative analytic approaches. Counselling and Psychotherapy Research 21, 1 (2021), 37--47.Google Scholar
Digital Library
- William G. Chase and Herbert A. Simon. 1973. Perception in chess. Cognitive psychology 4, 1 (1973), 55--81.Google Scholar
- Michelene TH Chi, Paul J Feltovich, and Robert Glaser. 1981. Categorization and representation of physics problems by experts and novices. Cognitive science 5, 2 (1981), 121--152.Google Scholar
- Richard Clark. 2014. Cognitive task analysis for expert-based instruction in healthcare. In Handbook of research on educational communications and technology. Springer, 541--551.Google Scholar
- Richard E Clark, Carla M Pugh, Kenneth A Yates, Kenji Inaba, Donald J Green, and Maura E Sullivan. 2012. The use of cognitive task analysis to improve instructional descriptions of procedures. Journal of Surgical Research 173, 1 (2012), e37--e42.Google Scholar
Cross Ref
- Allan Collins, John Seely Brown, and Susan E Newman. 1988. Cognitive apprenticeship: Teaching the craft of reading, writing and mathematics. Thinking: The Journal of Philosophy for Children 8, 1 (1988), 2--10.Google Scholar
Cross Ref
- Daniel Cook. 2007. The chemistry of game design. Gamasutra. http://www.gamasutra.com/view/feature/1524/the_chemistry_of_game_design.php. (2007).Google Scholar
- Nancy J Cooke. 1999. Knowledge elicitation. Handbook of applied cognition (1999), 479--509.Google Scholar
- Seth Cooper. 2014. A framework for scientific discovery through video games. Morgan & Claypool.Google Scholar
- Seth Cooper, Firas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, Andrew Leaver-Fay, David Baker, Zoran Popović, and others. 2010. Predicting protein structures with a multiplayer online game. Nature 466, 7307 (2010), 756--760.Google Scholar
Cross Ref
- Benjamin Cowley and Darryl Charles. 2016. Behavlets: a method for practical player modelling using psychology-based player traits and domain specific features. User Modeling and User-Adapted Interaction 26, 2--3 (2016), 257--306.Google Scholar
Digital Library
- Beth Crandall, Gary Klein, Gary A Klein, Robert R Hoffman, and others. 2006. Working minds: A practitioner's guide to cognitive task analysis. Mit Press.Google Scholar
- John W. Creswell and Dana L. Miller. 2000. Determining validity in qualitative inquiry. Theory Into Practice 39, 3 (Aug. 2000), 124--130. http://dx.doi.org/10.1207/s15430421tip3903_2Google Scholar
Cross Ref
- Vickie Curtis. 2015. Motivation to participate in an online citizen science game: A study of Foldit. Science Communication 37, 6 (2015), 723--746.Google Scholar
Cross Ref
- Vickie Curtis. 2018. Who Takes Part in Online Citizen Science? In Online Citizen Science and the Widening of Academia. Springer, 45--68.Google Scholar
- Michel C. Desmarais and Ryan S.J.d. Baker. 2012. A review of recent advances in learner and skill modeling in intelligent learning environments. User Modeling and User-Adapted Interaction 22, 1--2 (2012), 9--38.Google Scholar
Digital Library
- Sebastian Deterding. 2015. The lens of intrinsic skill atoms: A method for gameful design. Human-Computer Interaction 30, 3--4 (2015), 294--335.Google Scholar
Digital Library
- Scott Donaldson. 2017. Mechanics and metagame: Exploring binary expertise in League of Legends. Games and Culture 12, 5 (2017), 426--444.Google Scholar
Cross Ref
- Carlos Díaz, Marisa Ponti, Pinja Haikka, Rajiv Basaiawmoit, and Jacob Sherson. 2020. More than data gatherers: exploring player experience in a citizen science game. Qual User Exp 5, 1 (Dec. 2020), 1. http://dx.doi.org/10.1007/s41233-019-0030--8Google Scholar
Cross Ref
- Joey R Fanfarelli. 2018. Expertise in professional overwatch play. International Journal of Gaming and Computer-Mediated Simulations (IJGCMS) 10, 1 (2018), 1--22.Google Scholar
Digital Library
- Zach Gage. 2014. Designing for Problem-Solvers. PRACTICE: Game Design in Detail. (2014). Retrieved from http://stfj.net/DesigningForProblem-Solvers/ on May 13, 2021.Google Scholar
- Barry Garelick. 2013. Problem Solving: Moving from Routine to Nonroutine and Beyond. Notices of the AMs 60, 10 (2013).Google Scholar
- James Paul Gee. 2005. Good video games and good learning. In Phi Kappa Phi Forum, Vol. 85. The Honor Society of Phi Kappa Phi, 33.Google Scholar
- Carolina González, Juan C Burguillo, and Mart'in Llamas. 2006. A qualitative comparison of techniques for student modeling in intelligent tutoring systems. In Proceedings. Frontiers in Education. 36th Annual Conference. IEEE, 13--18.Google Scholar
Cross Ref
- Charles Goodwin. 1994. Professional vision. American Anthropologist 96, 3 (1994), 606.Google Scholar
- M.P. Jacob Habgood and Shaaron E. Ainsworth. 2011. Motivating children to learn effectively: Exploring the value of intrinsic integration in educational games. The Journal of the Learning Sciences 20, 2 (2011), 169--206.Google Scholar
Cross Ref
- Joseph Hesketh, Christoph Sebastian Deterding, and Jeremy Gow. 2020. How Players Learn Team-versus-Team Esports: First Results from A Grounded Theory Study. In DiGRA'20 Abstract-Proceedings of the 2020 DiGRA International Conference. York.Google Scholar
- Robert R. Hoffman, John W. Coffey, Mary Jo Carnot, and Joseph D. Novak. 2002. An empirical comparison of methods for eliciting and modeling expert knowledge. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 46. SAGE Publications Sage CA: Los Angeles, CA, 482--486.Google Scholar
- Britton Horn. 2019. A Skill-based Framework to Analyze the Generation and Presentation of Instructional Videogame Content. Ph.D. Dissertation. Northeastern University.Google Scholar
- Britton Horn, Seth Cooper, and Sebastian Deterding. 2017. Adapting cognitive task analysis to elicit the skill chain of a game. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play. 277--289.Google Scholar
Digital Library
- Britton Horn, Josh Aaron Miller, Gillian Smith, and Seth Cooper. 2018. A Monte Carlo approach to skill-based automated playtesting. In Fourteenth Artificial Intelligence and Interactive Digital Entertainment Conference.Google Scholar
Digital Library
- Jeff Huang, Eddie Yan, Gifford Cheung, Nachiappan Nagappan, and Thomas Zimmermann. 2017. Master maker: Understanding gaming skill through practice and habit from gameplay behavior. Topics in cognitive science 9, 2 (2017), 437--466.Google Scholar
- Robin Hunicke, Marc LeBlanc, and Robert Zubek. 2004. MDA: A formal approach to game design and game research. In Proceedings of the AAAI Workshop on Challenges in Game AI, Vol. 4. 1722.Google Scholar
- Ioanna Iacovides, Anna L. Cox, Patrick McAndrew, James Aczel, and Eileen Scanlon. 2015. Game-play breakdowns and breakthroughs: exploring the relationship between action, understanding, and involvement. Human-computer interaction 30, 3--4 (2015), 202--231.Google Scholar
- Ioanna Iacovides, Charlene Jennett, Cassandra Cornish-Trestrail, and Anna L Cox. 2013. Do games attract or sustain engagement in citizen science?: a study of volunteer motivations. In CHI'13 Extended Abstracts on Human Factors in Computing Systems. ACM, 1101--1106.Google Scholar
Digital Library
- Charlene Jennett, Laure Kloetzer, Daniel Schneider, Ioanna Iacovides, Anna Cox, Margaret Gold, Brian Fuchs, Alexandra Eveleigh, Kathleen Methieu, Zoya Ajani, and others. 2016. Motivations, learning and creativity in online citizen science. Journal of Science Communication 15, 3 (2016).Google Scholar
Cross Ref
- Daniel Kahneman. 2011. Thinking, fast and slow. Macmillan.Google Scholar
- Pavan Kantharaju, Katelyn Alderfer, Jichen Zhu, Bruce Char, Brian Smith, and Santiago Ontanón. 2018. Tracing player knowledge in a parallel programming educational game. In Fourteenth Artificial Intelligence and Interactive Digital Entertainment Conference.Google Scholar
Digital Library
- Benjamin Ernest Keep. 2018. Becoming Expert Problem Solvers: A Case Study in what Develops and how. Ph.D. Dissertation. Stanford University.Google Scholar
- Diana Laurillard. 2013. Teaching as a design science: Building pedagogical patterns for learning and technology. Routledge.Google Scholar
- Jeehyung Lee, Wipapat Kladwang, Minjae Lee, Daniel Cantu, Martin Azizyan, Hanjoo Kim, Alex Limpaecher, Snehal Gaikwad, Sungroh Yoon, Adrien Treuille, and others. 2014. RNA design rules from a massive open laboratory. Proceedings of the National Academy of Sciences 111, 6 (2014), 2122--2127.Google Scholar
Cross Ref
- Kisuk Lee. 2015. Crowd Intelligence in EyeWire. Technical report. (2015). Retrieved from http://kisuklee.wikidot.com/research#toc2 on May 6, 2021.Google Scholar
- John Lindstedt and Wayne Gray. 2013. Extreme expertise: Exploring expert behavior in Tetris. In Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 35.Google Scholar
- Tim Marsh. 2016. Slow serious games, interactions and play: Designing for positive and serious experience and reflection. Entertainment computing 14 (2016), 45--53.Google Scholar
- Josh Aaron Miller, Vivian Lee, Seth Cooper, and Magy Seif El-Nasr. 2019a. Large-Scale Analysis of Visualization Options in a Citizen Science Game. In Extended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts. 535--542.Google Scholar
Digital Library
- Josh Aaron Miller, Uttkarsh Narayan, Matthew Hantsbarger, Seth Cooper, and Magy Seif El-Nasr. 2019b. Expertise and engagement: re-designing citizen science games with players' minds in mind. In Proceedings of the 14th International Conference on the Foundations of Digital Games. 1--11.Google Scholar
Digital Library
- Huy Anh Nguyen, Xinying Hou, John Stamper, and Bruce M McLaren. 2020. Moving beyond test scores: analyzing the effectiveness of a digital learning game through learning analytics. In Proceedings of The 13th International Conference on Educational Data Mining (EDM 2020). 487--495.Google Scholar
- David Osborne. 2016. Tales From a Crowdsourced Game Design. (2016). https://www.gdcvault.com/browse/gdc-16/play/1023216Google Scholar
- Marisa Ponti, Igor Stankovic, Wolmet Barendregt, Bruno Kestemont, and Lyn Bain. 2018. Chefs know more than just recipes: professional vision in a citizen science game. Human Computation 5 (2018), 1--12.Google Scholar
Cross Ref
- Nathan Prestopnik, Kevin Crowston, and Jun Wang. 2014. Exploring data quality in games with a purpose. iConference 2014 Proceedings (2014).Google Scholar
- Ruziana Binti Mohamad Rasli, Faudziah Ahmad, and Siti Sakira Kamaruddin. 2014. A comparative study of conceptual graph and concept map. Journal of Engineering and Applied Sciences 9, 9 (2014), 1442--1446.Google Scholar
- Anurag Sarkar and Seth Cooper. 2019. Using a Disjoint Skill Model for Game and Task Difficulty in Human Computation Games. In Extended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts. 661--669.Google Scholar
Digital Library
- Jesse Schell. 2008. The Art of Game Design: A book of lenses. CRC press.Google Scholar
Digital Library
- Dale H Schunk. 2012. Learning theories an educational perspective sixth edition. Pearson.Google Scholar
- Norman J Slamecka and Peter Graf. 1978. The generation effect: Delineation of a phenomenon. Journal of experimental Psychology: Human learning and Memory 4, 6 (1978), 592.Google Scholar
Cross Ref
- Bethesda Softworks. 1994--2020. The Elder Scrolls. (1994--2020).Google Scholar
- Kurt Squire. 2012. Designed Cultures. In Games, learning, and society: Learning and meaning in the digital age, Constance Steinkuehler, Kurt Squire, and Sasha Barab (Eds.). Cambridge University Press, Chapter 2.Google Scholar
- Igor Stankovic, Mar'ia Postigo Camps, Daniel Cuadrado Sánchez, Miguel A Luengo-Oroz, and Marisa Ponti. 2017. Performance of Players and Data Quality in a Citizen Science Casual Game. (2017). https://osf.io/preprints/socarxiv/vztxn/Google Scholar
- Ramine Tinati, Markus Luczak-Roesch, Elena Simperl, and Wendy Hall. 2017. An investigation of player motivations in Eyewire, a gamified citizen science project. Computers in Human Behavior 73 (2017), 527--540.Google Scholar
Digital Library
- Gustavo F Tondello and Lennart E Nacke. 2019. A Pilot Study of a Digital Skill Tree in Gameful Education. (2019).Google Scholar
- Valve. 2007. Portal. (2007).Google Scholar
- Jeroen JG Van Merriënboer, Otto Jelsma, and Fred GWC Paas. 1992. Training for reflective expertise: A four-component instructional design model for complex cognitive skills. Educational Technology Research and Development 40, 2 (1992), 23--43.Google Scholar
Cross Ref
- Jeroen JG Van Merriënboer and Paul A Kirschner. 2017. Ten steps to complex learning: A systematic approach to four-component instructional design. Routledge.Google Scholar
- Luis Von Ahn. 2008. Human computation. In 2008 IEEE 24th international conference on data engineering. IEEE, 1--2.Google Scholar
Digital Library
- Shuhan Wang, Fang He, and Erik Andersen. 2017. A unified framework for knowledge assessment and progression analysis and design. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 937--948.Google Scholar
Digital Library
- G. Christopher Williams. 2009. Active Learning: The Pedagogy of the Game Tutorial. (2009). https://www.popmatters.com/111485-active-learning-the-pedagogy-of-the-game-tutorial-2496084217.htmlGoogle Scholar
Index Terms
How do Players and Developers of Citizen Science Games Conceptualize Skill Chains?
Recommendations
The differences in motivations of online game players and offline game players: A combined analysis of three studies at higher education level
Computer games have become a highly popular form of entertainment and have had a large impact on how University students spend their leisure time. Due to their highly motivating properties computer games have come to the attention of educationalists who ...
n-Player impartial combinatorial games with random players
In this paper we develop a method of analyzing n-player impartial combinatorial games where n - 1 players behave optimally while one of the players plays randomly. More specifically, we develop a function g r that maps impartial combinatorial games to n-...
Adapting Cognitive Task Analysis to Elicit the Skill Chain of a Game
CHI PLAY '17: Proceedings of the Annual Symposium on Computer-Human Interaction in PlayPlaying a game is a complex skill comprising a set of more basic skills. These skills commonly map onto the main mechanics of the game, and build and depend on each other in a nested learning hierarchy, which game designers have modeled as skill chains ...






Comments