skip to main content
research-article

How do Players and Developers of Citizen Science Games Conceptualize Skill Chains?

Authors Info & Claims
Published:06 October 2021Publication History
Skip Abstract Section

Abstract

For citizen science games (CSGs) to be successful in advancing scientific research, they must effectively train players. Designing tutorials for training can be aided through developing a skill chain of required skills and their dependencies, but skill chain development is an intensive process. In this work, we hypothesized that free recall may be a simpler yet effective method of directly eliciting skill chains. We elicited 23 skill chains from players and developers and augmented our reflexive thematic analysis with 11 semi-structured interviews in order to determine how players and developers conceptualize skill trees and whether free recall can be used as an alternative to more resource-intensive cognitive task analyses. We provide three main contributions: (1) a comparison of skill chain conceptualizations between players and developers and across prior literature; (2) insights to the process of free recall in eliciting CSG skill chains; and (3) a preliminary toolkit of CSG skill-based design recommendations based on our findings. We conclude CSG developers should: give the big picture up front; embrace social learning and paratext use; reinforce the intended structure of knowledge; situate learning within applicable, meaningful contexts; design for discovery and self-reflection; and encourage practice and learning beyond the tutorial. Free recall was ineffective for determining a traditional skill chain but was able to elicit the core gameplay loops, tutorial overviews, and some expert insights.

References

  1. Lada A Adamic. 2000. Zipf, power-laws, and pareto-a ranking tutorial. Xerox Palo Alto Research Center, Palo Alto, CA (2000).Google ScholarGoogle Scholar
  2. Vincent Aleven and Kenneth R Koedinger. 2013. Knowledge component (KC) approaches to learner modeling. Design Recommendations for Intelligent Tutoring Systems 1 (2013), 165--182.Google ScholarGoogle Scholar
  3. Erik Andersen, Eleanor O'Rourke, Yun-En Liu, Rich Snider, Jeff Lowdermilk, David Truong, Seth Cooper, and Zoran Popovic. 2012. The impact of tutorials on games of varying complexity. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 59--68.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Sylvester Arnab, Theodore Lim, Maira B Carvalho, Francesco Bellotti, Sara De Freitas, Sandy Louchart, Neil Suttie, Riccardo Berta, and Alessandro De Gloria. 2015. Mapping learning and game mechanics for serious games analysis. British Journal of Educational Technology 46, 2 (2015), 391--411.Google ScholarGoogle ScholarCross RefCross Ref
  5. Batu Aytemiz, Isaac Karth, Jesse Harder, Adam M Smith, and Jim Whitehead. 2018. Talin: A Framework for Dynamic Tutorials based on the Skill Atoms Theory. In Fourteenth Artificial Intelligence and Interactive Digital Entertainment Conference.Google ScholarGoogle Scholar
  6. Sander CJ Bakkes, Pieter HM Spronck, and Giel van Lankveld. 2012. Player behavioural modelling for video games. Entertainment Computing 3, 3 (2012), 71--79.Google ScholarGoogle ScholarCross RefCross Ref
  7. Aaron Bauer and Zoran Popović. 2017. Collaborative problem solving in an open-ended scientific discovery game. Proceedings of the ACM on human-computer interaction 1, CSCW (2017), 1--21.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77--101.Google ScholarGoogle Scholar
  9. Virginia Braun and Victoria Clarke. 2019. Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health 11, 4 (2019), 589--597.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Virginia Braun and Victoria Clarke. 2021. Can I use TA? Should I use TA? Should I not use TA? Comparing reflexive thematic analysis and other pattern-based qualitative analytic approaches. Counselling and Psychotherapy Research 21, 1 (2021), 37--47.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. William G. Chase and Herbert A. Simon. 1973. Perception in chess. Cognitive psychology 4, 1 (1973), 55--81.Google ScholarGoogle Scholar
  12. Michelene TH Chi, Paul J Feltovich, and Robert Glaser. 1981. Categorization and representation of physics problems by experts and novices. Cognitive science 5, 2 (1981), 121--152.Google ScholarGoogle Scholar
  13. Richard Clark. 2014. Cognitive task analysis for expert-based instruction in healthcare. In Handbook of research on educational communications and technology. Springer, 541--551.Google ScholarGoogle Scholar
  14. Richard E Clark, Carla M Pugh, Kenneth A Yates, Kenji Inaba, Donald J Green, and Maura E Sullivan. 2012. The use of cognitive task analysis to improve instructional descriptions of procedures. Journal of Surgical Research 173, 1 (2012), e37--e42.Google ScholarGoogle ScholarCross RefCross Ref
  15. Allan Collins, John Seely Brown, and Susan E Newman. 1988. Cognitive apprenticeship: Teaching the craft of reading, writing and mathematics. Thinking: The Journal of Philosophy for Children 8, 1 (1988), 2--10.Google ScholarGoogle ScholarCross RefCross Ref
  16. Daniel Cook. 2007. The chemistry of game design. Gamasutra. http://www.gamasutra.com/view/feature/1524/the_chemistry_of_game_design.php. (2007).Google ScholarGoogle Scholar
  17. Nancy J Cooke. 1999. Knowledge elicitation. Handbook of applied cognition (1999), 479--509.Google ScholarGoogle Scholar
  18. Seth Cooper. 2014. A framework for scientific discovery through video games. Morgan & Claypool.Google ScholarGoogle Scholar
  19. Seth Cooper, Firas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, Andrew Leaver-Fay, David Baker, Zoran Popović, and others. 2010. Predicting protein structures with a multiplayer online game. Nature 466, 7307 (2010), 756--760.Google ScholarGoogle ScholarCross RefCross Ref
  20. Benjamin Cowley and Darryl Charles. 2016. Behavlets: a method for practical player modelling using psychology-based player traits and domain specific features. User Modeling and User-Adapted Interaction 26, 2--3 (2016), 257--306.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Beth Crandall, Gary Klein, Gary A Klein, Robert R Hoffman, and others. 2006. Working minds: A practitioner's guide to cognitive task analysis. Mit Press.Google ScholarGoogle Scholar
  22. John W. Creswell and Dana L. Miller. 2000. Determining validity in qualitative inquiry. Theory Into Practice 39, 3 (Aug. 2000), 124--130. http://dx.doi.org/10.1207/s15430421tip3903_2Google ScholarGoogle ScholarCross RefCross Ref
  23. Vickie Curtis. 2015. Motivation to participate in an online citizen science game: A study of Foldit. Science Communication 37, 6 (2015), 723--746.Google ScholarGoogle ScholarCross RefCross Ref
  24. Vickie Curtis. 2018. Who Takes Part in Online Citizen Science? In Online Citizen Science and the Widening of Academia. Springer, 45--68.Google ScholarGoogle Scholar
  25. Michel C. Desmarais and Ryan S.J.d. Baker. 2012. A review of recent advances in learner and skill modeling in intelligent learning environments. User Modeling and User-Adapted Interaction 22, 1--2 (2012), 9--38.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Sebastian Deterding. 2015. The lens of intrinsic skill atoms: A method for gameful design. Human-Computer Interaction 30, 3--4 (2015), 294--335.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Scott Donaldson. 2017. Mechanics and metagame: Exploring binary expertise in League of Legends. Games and Culture 12, 5 (2017), 426--444.Google ScholarGoogle ScholarCross RefCross Ref
  28. Carlos Díaz, Marisa Ponti, Pinja Haikka, Rajiv Basaiawmoit, and Jacob Sherson. 2020. More than data gatherers: exploring player experience in a citizen science game. Qual User Exp 5, 1 (Dec. 2020), 1. http://dx.doi.org/10.1007/s41233-019-0030--8Google ScholarGoogle ScholarCross RefCross Ref
  29. Joey R Fanfarelli. 2018. Expertise in professional overwatch play. International Journal of Gaming and Computer-Mediated Simulations (IJGCMS) 10, 1 (2018), 1--22.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Zach Gage. 2014. Designing for Problem-Solvers. PRACTICE: Game Design in Detail. (2014). Retrieved from http://stfj.net/DesigningForProblem-Solvers/ on May 13, 2021.Google ScholarGoogle Scholar
  31. Barry Garelick. 2013. Problem Solving: Moving from Routine to Nonroutine and Beyond. Notices of the AMs 60, 10 (2013).Google ScholarGoogle Scholar
  32. James Paul Gee. 2005. Good video games and good learning. In Phi Kappa Phi Forum, Vol. 85. The Honor Society of Phi Kappa Phi, 33.Google ScholarGoogle Scholar
  33. Carolina González, Juan C Burguillo, and Mart'in Llamas. 2006. A qualitative comparison of techniques for student modeling in intelligent tutoring systems. In Proceedings. Frontiers in Education. 36th Annual Conference. IEEE, 13--18.Google ScholarGoogle ScholarCross RefCross Ref
  34. Charles Goodwin. 1994. Professional vision. American Anthropologist 96, 3 (1994), 606.Google ScholarGoogle Scholar
  35. M.P. Jacob Habgood and Shaaron E. Ainsworth. 2011. Motivating children to learn effectively: Exploring the value of intrinsic integration in educational games. The Journal of the Learning Sciences 20, 2 (2011), 169--206.Google ScholarGoogle ScholarCross RefCross Ref
  36. Joseph Hesketh, Christoph Sebastian Deterding, and Jeremy Gow. 2020. How Players Learn Team-versus-Team Esports: First Results from A Grounded Theory Study. In DiGRA'20 Abstract-Proceedings of the 2020 DiGRA International Conference. York.Google ScholarGoogle Scholar
  37. Robert R. Hoffman, John W. Coffey, Mary Jo Carnot, and Joseph D. Novak. 2002. An empirical comparison of methods for eliciting and modeling expert knowledge. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 46. SAGE Publications Sage CA: Los Angeles, CA, 482--486.Google ScholarGoogle Scholar
  38. Britton Horn. 2019. A Skill-based Framework to Analyze the Generation and Presentation of Instructional Videogame Content. Ph.D. Dissertation. Northeastern University.Google ScholarGoogle Scholar
  39. Britton Horn, Seth Cooper, and Sebastian Deterding. 2017. Adapting cognitive task analysis to elicit the skill chain of a game. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play. 277--289.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Britton Horn, Josh Aaron Miller, Gillian Smith, and Seth Cooper. 2018. A Monte Carlo approach to skill-based automated playtesting. In Fourteenth Artificial Intelligence and Interactive Digital Entertainment Conference.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Jeff Huang, Eddie Yan, Gifford Cheung, Nachiappan Nagappan, and Thomas Zimmermann. 2017. Master maker: Understanding gaming skill through practice and habit from gameplay behavior. Topics in cognitive science 9, 2 (2017), 437--466.Google ScholarGoogle Scholar
  42. Robin Hunicke, Marc LeBlanc, and Robert Zubek. 2004. MDA: A formal approach to game design and game research. In Proceedings of the AAAI Workshop on Challenges in Game AI, Vol. 4. 1722.Google ScholarGoogle Scholar
  43. Ioanna Iacovides, Anna L. Cox, Patrick McAndrew, James Aczel, and Eileen Scanlon. 2015. Game-play breakdowns and breakthroughs: exploring the relationship between action, understanding, and involvement. Human-computer interaction 30, 3--4 (2015), 202--231.Google ScholarGoogle Scholar
  44. Ioanna Iacovides, Charlene Jennett, Cassandra Cornish-Trestrail, and Anna L Cox. 2013. Do games attract or sustain engagement in citizen science?: a study of volunteer motivations. In CHI'13 Extended Abstracts on Human Factors in Computing Systems. ACM, 1101--1106.Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Charlene Jennett, Laure Kloetzer, Daniel Schneider, Ioanna Iacovides, Anna Cox, Margaret Gold, Brian Fuchs, Alexandra Eveleigh, Kathleen Methieu, Zoya Ajani, and others. 2016. Motivations, learning and creativity in online citizen science. Journal of Science Communication 15, 3 (2016).Google ScholarGoogle ScholarCross RefCross Ref
  46. Daniel Kahneman. 2011. Thinking, fast and slow. Macmillan.Google ScholarGoogle Scholar
  47. Pavan Kantharaju, Katelyn Alderfer, Jichen Zhu, Bruce Char, Brian Smith, and Santiago Ontanón. 2018. Tracing player knowledge in a parallel programming educational game. In Fourteenth Artificial Intelligence and Interactive Digital Entertainment Conference.Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Benjamin Ernest Keep. 2018. Becoming Expert Problem Solvers: A Case Study in what Develops and how. Ph.D. Dissertation. Stanford University.Google ScholarGoogle Scholar
  49. Diana Laurillard. 2013. Teaching as a design science: Building pedagogical patterns for learning and technology. Routledge.Google ScholarGoogle Scholar
  50. Jeehyung Lee, Wipapat Kladwang, Minjae Lee, Daniel Cantu, Martin Azizyan, Hanjoo Kim, Alex Limpaecher, Snehal Gaikwad, Sungroh Yoon, Adrien Treuille, and others. 2014. RNA design rules from a massive open laboratory. Proceedings of the National Academy of Sciences 111, 6 (2014), 2122--2127.Google ScholarGoogle ScholarCross RefCross Ref
  51. Kisuk Lee. 2015. Crowd Intelligence in EyeWire. Technical report. (2015). Retrieved from http://kisuklee.wikidot.com/research#toc2 on May 6, 2021.Google ScholarGoogle Scholar
  52. John Lindstedt and Wayne Gray. 2013. Extreme expertise: Exploring expert behavior in Tetris. In Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 35.Google ScholarGoogle Scholar
  53. Tim Marsh. 2016. Slow serious games, interactions and play: Designing for positive and serious experience and reflection. Entertainment computing 14 (2016), 45--53.Google ScholarGoogle Scholar
  54. Josh Aaron Miller, Vivian Lee, Seth Cooper, and Magy Seif El-Nasr. 2019a. Large-Scale Analysis of Visualization Options in a Citizen Science Game. In Extended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts. 535--542.Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Josh Aaron Miller, Uttkarsh Narayan, Matthew Hantsbarger, Seth Cooper, and Magy Seif El-Nasr. 2019b. Expertise and engagement: re-designing citizen science games with players' minds in mind. In Proceedings of the 14th International Conference on the Foundations of Digital Games. 1--11.Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. Huy Anh Nguyen, Xinying Hou, John Stamper, and Bruce M McLaren. 2020. Moving beyond test scores: analyzing the effectiveness of a digital learning game through learning analytics. In Proceedings of The 13th International Conference on Educational Data Mining (EDM 2020). 487--495.Google ScholarGoogle Scholar
  57. David Osborne. 2016. Tales From a Crowdsourced Game Design. (2016). https://www.gdcvault.com/browse/gdc-16/play/1023216Google ScholarGoogle Scholar
  58. Marisa Ponti, Igor Stankovic, Wolmet Barendregt, Bruno Kestemont, and Lyn Bain. 2018. Chefs know more than just recipes: professional vision in a citizen science game. Human Computation 5 (2018), 1--12.Google ScholarGoogle ScholarCross RefCross Ref
  59. Nathan Prestopnik, Kevin Crowston, and Jun Wang. 2014. Exploring data quality in games with a purpose. iConference 2014 Proceedings (2014).Google ScholarGoogle Scholar
  60. Ruziana Binti Mohamad Rasli, Faudziah Ahmad, and Siti Sakira Kamaruddin. 2014. A comparative study of conceptual graph and concept map. Journal of Engineering and Applied Sciences 9, 9 (2014), 1442--1446.Google ScholarGoogle Scholar
  61. Anurag Sarkar and Seth Cooper. 2019. Using a Disjoint Skill Model for Game and Task Difficulty in Human Computation Games. In Extended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts. 661--669.Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Jesse Schell. 2008. The Art of Game Design: A book of lenses. CRC press.Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. Dale H Schunk. 2012. Learning theories an educational perspective sixth edition. Pearson.Google ScholarGoogle Scholar
  64. Norman J Slamecka and Peter Graf. 1978. The generation effect: Delineation of a phenomenon. Journal of experimental Psychology: Human learning and Memory 4, 6 (1978), 592.Google ScholarGoogle ScholarCross RefCross Ref
  65. Bethesda Softworks. 1994--2020. The Elder Scrolls. (1994--2020).Google ScholarGoogle Scholar
  66. Kurt Squire. 2012. Designed Cultures. In Games, learning, and society: Learning and meaning in the digital age, Constance Steinkuehler, Kurt Squire, and Sasha Barab (Eds.). Cambridge University Press, Chapter 2.Google ScholarGoogle Scholar
  67. Igor Stankovic, Mar'ia Postigo Camps, Daniel Cuadrado Sánchez, Miguel A Luengo-Oroz, and Marisa Ponti. 2017. Performance of Players and Data Quality in a Citizen Science Casual Game. (2017). https://osf.io/preprints/socarxiv/vztxn/Google ScholarGoogle Scholar
  68. Ramine Tinati, Markus Luczak-Roesch, Elena Simperl, and Wendy Hall. 2017. An investigation of player motivations in Eyewire, a gamified citizen science project. Computers in Human Behavior 73 (2017), 527--540.Google ScholarGoogle ScholarDigital LibraryDigital Library
  69. Gustavo F Tondello and Lennart E Nacke. 2019. A Pilot Study of a Digital Skill Tree in Gameful Education. (2019).Google ScholarGoogle Scholar
  70. Valve. 2007. Portal. (2007).Google ScholarGoogle Scholar
  71. Jeroen JG Van Merriënboer, Otto Jelsma, and Fred GWC Paas. 1992. Training for reflective expertise: A four-component instructional design model for complex cognitive skills. Educational Technology Research and Development 40, 2 (1992), 23--43.Google ScholarGoogle ScholarCross RefCross Ref
  72. Jeroen JG Van Merriënboer and Paul A Kirschner. 2017. Ten steps to complex learning: A systematic approach to four-component instructional design. Routledge.Google ScholarGoogle Scholar
  73. Luis Von Ahn. 2008. Human computation. In 2008 IEEE 24th international conference on data engineering. IEEE, 1--2.Google ScholarGoogle ScholarDigital LibraryDigital Library
  74. Shuhan Wang, Fang He, and Erik Andersen. 2017. A unified framework for knowledge assessment and progression analysis and design. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 937--948.Google ScholarGoogle ScholarDigital LibraryDigital Library
  75. G. Christopher Williams. 2009. Active Learning: The Pedagogy of the Game Tutorial. (2009). https://www.popmatters.com/111485-active-learning-the-pedagogy-of-the-game-tutorial-2496084217.htmlGoogle ScholarGoogle Scholar

Index Terms

  1. How do Players and Developers of Citizen Science Games Conceptualize Skill Chains?

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!