research-article

Designing for Citizen Data Analysis: A Cross-Sectional Case Study of a Multi-Domain Citizen Science Platform

Authors:

Markus Luczak-Rösch,

Robert Simpson,

Nigel ShadboltAuthors Info & Claims

CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

Pages 4069 - 4078

https://doi.org/10.1145/2702123.2702420

Published: 18 April 2015 Publication History

Abstract

Designing an effective and sustainable citizen science (CS)project requires consideration of a great number of factors. This makes the overall process unpredictable, even when a sound, user-centred design approach is followed by an experienced team of UX designers. Moreover, when such systems are deployed, the complexity of the resulting interactions challenges any attempt to generalisation from retrospective analysis. In this paper, we present a case study of the largest single platform of citizen driven data analysis projects to date, the Zooniverse. By eliciting, through structured reflection, experiences of core members of its design team, our grounded analysis yielded four sets of themes, focusing on Task Specificity, Community Development, Task Design and Public Relations and Engagement, supported by two-to-four specific design claims each. For each, we propose a set of design claims (DCs), drawing comparisons to the literature on crowdsourcing and online communities to contextualise our findings.

References

[1]

Arguello, J., Butler, B., and Joyce, E. Talk to me: foundations for successful individual-group interactions in online communities. Proceedings of the ... (2006), 1--10.

Digital Library

[2]

Backstrom, L., Huttenlocher, D., Kleinberg, J., and Lan, X. Group formation in large social networks: Membership, growth, and evolution. In Proceedings of the 12th ACM SIGKDD, KDD '06, ACM (New York, NY, USA, 2006), 44--54.

Digital Library

[3]

Bruckman, A., and Resnick, M. The mediamoo project constructionism and professional community. Convergence: The International Journal of Research into New Media Technologies 1, 1 (1995), 94--109.

[4]

C, B. Helping to get you started reflecting on your experiences, 3rd edition ed. Oxford Blackwell, 2004.

[5]

Cardamone, C., Schawinski, K., Sarzi, M., Bamford, S. P., Bennert, N., Urry, C., Lintott, C., Keel, W. C., Parejko, J., Nichol, R. C., et al. Galaxy zoo green peas: discovery of a class of compact extremely star-forming galaxies. Monthly Notices of the Royal Astronomical Society 399, 3 (2009), 1191--1205.

[6]

Cardamone, C. N. The story of the peas: Writing a scientific paper. http://blog.galaxyzoo.org/2009/07/02/the-storyof-the-peas-writing-a-scientific-paper, 2009.

[7]

Cooper, C. Zen in the art of citizen science: Apps for collective discovery and the 1 percent rule of the web. http://blogs.scientificamerican.com/guestblog/2013/09/11/zen-in-the-art-of-citizenscience-apps-for-collective-discovery-and-the1-rule-of-the-web/, 2013.

[8]

Cordero, P., Lucks, J. B., and Das, R. An rna mapping database for curating rna structure mapping experiments. Bioinformatics 28, 22 (2012), 3006--3008.

Digital Library

[9]

Dawson, R., and Bynghall, S. Getting Results from Crowds. Advanced Human Technologies, 2012.

[10]

Druschke, C. G., and Seltzer, C. E. Failures of engagement: Lessons learned from a citizen science pilot study. Applied Environmental Education & Communication 11, 3--4 (2012), 178--188.

[11]

Ebner, W., Leimeister, J. M., and Krcmar, H. Community engineering for innovations: the ideas competition as a method to nurture a virtual community for innovations. R&D Management 39, 4 (2009), 342--356.

[12]

Fortson, L., Masters, K., Nichol, R., Borne, K., Edmondson, E., Lintott, C., Raddick, J., Schawinski, K., and Wallin, J. Galaxy zoo: Morphological classification and citizen science. arXiv preprint arXiv:1104.5513 (2011).

[13]

Harper, F. M., Moy, D., and Konstan, J. A. Facts or friends?: Distinguishing informational and conversational questions in social q&a sites. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '09, ACM (New York, NY, USA, 2009), 759--768.

Digital Library

[14]

Hassman, K. D., Mugar, G., Østerlund, C., and Jackson, C. Learning at the seafloor, looking at the sky: The relationship between individual tasks and collaborative engagement in two citizen science projects. In Computer Supported Collaborative Learning (2013).

[15]

Heinzelman, J., and Waters, C. Crowdsourcing crisis information in disaster-affected haiti. http://www.humanitarianforum.org/data/files/ crowdsourcinghaiti.pdf, 2010.

[16]

Hilliard, C. Using structured reflection on a critical incident to develop a professional portfolio. Nursing standard 21, 2 (2006), 35--40.

[17]

http://OpenScientist.org. Finalizing a definition of "citizen science" and "citizen scientists". http://www.openscientist.org/2011/09/ finalizing-definition-of-citizen.html.

[18]

Kairam, S. R., Wang, D. J., and Leskovec, J. The life and death of online groups: Predicting group growth and longevity. In Proceedings of the Fifth ACM International Conference on Web Search and Data Mining, WSDM '12, ACM (New York, NY, USA, 2012), 673--682.

Digital Library

[19]

Kaufmann, N., Schulze, T., and Veit, D. More than fun and money. worker motivation in crowdsourcing-a study on mechanical turk. In AMCIS (2011).

[20]

Kawrykow, A., Roumanis, G., Kam, A., Kwak, D., Leung, C., Wu, C., Zarour, E., Sarmenta, L., Blanchette, M., Waldispühl, J., et al. Phylo: a citizen science approach for improving multiple sequence alignment. PloS one 7, 3 (2012), e31362.

[21]

Khatib, F., Cooper, S., Tyka, M. D., Xu, K., Makedon, I., Popović, Z., Baker, D., and Players, F. Algorithm discovery by protein folding game players. Proceedings of the National Academy of Sciences 108, 47 (2011), 18949--18953.

[22]

Kittur, A., and Kraut, R. E. Harnessing the wisdom of crowds in wikipedia: Quality through coordination. In Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work, CSCW '08, ACM (New York, NY, USA, 2008), 37--46.

Digital Library

[23]

Kluger, A. N., and DeNisi, A. The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological bulletin 119, 2 (1996), 254.

[24]

Kosmala, M. We need an 'i dont know' button! http://blog.snapshotserengeti.org/2012/12/14/ we-need-an-i-dont-know-button/, 2012.

[25]

Kraut, R. E., Resnick, P., Kiesler, S., Burke, M., Chen, Y., Kittur, N., Konstan, J., Ren, Y., and Riedl, J. Building successful online communities: Evidence-based social design. Mit Press, 2012.

[26]

Krieger, M., and Stark, E. Coordinating tasks on the commons: designing for personal goals, expertise and serendipity. Proceedings of the 27th International Conference on Human Factors in Computing Systems (CHI) (2009), 1485--1494.

Digital Library

[27]

Kuittinen, J., Kultima, A., Niemelä, J., and Paavilainen, J. Casual games discussion. In Proceedings of the 2007 Conference on Future Play, Future Play '07, ACM (New York, NY, USA, 2007), 105--112.

Digital Library

[28]

Kumar, R., Novak, J., and Tomkins, A. Structure and evolution of online social networks. Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '06 (2006), 611.

Digital Library

[29]

Lampe, C., Wash, R., Velasquez, A., and Ozkaya, E. Motivations to participate in online communities. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2010), 1927--1936.

Digital Library

[30]

Law, E., and von Ahn, L. Input-agreement: A new mechanism for collecting data using human computation games. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '09, ACM (2009), 1197--1206.

Digital Library

[31]

Lintott, C. J., et al. Galaxy zoo: morphologies derived from visual inspection of galaxies from the sloan digital sky survey. Monthly Notices of the Royal Astronomical Society 389, 3 (2008), 1179--1189.

[32]

Lintott, C. J., et al. Galaxy zoo:hanny's voorwerp, a quasar light echo?. Monthly Notices of the Royal Astronomical Society 399, 1 (2009), 129--140.

[33]

Lukyanenko, R., Parsons, J., and Wiersma, Y. F. The iq of the crowd: Understanding and improving information quality in structured user-generated content. Information Systems Research 25, 4 (2014), 669--689.

Digital Library

[34]

Mazzolini, M., and Maddison, S. Sage, guide or ghost? the effect of instructor intervention on student participation in online discussion forums. Computers & Education 40, 3 (2003), 237--253.

Digital Library

[35]

Milligan, C., Littlejohn, A., and Margaryan, A. Patterns of engagement in connectivist moocs. MERLOT Journal of Online Learning and Teaching 9, 2 (2013).

[36]

Mugar, G., Osterlund, C., Hassman, K. D., Crowston, K., and Jackson, C. B. Planet hunters and seafloor explorers: Legitimate peripheral participation through practice proxies in online citizen science. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work, CSCW '14, ACM (New York, NY, USA, 2014), 109--119.

Digital Library

[37]

Murphy, E., and Coleman, E. Graduate students' experiences of challenges in online asynchronous discussions. Canadian Journal of Learning and Technology 30, 2 (2004).

[38]

Norman, D. A. The design of everyday things. Basic books, 2002.

Digital Library

[39]

Ockerman, J., and Pritchett, A. A review and reappraisal of task guidance: Aiding workers in procedure following. International Journal of Cognitive Ergonomics 4, 3 (2000), 191--212.

[40]

Oleson, D., Sorokin, A., Laughlin, G. P., Hester, V., Le, J., and Biewald, L. Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. In Human Computation, vol. WS-11--11 of AAAI Workshops, AAAI (2011).

[41]

Raddick, J., Lintott, C., Bamford, S., Land, K., Locksmith, D., Murray, P., Nichol, B., Schawinski, K., Slosar, A., Szalay, A., et al. Galaxy zoo: Motivations of citizen scientists. In Bulletin of the American Astronomical Society, vol. 40 (2008), 240.

[42]

Raddick, M. J., Bracey, G., Gay, P. L., Lintott, C. J., Murray, P., Schawinski, K., Szalay, A. S., and Vandenberg, J. Galaxy zoo: Exploring the motivations of citizen science volunteers. Astronomy Education Review 9 (2010), 010103.

[43]

Rotman, D., Preece, J., Hammock, J., Procita, K., Hansen, D., Parr, C., Lewis, D., and Jacobs, D. Dynamic changes in motivation in collaborative citizen-science projects. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, ACM (2012), 217--226.

Digital Library

[44]

Schroer, J., and Hertel, G. Voluntary engagement in an open web-based encyclopedia: Wikipedians and why they do it. Media Psychology 12, 1 (2009), 96--120.

[45]

Schwamb, M. E., Orosz, J. A., Carter, J. A., Welsh, W. F., Fischer, D. A., Torres, G., Howard, A. W., Crepp, J. R., Keel, W. C., Lintott, C. J., et al. Planet hunters: A transiting circumbinary planet in a quadruple star system. The Astrophysical Journal 768, 2 (2013), 127.

[46]

Silvertown, J. A new dawn for citizen science. Trends in ecology & evolution 24, 9 (2009), 467--471.

[47]

Szell, M., Lambiotte, R., and Thurner, S. Multirelational organization of large-scale social networks in an online world. Proceedings of the National Academy of Sciences 107, 31 (2010), 13636--13641.

[48]

Thaler, S., Simperl, E., Siorpaes, K., and Wlger, S. SpotTheLink: A Game-based Approach to the Alignment of Ontologies. IGI Global, 2012.

[49]

Tinati, R., Luczak-Roesch, M., Simperl, E., and Shadbolt, N. Motivations of citizen scientists: A quantitative investigation of forum participation. In Proceedings of the 2014 ACM Conference on Web Science, WebSci '14, ACM (New York, NY, USA, 2014), 295--296.

Digital Library

[50]

Vasilescu, B., Filkov, V., and Serebrenik, A. Stackoverflow and github: associations between software development and crowdsourced knowledge. In Social Computing (SocialCom), 2013 International Conference on, IEEE (2013), 188--195.

Digital Library

[51]

von Ahn, L., and Dabbish, L. Designing games with a purpose. Communications of the ACM 51, 8 (2008), 58--67.

Digital Library

[52]

Westphal, A. J., Butterworth, A. L., Snead, C. J., Craig, N., Anderson, D., Jones, S. M., Brownlee, D. E., Farnsworth, R., and Zolensky, M. E. Stardust@ home: a massively distributed public search for interstellar dust in the stardust interstellar dust collector. In 36th Annual Lunar and Planetary Science Conference, vol. 36 (2005), 1908.

[53]

Wiggins, A., and Crowston, K. From conservation to crowdsourcing: A typology of citizen science. In System Sciences (HICSS), 2011 44th Hawaii International Conference on, IEEE (2011), 1--10.

Digital Library

[54]

Zichermann, G., and Cunningham, C. Gamification by Design: Implementing Game Mechanics in Web and Mobile Apps, 1st ed. O'Reilly Media, Inc., 2011.

Digital Library

[55]

Zook, M., Graham, M., Shelton, T., and Gorman, S. Volunteered geographic information and crowdsourcing disaster relief: a case study of the haitian earthquake. World Medical & Health Policy 2, 2 (2010), 7--33.

Cited By

He CDeng YHe LGuo QZhang YLu ZLi B(2024)Engage Wider Audience or Facilitate Quality Answers? a Mixed-methods Analysis of Questioning Strategies for Research Sensemaking on a Community Q&A SiteProceedings of the ACM on Human-Computer Interaction10.1145/36373278:CSCW1(1-31)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637327
Zannettou SNemes-Nemeth OAyalon OGoetzen AGummadi KRedmiles ERoesner F(2024)Analyzing User Engagement with TikTok's Short Format Video Recommendations using Data DonationsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642433(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642433
Jeong EJackson CDowthwaite LAhmad TTrouille L(2023)Assessing the Value Orientations of Contributors to Virtual Citizen Science ProjectsProceedings of the 11th International Conference on Communities and Technologies10.1145/3593743.3593782(191-202)Online publication date: 29-May-2023
https://dl.acm.org/doi/10.1145/3593743.3593782
Show More Cited By

Index Terms

Designing for Citizen Data Analysis: A Cross-Sectional Case Study of a Multi-Domain Citizen Science Platform
1. Human-centered computing

Recommendations

Co-designing technologies for citizen scientist engagement & saving species
OzCHI '17: Proceedings of the 29th Australian Conference on Computer-Human Interaction

This full-day workshop will explore how human computer interaction (HCI) design approaches can expand, diversify, and improve ways that members of the public engage with nature and science to help save species as citizen scientists. Prospective ...
User-centered design, activity-centered design, and goal-directed design: a review of three methods for designing web applications
SIGDOC '09: Proceedings of the 27th ACM international conference on Design of communication

When conducting research with users in order to design web applications, the practitioner has a variety of methods from which to choose. This paper examines three such methods'User-Centered Design (UCD), Goal-Directed Design (GDD), and Activity-Centered ...
Analysis of Reviews on Greek Municipalities to Improve Public Service Delivery and Citizen Satisfaction: A Tool for Co-creation and Co-design
PCI '22: Proceedings of the 26th Pan-Hellenic Conference on Informatics

The plethora of digital means of expression available to the general public has provided citizens with the opportunity to share their views and opinions on government services online. Citizen (or customer) satisfaction is determined and influenced by a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

April 2015

4290 pages

ISBN:9781450331456

DOI:10.1145/2702123

General Chairs:
Bo Begole
Huawei, USA
,
Jinwoo Kim
Yonsei University, Korea
,
Program Chairs:
Kori Inkpen
Microsoft Research, USA
,
Woontack Woo
KAIST, Korea

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 April 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

UK Engineering and Physical Sciences Research Council (EPSRC

Conference

CHI '15

Sponsor:

SIGCHI

CHI '15: CHI Conference on Human Factors in Computing Systems

April 18 - 23, 2015

Seoul, Republic of Korea

Acceptance Rates

CHI '15 Paper Acceptance Rate 486 of 2,120 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

59
Total Citations
View Citations
1,095
Total Downloads

Downloads (Last 12 months)86
Downloads (Last 6 weeks)8

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

He CDeng YHe LGuo QZhang YLu ZLi B(2024)Engage Wider Audience or Facilitate Quality Answers? a Mixed-methods Analysis of Questioning Strategies for Research Sensemaking on a Community Q&A SiteProceedings of the ACM on Human-Computer Interaction10.1145/36373278:CSCW1(1-31)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637327
Zannettou SNemes-Nemeth OAyalon OGoetzen AGummadi KRedmiles ERoesner F(2024)Analyzing User Engagement with TikTok's Short Format Video Recommendations using Data DonationsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642433(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642433
Jeong EJackson CDowthwaite LAhmad TTrouille L(2023)Assessing the Value Orientations of Contributors to Virtual Citizen Science ProjectsProceedings of the 11th International Conference on Communities and Technologies10.1145/3593743.3593782(191-202)Online publication date: 29-May-2023
https://dl.acm.org/doi/10.1145/3593743.3593782
Crosby AOrenstein EPoulton SBell KWoodward BRuhl HKatija KForbes A(2023)Designing Ocean Vision AI: An Investigation of Community Needs for Imaging-based Ocean ConservationProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580886(1-16)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580886
Paulino DCorreia AYagui MBarroso JLiberato MVivacqua AGrover ABigham JParedes H(2023)Exploring Stigmergic Collaboration and Task Modularity Through an Expert Crowdsourcing Annotation System: The Case of Storm Phenomena in the Euro-Atlantic RegionIEEE Access10.1109/ACCESS.2023.331959711(106485-106502)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3319597
Vasconcelos LTrevisan DViterbo JSantos CBento Villela MRios da Hora Rodrigues Kde Gois Ribeiro Darin T(2022)Applying the engagement by design methodologyProceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems10.1145/3554364.3559125(1-10)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3554364.3559125
Vella KEsteban MPloderer BBrereton M(2022)Networked Gardens: Remediating Local Nature Data Through the Internet of ThingsProceedings of the 2022 ACM Designing Interactive Systems Conference10.1145/3532106.3533497(1597-1611)Online publication date: 13-Jun-2022
https://dl.acm.org/doi/10.1145/3532106.3533497
Norris WVoida AVoida S(2022)People Talk in Stories. Responders Talk in Data: A Framework for Temporal Sensemaking in Time- and Safety-critical WorkProceedings of the ACM on Human-Computer Interaction10.1145/35129556:CSCW1(1-23)Online publication date: 7-Apr-2022
https://dl.acm.org/doi/10.1145/3512955
Oh CLuo YSt. Jean BChoe E(2022)Patients Waiting for Cues: Information Asymmetries and Challenges in Sharing Patient-Generated Data in the ClinicProceedings of the ACM on Human-Computer Interaction10.1145/35129546:CSCW1(1-23)Online publication date: 7-Apr-2022
https://dl.acm.org/doi/10.1145/3512954
Cooney SRaghavan B(2022)Opening the Gate to Urban Repair: A Tool for Citizen-Led DesignProceedings of the ACM on Human-Computer Interaction10.1145/35129526:CSCW1(1-25)Online publication date: 7-Apr-2022
https://dl.acm.org/doi/10.1145/3512952
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents