Abstract

Traditional employment usually provides mechanisms for workers to improve their skills to access better opportunities. However, crowd work platforms like Amazon Mechanical Turk (AMT) generally do not support skill development (i.e., becoming faster and better at work). While researchers have started to tackle this problem, most solutions are dependent on experts or requesters willing to help. However, requesters generally lack the necessary knowledge, and experts are rare and expensive. To further facilitate crowd workers' skill growth, we present Crowd Coach, a system that enables workers to receive peer coaching while on the job. We conduct a field experiment and real world deployment to study Crowd Coach in the wild. Hundreds of workers used Crowd Coach in a variety of tasks, including writing, doing surveys, and labeling images. We find that Crowd Coach enhances workers' speed without sacrificing their work quality, especially in audio transcription tasks. We posit that peer coaching systems hold potential for better supporting crowd workers' skill development while on the job. We finish with design implications from our research.
Supplemental Material
- Ali Alkhatib, Michael S Bernstein, and Margaret Levi. 2017. Examining crowd work and gig work through the historical lens of piecework. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 4599--4616. Google Scholar
Digital Library
- Walter Ángel, Saiph Savage, and Nataly Moreno. 2015. Participatory Stoves: Designing Renewable Energy Technologies for the Rural Sector. In Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work & Social Computing. ACM, 259--262. Google Scholar
Digital Library
- Thomas A Angelo and K Patricia Cross. 1993. Classroom assessment techniques: A handbookfor college teachers. San Francisco: Jossey-Bas (1993).Google Scholar
- Fabricio Balcazar, Bill L Hopkins, and Yolanda Suarez. 1985. A critical, objective review of performance feedback. Journal of Organizational Behavior Management , Vol. 7, 3--4 (1985), 65--89.Google Scholar
Cross Ref
- Martina Balestra, Coye Cheshire, Ofer Arazy, and Oded Nov. 2017. Investigating the Motivational Paths of Peer Production Newcomers. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 6381--6385. Google Scholar
Digital Library
- Janine Berg. 2015. Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers. Comparative Labor Law and Policy Journal , Vol. 37 (2015), 543.Google Scholar
- Jeffrey P Bigham, Raja Kushalnagar, Ting-Hao Kenneth Huang, Juan Pablo Flores, and Saiph Savage. 2017a. On How Deaf People Might Use Speech to Control Devices. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, 383--384. Google Scholar
Digital Library
- Jeffrey P Bigham, Kristin Williams, Nila Banerjee, and John Zimmerman. 2017b. Scopist: Building a Skill Ladder into Crowd Transcription. (2017).Google Scholar
Digital Library
- Stephen Billett. 2001. Learning through work: workplace affordances and individual engagement. Journal of Workplace Learning , Vol. 13, 5 (2001), 209--214.Google Scholar
Cross Ref
- David Boud, Ruth Cohen, and Jane Sampson. 1999. Peer learning and assessment. Assessment & Evaluation in Higher Education , Vol. 24, 4 (1999), 413--426.Google Scholar
- Daren C Brabham. 2008. Crowdsourcing as a model for problem solving: An introduction and cases. Convergence , Vol. 14, 1 (2008), 75--90.Google Scholar
Cross Ref
- Scott R Braithwaite, Christophe Giraud-Carrier, Josh West, Michael D Barnes, and Carl Lee Hanson. 2016. Validating machine learning algorithms for Twitter data against established measures of suicidality. JMIR Mental Health , Vol. 3, 2 (2016).Google Scholar
Cross Ref
- Robin Brewer, Meredith Ringel Morris, and Anne Marie Piper. 2016. Why would anybody do this?: Understanding older adults' motivations and challenges in crowd work. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2246--2257. Google Scholar
Digital Library
- Bill Buxton. 2010. Sketching user experiences: getting the design right and the right design .Morgan Kaufmann. Google Scholar
Digital Library
- Carrie J Cai, Shamsi T Iqbal, and Jaime Teevan. 2016. Chain reactions: The impact of order on microtask chains. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 3143--3154. Google Scholar
Digital Library
- Chris Callison-Burch. 2014. Crowd-workers: Aggregating information across turkers to help them find higher paying work. In Second AAAI Conference on Human Computation and Crowdsourcing .Google Scholar
- Patricia A Carlson and Frederick C Berry. 2003. Calibrated peer review/sup TM/and assessing learning outcomes. In Frontiers in Education Conference. IEEE, F3E1--6.Google Scholar
Cross Ref
- Susan Cartwright and Nicola Holmes. 2006. The meaning of work: The challenge of regaining employee engagement and reducing cynicism. Human Resource Management Review , Vol. 16, 2 (2006), 199--208.Google Scholar
Cross Ref
- Joel Chan, Steven Dang, and Steven P Dow. 2016. Comparing different sensemaking approaches for large-scale ideation. In Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, 2717--2728. Google Scholar
Digital Library
- Jesse Chandler, Gabriele Paolacci, and Pam Mueller. 2013. Risks and Rewards of Crowdsourcing Marketplaces .Springer New York, New York, NY, 377--392.Google Scholar
- Chun-Wei Chiang, Eber Betanzos, and Saiph Savage. 2018. Exploring Blockchain for Trustful Collaborations between Immigrants and Governments. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems . ACM, LBW531. Google Scholar
Digital Library
- Derrick Coetzee, Seongtaek Lim, Armando Fox, Bjorn Hartmann, and Marti A Hearst. 2015. Structuring interactions for large-scale synchronous peer learning. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, 1139--1152. Google Scholar
Digital Library
- Xuefei Nancy Deng and KD Joshi. 2013. Is Crowdsourcing a Source of Worker Empowerment or Exploitation? Understanding Crowd Workers' Perceptions of Crowdsourcing Career. (2013).Google Scholar
- Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, Panagiotis G Ipeirotis, and Philippe Cudré-Mauroux. 2015. The dynamics of micro-task crowdsourcing: The case of amazon mturk. In Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 238--247. Google Scholar
Digital Library
- Martin Dittus and Licia Capra. 2017. Private Peer Feedback As Engagement Driver in Humanitarian Mapping. Proc. ACM Hum.-Comput. Interact. , Vol. 1, CSCW, Article 40 (Dec. 2017), bibinfonumpages18 pages. Google Scholar
Digital Library
- Mira Dontcheva, Robert R Morris, Joel R Brandt, and Elizabeth M Gerber. 2014. Combining crowdsourcing and learning to improve engagement and performance. In Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems. ACM, 3379--3388. Google Scholar
Digital Library
- Shayan Doroudi, Ece Kamar, Emma Brunskill, and Eric Horvitz. 2016. Toward a learning science for complex crowdsourcing tasks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2623--2634. Google Scholar
Digital Library
- Steven P Dow, Kate Heddleston, and Scott R Klemmer. 2009. The efficacy of prototyping under time constraints. In Proceedings of the seventh ACM conference on Creativity and cognition. ACM, 165--174. Google Scholar
Digital Library
- David Durward, Ivo Blohm, and Jan Marco Leimeister. 2016. Crowd work. Business & Information Systems Engineering , Vol. 58, 4 (2016), 281--286.Google Scholar
Cross Ref
- Roberta L Duyff. 1999. The value of lifelong learning: key element in professional career development. Journal of the Academy of Nutrition and Dietetics , Vol. 99, 5 (1999), 538.Google Scholar
- Darren Edge, Elly Searle, Kevin Chiu, Jing Zhao, and James A Landay. 2011. MicroMandarin: mobile language learning in context. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 3169--3178. Google Scholar
Digital Library
- Nancy Falchikov and Judy Goldfinch. 2000. Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of educational research , Vol. 70, 3 (2000), 287--322.Google Scholar
- Benjamin V Hanrahan, Jutta K Willamowski, Saiganesh Swaminathan, and David B Martin. 2015. TurkBench: Rendering the market for Turkers. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1613--1616. Google Scholar
Digital Library
- Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P Bigham. 2018. A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 449. Google Scholar
Digital Library
- Brent Harrison and Mark O Riedl. 2016. Learning from stories: using crowdsourced narratives to train virtual agents. Proceedings of the Artificial Intelligence and Interactive Digital Entertainment (2016), 183--189.Google Scholar
- John Hattie and Helen Timperley. 2007. The power of feedback. Review of Educational Research , Vol. 77, 1 (2007), 81--112.Google Scholar
Cross Ref
- Yurong He, Jennifer Preece, Carol Boston, Anne Bowser, Derek Hansen, and Jen Hammock. 2014. The Effects of Individualized Feedback on College Students' Contributions to Citizen Science. In Proceedings of the Companion Publication of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW Companion '14). ACM, New York, NY, USA, 165--168. Google Scholar
Digital Library
- Pamela J Hinds, Michael Patterson, and Jeffrey Pfeffer. 2001. Bothered by abstraction: The effect of expertise on knowledge transfer and subsequent novice performance. Journal of applied psychology , Vol. 86, 6 (2001), 1232.Google Scholar
Cross Ref
- John B. Horrigan. 2016. Civic engagement in the digital age. Pew Internet & American Life Project (2016).Google Scholar
- Philip N Howard, Saiph Savage, Claudia Flores Saviaga, Carlos Toxtli, and Andrés Monroy-Hemández. 2016. Social media, civic engagement, and the slacktivism hypothesis: Lessons from Mexico's “El Bronco”. Journal of International Affairs , Vol. 70, 1 (2016).Google Scholar
- Ting-Hao Kenneth Huang, Walter S Lasecki, Amos Azaria, and Jeffrey P Bigham. 2016. " Is There Anything Else I Can Help You With?" Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent. In Fourth AAAI Conference on Human Computation and Crowdsourcing .Google Scholar
- Kazushi Ikeda and Michael S Bernstein. 2016. Pay It Backward: Per-Task Payments on Crowdsourcing Platforms Reduce Productivity. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4111--4121. Google Scholar
Digital Library
- Lilly C Irani and M Silberman. 2013. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 611--620. Google Scholar
Digital Library
- Lilly C Irani and M Silberman. 2016. Stories we tell about labor: Turkopticon and the trouble with design. In Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, 4573--4586. Google Scholar
Digital Library
- Nila Banerjee Jeffrey P. Bigham, Kristin Williams and John Zimmerman. 2017. Scopist: Building a Skill Ladder into Crowd Work. In Proceedings of the Web for All Conference (W4A '17). ACM, New York, NY, USA, 10. Google Scholar
Digital Library
- Nicolas Kaufmann, Thimo Schulze, and Daniel Veit. 2011. More than fun and money. Worker Motivation in Crowdsourcing-A Study on Mechanical Turk.. In AMCIS, Vol. 11. 1--11.Google Scholar
- Clare Kelliher and Deirdre Anderson. 2008. For better or for worse? An analysis of how flexible working practices influence employees' perceptions of job quality. The International Journal of Human Resource Management, , Vol. 19, 3 (2008), 419--431.Google Scholar
- Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 1301--1318. Google Scholar
Digital Library
- Avraham N Kluger and Angelo DeNisi. 1996. The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological bulletin , Vol. 119, 2 (1996), 254.Google Scholar
- Kenneth R Koedinger, Albert T Corbett, and Charles Perfetti. 2012. The Knowledge-Learning-Instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive science , Vol. 36, 5 (2012), 757--798.Google Scholar
- Anand Kulkarni, Matthew Can, and Björn Hartmann. 2012. Collaboratively Crowdsourcing Workflows with Turkomatic. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW '12). ACM, New York, NY, USA, 1003--1012. Google Scholar
Digital Library
- Chinmay Kulkarni, Koh Pang Wei, Huy Le, Daniel Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, and Scott R Klemmer. 2013. Peer and self assessment in massive online classes. ACM Transactions on Computer-Human Interaction (TOCHI) , Vol. 20, 6 (2013), 33. Google Scholar
Digital Library
- Chinmay E Kulkarni, Michael S Bernstein, and Scott R Klemmer. 2015. PeerStudio: rapid peer feedback emphasizes revision and improves performance. In Proceedings of the Second (2015) ACM Conference on Learning@ Scale. ACM, 75--84. Google Scholar
Digital Library
- Airi Lampinen, Victoria Bellotti, Coye Cheshire, and Mary Gray. 2016. CSCW and theSharing Economy: The Future of Platforms as Sites of Work Collaboration and Trust. In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion. ACM, 491--497. Google Scholar
Digital Library
- Gary P Latham and Edwin A Locke. 1991. Self-regulation through goal setting. Organizational behavior and human decision processes , Vol. 50, 2 (1991), 212--247.Google Scholar
- Carol A Lundberg. 2003. The influence of time-limitations, faculty, and peer relationships on adult student learning: A causal model. The Journal of Higher Education , Vol. 74, 6 (2003), 665--688.Google Scholar
Cross Ref
- David Martin, Benjamin V Hanrahan, Jacki O'Neill, and Neha Gupta. 2014. Being a turker. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 224--235. Google Scholar
Digital Library
- Winter Mason and Duncan J Watts. 2010. Financial incentives and the performance of crowds. ACM SigKDD Explorations Newsletter , Vol. 11, 2 (2010), 100--108. Google Scholar
Digital Library
- Alexander H Miller, Will Feng, Adam Fisch, Jiasen Lu, Dhruv Batra, Antoine Bordes, Devi Parikh, and Jason Weston. 2017. Parlai: A dialog research software platform. arXiv preprint arXiv:1705.06476 (2017).Google Scholar
- Meredith Ringel Morris, Jeffrey P Bigham, Robin Brewer, Jonathan Bragg, Anand Kulkarni, Jessie Li, and Saiph Savage. 2017. Subcontracting microwork. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 1867--1876. Google Scholar
Digital Library
- Melissa M Nelson and Christian D Schunn. 2009. The nature of feedback: How different types of peer feedback affect writing performance. Instructional Science , Vol. 37, 4 (2009), 375--401.Google Scholar
Cross Ref
- Jeffrey Nichols and Jeon-Hyung Kang. 2012. Asking questions of targeted strangers on social networks. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, 999--1002. Google Scholar
Digital Library
- Jeffrey Nichols, Michelle Zhou, Huahai Yang, Jeon-Hyung Kang, and Xiao Hua Sun. 2013. Analyzing the quality of information solicited from targeted strangers on social media. In Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 967--976. Google Scholar
Digital Library
- Jon Noronha, Eric Hysen, Haoqi Zhang, and Krzysztof Z Gajos. 2011. Platemate: crowdsourcing nutritional analysis from food photographs. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology. ACM, 1--12. Google Scholar
Digital Library
- Scott Novotney and Chris Callison-Burch. 2010. Cheap, fast and good enough: Automatic speech recognition with non-expert transcription. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 207--215. Google Scholar
Digital Library
- Gabriele Paolacci, Jesse Chandler, and Panagiotis G Ipeirotis. 2010. Running experiments on Amazon Mechanical Turk. (2010).Google Scholar
- Melissa M Patchan and Christian D Schunn. 2015. Understanding the benefits of providing peer feedback: how students respond to peers' texts of varying quality. Instructional Science , Vol. 43, 5 (2015), 591--614.Google Scholar
Cross Ref
- Sunil Ramlall. 2004. A review of employee motivation theories and their implications for employee retention within organizations. Journal of American Academy of Business , Vol. 5, 1/2 (2004), 52--63.Google Scholar
- Alex Rosenblat and Luke Stark. 2016. Algorithmic labor and information asymmetries: A case study of Uber's drivers. (2016).Google Scholar
- Joel Ross, Andrew Zaldivar, Lilly Irani, and Bill Tomlinson. 2009. Who are the turkers? Worker demographics in amazon mechanical turk. Department of Informatics, University of California, Irvine, USA, Tech. Rep (2009).Google Scholar
- Richard M Ryan and Edward L Deci. 2000. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American psychologist , Vol. 55, 1 (2000), 68.Google Scholar
- Saiph Savage, Andres Monroy-Hernandez, and Tobias Höllerer. 2016. Botivist: Calling volunteers to action using online bots. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM, 813--822. Google Scholar
Digital Library
- Jodi Schneider, Bluma S. Gelley, and Aaron Halfaker. 2014. Accept, Decline, Postpone: How Newcomer Productivity is Reduced in English Wikipedia by Pre-publication Review. In Proceedings of The International Symposium on Open Collaboration (OpenSym '14). ACM, New York, NY, USA, Article 26, bibinfonumpages10 pages. Google Scholar
Digital Library
- M Silberman, Lilly Irani, and Joel Ross. 2010a. Ethics and tactics of professional crowdwork. XRDS: Crossroads, The ACM Magazine for Students , Vol. 17, 2 (2010), 39--43. Google Scholar
Digital Library
- M. Six Silberman, Joel Ross, Lilly Irani, and Bill Tomlinson. 2010b. Sellers' Problems in Human Computation Markets. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP '10). ACM, New York, NY, USA, 18--21. Google Scholar
Digital Library
- Ryo Suzuki, Niloufar Salehi, Michelle S Lam, Juan C Marroquin, and Michael S Bernstein. 2016. Atelier: Repurposing expert crowdsourcing tasks as micro-internships. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2645--2656. Google Scholar
Digital Library
- William Thies, Aishwarya Ratan, and James Davis. 2011. Paid crowdsourcing as a vehicle for global development. In CHI Workshop on Crowdsourcing and Human Computation .Google Scholar
- David Tinapple, Loren Olson, and John Sadauskas. 2013. CritViz: Web-based software supporting peer critique in large creative classrooms. Bulletin of the IEEE Technical Committee on Learning Technology , Vol. 15, 1 (2013), 29.Google Scholar
- Bonnie Urciuoli. 2008. Skills and selves in the new workplace. American Ethnologist , Vol. 35, 2 (2008), 211--228.Google Scholar
Cross Ref
- Rajan Vaish, Keith Wyngarden, Jingshu Chen, Brandon Cheung, and Michael S Bernstein. 2014. Twitch crowdsourcing: crowd contributions in short bursts of time. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 3645--3654. Google Scholar
Digital Library
- Niels van Doorn. 2017. Platform labor: on the gendered and racialized exploitation of low-income service work in the `on-demand' economy. Information, Communication & Society , Vol. 20, 6 (2017), 898--914.Google Scholar
Cross Ref
- Sarah Weir, Juho Kim, Krzysztof Z. Gajos, and Robert C. Miller. 2015. Learnersourcing Subgoal Labels for How-to Videos. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work (CSCW '15). ACM, New York, NY, USA, 405--416. Google Scholar
Digital Library
- Mark E Whiting, Dilrukshi Gamage, Snehalkumar S Gaikwad, Aaron Gilbee, Shirish Goyal, Alipta Ballav, Dinesh Majeti, Nalin Chhibber, Angela Richmond-Fuller, Freddie Vargus, et almbox. 2016. Crowd guilds: Worker-led reputation and feedback on crowdsourcing platforms. arXiv preprint arXiv:1611.01572 (2016). Google Scholar
Digital Library
- Joseph Jay Williams, Juho Kim, Anna Rafferty, Samuel Maldonado, Krzysztof Z Gajos, Walter S Lasecki, and Neil Heffernan. 2016. Axis: Generating explanations at scale with learnersourcing and machine learning. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale. ACM, 379--388. Google Scholar
Digital Library
- Bill Williamson. 1998. Lifeworlds and Learning. Essays in the Theory, Philosophy and Practice of Lifelong Learning. (1998).Google Scholar
- Yu-Chun Grace Yen, Steven P Dow, Elizabeth Gerber, and Brian P Bailey. 2016. Social Network, Web Forum, or Task Market?: Comparing Different Crowd Genres for Design Feedback Exchange. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. ACM, 773--784. Google Scholar
Digital Library
- Dongwook Yoon, Nicholas Chen, Bernie Randles, Amy Cheatle, Corinna E. Löckenhoff, Steven J. Jackson, Abigail Sellen, and Franccois Guimbretière. 2016. RichReviewGoogle Scholar
- : Deployment of a Collaborative Multi-modal Annotation System for Instructor Feedback and Peer Discussion. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16). ACM, New York, NY, USA, 195--205. Google Scholar
Digital Library
- Haiyi Zhu, Steven P Dow, Robert E Kraut, and Aniket Kittur. 2014. Reviewing versus doing: Learning and performance in crowd assessment. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 1445--1455. Google Scholar
Digital Library
- Haiyi Zhu, Amy Zhang, Jiping He, Robert E. Kraut, and Aniket Kittur. 2013. Effects of Peer Feedback on Contribution: A Field Experiment in Wikipedia. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 2253--2262. Google Scholar
Digital Library
Index Terms
Crowd Coach: Peer Coaching for Crowd Workers' Skill Growth
Recommendations
A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing SystemsA growing number of people are working as part of on-line crowd work. Crowd work is often thought to be low wage work. However, we know little about the wage distribution in practice and what causes low/high earnings in this setting. We recorded 2,676 ...
Efficient Crowd Exploration of Large Networks: The Case of Causal Attribution
Accurately and efficiently crowdsourcing complex, open-ended tasks can be difficult, as crowd participants tend to favor short, repetitive "microtasks". We study the crowdsourcing of large networks where the crowd provides the network topology via ...
How many crowdsourced workers should a requester hire?
Recent years have seen an increased interest in crowdsourcing as a way of obtaining information from a potentially large group of workers at a reduced cost. The crowdsourcing process, as we consider in this paper, is as follows: a requester hires a ...






Comments