skip to main content
research-article

Crowd Coach: Peer Coaching for Crowd Workers' Skill Growth

Authors Info & Claims
Published:01 November 2018Publication History
Skip Abstract Section

Abstract

Traditional employment usually provides mechanisms for workers to improve their skills to access better opportunities. However, crowd work platforms like Amazon Mechanical Turk (AMT) generally do not support skill development (i.e., becoming faster and better at work). While researchers have started to tackle this problem, most solutions are dependent on experts or requesters willing to help. However, requesters generally lack the necessary knowledge, and experts are rare and expensive. To further facilitate crowd workers' skill growth, we present Crowd Coach, a system that enables workers to receive peer coaching while on the job. We conduct a field experiment and real world deployment to study Crowd Coach in the wild. Hundreds of workers used Crowd Coach in a variety of tasks, including writing, doing surveys, and labeling images. We find that Crowd Coach enhances workers' speed without sacrificing their work quality, especially in audio transcription tasks. We posit that peer coaching systems hold potential for better supporting crowd workers' skill development while on the job. We finish with design implications from our research.

Skip Supplemental Material Section

Supplemental Material

suppl.mov

Supplemental video

References

  1. Ali Alkhatib, Michael S Bernstein, and Margaret Levi. 2017. Examining crowd work and gig work through the historical lens of piecework. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 4599--4616. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Walter Ángel, Saiph Savage, and Nataly Moreno. 2015. Participatory Stoves: Designing Renewable Energy Technologies for the Rural Sector. In Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work & Social Computing. ACM, 259--262. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Thomas A Angelo and K Patricia Cross. 1993. Classroom assessment techniques: A handbookfor college teachers. San Francisco: Jossey-Bas (1993).Google ScholarGoogle Scholar
  4. Fabricio Balcazar, Bill L Hopkins, and Yolanda Suarez. 1985. A critical, objective review of performance feedback. Journal of Organizational Behavior Management , Vol. 7, 3--4 (1985), 65--89.Google ScholarGoogle ScholarCross RefCross Ref
  5. Martina Balestra, Coye Cheshire, Ofer Arazy, and Oded Nov. 2017. Investigating the Motivational Paths of Peer Production Newcomers. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 6381--6385. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Janine Berg. 2015. Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers. Comparative Labor Law and Policy Journal , Vol. 37 (2015), 543.Google ScholarGoogle Scholar
  7. Jeffrey P Bigham, Raja Kushalnagar, Ting-Hao Kenneth Huang, Juan Pablo Flores, and Saiph Savage. 2017a. On How Deaf People Might Use Speech to Control Devices. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, 383--384. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Jeffrey P Bigham, Kristin Williams, Nila Banerjee, and John Zimmerman. 2017b. Scopist: Building a Skill Ladder into Crowd Transcription. (2017).Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Stephen Billett. 2001. Learning through work: workplace affordances and individual engagement. Journal of Workplace Learning , Vol. 13, 5 (2001), 209--214.Google ScholarGoogle ScholarCross RefCross Ref
  10. David Boud, Ruth Cohen, and Jane Sampson. 1999. Peer learning and assessment. Assessment & Evaluation in Higher Education , Vol. 24, 4 (1999), 413--426.Google ScholarGoogle Scholar
  11. Daren C Brabham. 2008. Crowdsourcing as a model for problem solving: An introduction and cases. Convergence , Vol. 14, 1 (2008), 75--90.Google ScholarGoogle ScholarCross RefCross Ref
  12. Scott R Braithwaite, Christophe Giraud-Carrier, Josh West, Michael D Barnes, and Carl Lee Hanson. 2016. Validating machine learning algorithms for Twitter data against established measures of suicidality. JMIR Mental Health , Vol. 3, 2 (2016).Google ScholarGoogle ScholarCross RefCross Ref
  13. Robin Brewer, Meredith Ringel Morris, and Anne Marie Piper. 2016. Why would anybody do this?: Understanding older adults' motivations and challenges in crowd work. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2246--2257. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Bill Buxton. 2010. Sketching user experiences: getting the design right and the right design .Morgan Kaufmann. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Carrie J Cai, Shamsi T Iqbal, and Jaime Teevan. 2016. Chain reactions: The impact of order on microtask chains. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 3143--3154. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Chris Callison-Burch. 2014. Crowd-workers: Aggregating information across turkers to help them find higher paying work. In Second AAAI Conference on Human Computation and Crowdsourcing .Google ScholarGoogle Scholar
  17. Patricia A Carlson and Frederick C Berry. 2003. Calibrated peer review/sup TM/and assessing learning outcomes. In Frontiers in Education Conference. IEEE, F3E1--6.Google ScholarGoogle ScholarCross RefCross Ref
  18. Susan Cartwright and Nicola Holmes. 2006. The meaning of work: The challenge of regaining employee engagement and reducing cynicism. Human Resource Management Review , Vol. 16, 2 (2006), 199--208.Google ScholarGoogle ScholarCross RefCross Ref
  19. Joel Chan, Steven Dang, and Steven P Dow. 2016. Comparing different sensemaking approaches for large-scale ideation. In Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, 2717--2728. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Jesse Chandler, Gabriele Paolacci, and Pam Mueller. 2013. Risks and Rewards of Crowdsourcing Marketplaces .Springer New York, New York, NY, 377--392.Google ScholarGoogle Scholar
  21. Chun-Wei Chiang, Eber Betanzos, and Saiph Savage. 2018. Exploring Blockchain for Trustful Collaborations between Immigrants and Governments. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems . ACM, LBW531. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Derrick Coetzee, Seongtaek Lim, Armando Fox, Bjorn Hartmann, and Marti A Hearst. 2015. Structuring interactions for large-scale synchronous peer learning. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, 1139--1152. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Xuefei Nancy Deng and KD Joshi. 2013. Is Crowdsourcing a Source of Worker Empowerment or Exploitation? Understanding Crowd Workers' Perceptions of Crowdsourcing Career. (2013).Google ScholarGoogle Scholar
  24. Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, Panagiotis G Ipeirotis, and Philippe Cudré-Mauroux. 2015. The dynamics of micro-task crowdsourcing: The case of amazon mturk. In Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 238--247. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Martin Dittus and Licia Capra. 2017. Private Peer Feedback As Engagement Driver in Humanitarian Mapping. Proc. ACM Hum.-Comput. Interact. , Vol. 1, CSCW, Article 40 (Dec. 2017), bibinfonumpages18 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Mira Dontcheva, Robert R Morris, Joel R Brandt, and Elizabeth M Gerber. 2014. Combining crowdsourcing and learning to improve engagement and performance. In Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems. ACM, 3379--3388. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Shayan Doroudi, Ece Kamar, Emma Brunskill, and Eric Horvitz. 2016. Toward a learning science for complex crowdsourcing tasks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2623--2634. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Steven P Dow, Kate Heddleston, and Scott R Klemmer. 2009. The efficacy of prototyping under time constraints. In Proceedings of the seventh ACM conference on Creativity and cognition. ACM, 165--174. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. David Durward, Ivo Blohm, and Jan Marco Leimeister. 2016. Crowd work. Business & Information Systems Engineering , Vol. 58, 4 (2016), 281--286.Google ScholarGoogle ScholarCross RefCross Ref
  30. Roberta L Duyff. 1999. The value of lifelong learning: key element in professional career development. Journal of the Academy of Nutrition and Dietetics , Vol. 99, 5 (1999), 538.Google ScholarGoogle Scholar
  31. Darren Edge, Elly Searle, Kevin Chiu, Jing Zhao, and James A Landay. 2011. MicroMandarin: mobile language learning in context. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 3169--3178. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Nancy Falchikov and Judy Goldfinch. 2000. Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of educational research , Vol. 70, 3 (2000), 287--322.Google ScholarGoogle Scholar
  33. Benjamin V Hanrahan, Jutta K Willamowski, Saiganesh Swaminathan, and David B Martin. 2015. TurkBench: Rendering the market for Turkers. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1613--1616. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P Bigham. 2018. A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 449. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Brent Harrison and Mark O Riedl. 2016. Learning from stories: using crowdsourced narratives to train virtual agents. Proceedings of the Artificial Intelligence and Interactive Digital Entertainment (2016), 183--189.Google ScholarGoogle Scholar
  36. John Hattie and Helen Timperley. 2007. The power of feedback. Review of Educational Research , Vol. 77, 1 (2007), 81--112.Google ScholarGoogle ScholarCross RefCross Ref
  37. Yurong He, Jennifer Preece, Carol Boston, Anne Bowser, Derek Hansen, and Jen Hammock. 2014. The Effects of Individualized Feedback on College Students' Contributions to Citizen Science. In Proceedings of the Companion Publication of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW Companion '14). ACM, New York, NY, USA, 165--168. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Pamela J Hinds, Michael Patterson, and Jeffrey Pfeffer. 2001. Bothered by abstraction: The effect of expertise on knowledge transfer and subsequent novice performance. Journal of applied psychology , Vol. 86, 6 (2001), 1232.Google ScholarGoogle ScholarCross RefCross Ref
  39. John B. Horrigan. 2016. Civic engagement in the digital age. Pew Internet & American Life Project (2016).Google ScholarGoogle Scholar
  40. Philip N Howard, Saiph Savage, Claudia Flores Saviaga, Carlos Toxtli, and Andrés Monroy-Hemández. 2016. Social media, civic engagement, and the slacktivism hypothesis: Lessons from Mexico's “El Bronco”. Journal of International Affairs , Vol. 70, 1 (2016).Google ScholarGoogle Scholar
  41. Ting-Hao Kenneth Huang, Walter S Lasecki, Amos Azaria, and Jeffrey P Bigham. 2016. " Is There Anything Else I Can Help You With?" Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent. In Fourth AAAI Conference on Human Computation and Crowdsourcing .Google ScholarGoogle Scholar
  42. Kazushi Ikeda and Michael S Bernstein. 2016. Pay It Backward: Per-Task Payments on Crowdsourcing Platforms Reduce Productivity. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4111--4121. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Lilly C Irani and M Silberman. 2013. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 611--620. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Lilly C Irani and M Silberman. 2016. Stories we tell about labor: Turkopticon and the trouble with design. In Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, 4573--4586. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Nila Banerjee Jeffrey P. Bigham, Kristin Williams and John Zimmerman. 2017. Scopist: Building a Skill Ladder into Crowd Work. In Proceedings of the Web for All Conference (W4A '17). ACM, New York, NY, USA, 10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Nicolas Kaufmann, Thimo Schulze, and Daniel Veit. 2011. More than fun and money. Worker Motivation in Crowdsourcing-A Study on Mechanical Turk.. In AMCIS, Vol. 11. 1--11.Google ScholarGoogle Scholar
  47. Clare Kelliher and Deirdre Anderson. 2008. For better or for worse? An analysis of how flexible working practices influence employees' perceptions of job quality. The International Journal of Human Resource Management, , Vol. 19, 3 (2008), 419--431.Google ScholarGoogle Scholar
  48. Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 1301--1318. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Avraham N Kluger and Angelo DeNisi. 1996. The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological bulletin , Vol. 119, 2 (1996), 254.Google ScholarGoogle Scholar
  50. Kenneth R Koedinger, Albert T Corbett, and Charles Perfetti. 2012. The Knowledge-Learning-Instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive science , Vol. 36, 5 (2012), 757--798.Google ScholarGoogle Scholar
  51. Anand Kulkarni, Matthew Can, and Björn Hartmann. 2012. Collaboratively Crowdsourcing Workflows with Turkomatic. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW '12). ACM, New York, NY, USA, 1003--1012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Chinmay Kulkarni, Koh Pang Wei, Huy Le, Daniel Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, and Scott R Klemmer. 2013. Peer and self assessment in massive online classes. ACM Transactions on Computer-Human Interaction (TOCHI) , Vol. 20, 6 (2013), 33. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Chinmay E Kulkarni, Michael S Bernstein, and Scott R Klemmer. 2015. PeerStudio: rapid peer feedback emphasizes revision and improves performance. In Proceedings of the Second (2015) ACM Conference on Learning@ Scale. ACM, 75--84. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Airi Lampinen, Victoria Bellotti, Coye Cheshire, and Mary Gray. 2016. CSCW and theSharing Economy: The Future of Platforms as Sites of Work Collaboration and Trust. In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion. ACM, 491--497. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Gary P Latham and Edwin A Locke. 1991. Self-regulation through goal setting. Organizational behavior and human decision processes , Vol. 50, 2 (1991), 212--247.Google ScholarGoogle Scholar
  56. Carol A Lundberg. 2003. The influence of time-limitations, faculty, and peer relationships on adult student learning: A causal model. The Journal of Higher Education , Vol. 74, 6 (2003), 665--688.Google ScholarGoogle ScholarCross RefCross Ref
  57. David Martin, Benjamin V Hanrahan, Jacki O'Neill, and Neha Gupta. 2014. Being a turker. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 224--235. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Winter Mason and Duncan J Watts. 2010. Financial incentives and the performance of crowds. ACM SigKDD Explorations Newsletter , Vol. 11, 2 (2010), 100--108. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Alexander H Miller, Will Feng, Adam Fisch, Jiasen Lu, Dhruv Batra, Antoine Bordes, Devi Parikh, and Jason Weston. 2017. Parlai: A dialog research software platform. arXiv preprint arXiv:1705.06476 (2017).Google ScholarGoogle Scholar
  60. Meredith Ringel Morris, Jeffrey P Bigham, Robin Brewer, Jonathan Bragg, Anand Kulkarni, Jessie Li, and Saiph Savage. 2017. Subcontracting microwork. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 1867--1876. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Melissa M Nelson and Christian D Schunn. 2009. The nature of feedback: How different types of peer feedback affect writing performance. Instructional Science , Vol. 37, 4 (2009), 375--401.Google ScholarGoogle ScholarCross RefCross Ref
  62. Jeffrey Nichols and Jeon-Hyung Kang. 2012. Asking questions of targeted strangers on social networks. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, 999--1002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. Jeffrey Nichols, Michelle Zhou, Huahai Yang, Jeon-Hyung Kang, and Xiao Hua Sun. 2013. Analyzing the quality of information solicited from targeted strangers on social media. In Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 967--976. Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. Jon Noronha, Eric Hysen, Haoqi Zhang, and Krzysztof Z Gajos. 2011. Platemate: crowdsourcing nutritional analysis from food photographs. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology. ACM, 1--12. Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. Scott Novotney and Chris Callison-Burch. 2010. Cheap, fast and good enough: Automatic speech recognition with non-expert transcription. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 207--215. Google ScholarGoogle ScholarDigital LibraryDigital Library
  66. Gabriele Paolacci, Jesse Chandler, and Panagiotis G Ipeirotis. 2010. Running experiments on Amazon Mechanical Turk. (2010).Google ScholarGoogle Scholar
  67. Melissa M Patchan and Christian D Schunn. 2015. Understanding the benefits of providing peer feedback: how students respond to peers' texts of varying quality. Instructional Science , Vol. 43, 5 (2015), 591--614.Google ScholarGoogle ScholarCross RefCross Ref
  68. Sunil Ramlall. 2004. A review of employee motivation theories and their implications for employee retention within organizations. Journal of American Academy of Business , Vol. 5, 1/2 (2004), 52--63.Google ScholarGoogle Scholar
  69. Alex Rosenblat and Luke Stark. 2016. Algorithmic labor and information asymmetries: A case study of Uber's drivers. (2016).Google ScholarGoogle Scholar
  70. Joel Ross, Andrew Zaldivar, Lilly Irani, and Bill Tomlinson. 2009. Who are the turkers? Worker demographics in amazon mechanical turk. Department of Informatics, University of California, Irvine, USA, Tech. Rep (2009).Google ScholarGoogle Scholar
  71. Richard M Ryan and Edward L Deci. 2000. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American psychologist , Vol. 55, 1 (2000), 68.Google ScholarGoogle Scholar
  72. Saiph Savage, Andres Monroy-Hernandez, and Tobias Höllerer. 2016. Botivist: Calling volunteers to action using online bots. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM, 813--822. Google ScholarGoogle ScholarDigital LibraryDigital Library
  73. Jodi Schneider, Bluma S. Gelley, and Aaron Halfaker. 2014. Accept, Decline, Postpone: How Newcomer Productivity is Reduced in English Wikipedia by Pre-publication Review. In Proceedings of The International Symposium on Open Collaboration (OpenSym '14). ACM, New York, NY, USA, Article 26, bibinfonumpages10 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  74. M Silberman, Lilly Irani, and Joel Ross. 2010a. Ethics and tactics of professional crowdwork. XRDS: Crossroads, The ACM Magazine for Students , Vol. 17, 2 (2010), 39--43. Google ScholarGoogle ScholarDigital LibraryDigital Library
  75. M. Six Silberman, Joel Ross, Lilly Irani, and Bill Tomlinson. 2010b. Sellers' Problems in Human Computation Markets. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP '10). ACM, New York, NY, USA, 18--21. Google ScholarGoogle ScholarDigital LibraryDigital Library
  76. Ryo Suzuki, Niloufar Salehi, Michelle S Lam, Juan C Marroquin, and Michael S Bernstein. 2016. Atelier: Repurposing expert crowdsourcing tasks as micro-internships. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2645--2656. Google ScholarGoogle ScholarDigital LibraryDigital Library
  77. William Thies, Aishwarya Ratan, and James Davis. 2011. Paid crowdsourcing as a vehicle for global development. In CHI Workshop on Crowdsourcing and Human Computation .Google ScholarGoogle Scholar
  78. David Tinapple, Loren Olson, and John Sadauskas. 2013. CritViz: Web-based software supporting peer critique in large creative classrooms. Bulletin of the IEEE Technical Committee on Learning Technology , Vol. 15, 1 (2013), 29.Google ScholarGoogle Scholar
  79. Bonnie Urciuoli. 2008. Skills and selves in the new workplace. American Ethnologist , Vol. 35, 2 (2008), 211--228.Google ScholarGoogle ScholarCross RefCross Ref
  80. Rajan Vaish, Keith Wyngarden, Jingshu Chen, Brandon Cheung, and Michael S Bernstein. 2014. Twitch crowdsourcing: crowd contributions in short bursts of time. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 3645--3654. Google ScholarGoogle ScholarDigital LibraryDigital Library
  81. Niels van Doorn. 2017. Platform labor: on the gendered and racialized exploitation of low-income service work in the `on-demand' economy. Information, Communication & Society , Vol. 20, 6 (2017), 898--914.Google ScholarGoogle ScholarCross RefCross Ref
  82. Sarah Weir, Juho Kim, Krzysztof Z. Gajos, and Robert C. Miller. 2015. Learnersourcing Subgoal Labels for How-to Videos. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work (CSCW '15). ACM, New York, NY, USA, 405--416. Google ScholarGoogle ScholarDigital LibraryDigital Library
  83. Mark E Whiting, Dilrukshi Gamage, Snehalkumar S Gaikwad, Aaron Gilbee, Shirish Goyal, Alipta Ballav, Dinesh Majeti, Nalin Chhibber, Angela Richmond-Fuller, Freddie Vargus, et almbox. 2016. Crowd guilds: Worker-led reputation and feedback on crowdsourcing platforms. arXiv preprint arXiv:1611.01572 (2016). Google ScholarGoogle ScholarDigital LibraryDigital Library
  84. Joseph Jay Williams, Juho Kim, Anna Rafferty, Samuel Maldonado, Krzysztof Z Gajos, Walter S Lasecki, and Neil Heffernan. 2016. Axis: Generating explanations at scale with learnersourcing and machine learning. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale. ACM, 379--388. Google ScholarGoogle ScholarDigital LibraryDigital Library
  85. Bill Williamson. 1998. Lifeworlds and Learning. Essays in the Theory, Philosophy and Practice of Lifelong Learning. (1998).Google ScholarGoogle Scholar
  86. Yu-Chun Grace Yen, Steven P Dow, Elizabeth Gerber, and Brian P Bailey. 2016. Social Network, Web Forum, or Task Market?: Comparing Different Crowd Genres for Design Feedback Exchange. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. ACM, 773--784. Google ScholarGoogle ScholarDigital LibraryDigital Library
  87. Dongwook Yoon, Nicholas Chen, Bernie Randles, Amy Cheatle, Corinna E. Löckenhoff, Steven J. Jackson, Abigail Sellen, and Franccois Guimbretière. 2016. RichReviewGoogle ScholarGoogle Scholar
  88. : Deployment of a Collaborative Multi-modal Annotation System for Instructor Feedback and Peer Discussion. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16). ACM, New York, NY, USA, 195--205. Google ScholarGoogle ScholarDigital LibraryDigital Library
  89. Haiyi Zhu, Steven P Dow, Robert E Kraut, and Aniket Kittur. 2014. Reviewing versus doing: Learning and performance in crowd assessment. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 1445--1455. Google ScholarGoogle ScholarDigital LibraryDigital Library
  90. Haiyi Zhu, Amy Zhang, Jiping He, Robert E. Kraut, and Aniket Kittur. 2013. Effects of Peer Feedback on Contribution: A Field Experiment in Wikipedia. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 2253--2262. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Crowd Coach: Peer Coaching for Crowd Workers' Skill Growth

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!