Abstract
Crowdsourcing has emerged as an effective method of scaling-up tasks previously reserved for a small set of experts. Accordingly, researchers in the large-scale online learning space have begun to employ crowdworkers to conduct research about large-scale, open online learning. We here report results from a crowdsourcing study (N=135) to evaluate the extent to which crowdworkers and MOOC learners behave comparably on lecture viewing and quiz tasks---the most utilized learning activities in MOOCs. This serves to (i) validate the assumption of previous research that crowdworkers are indeed reliable proxies of online learners and (ii) address the potential of employing crowdworkers as a means of online learning environment testing. Overall, we observe mixed results---in certain contexts (quiz performance and video watching behavior) crowdworkers appear to behave comparably to MOOC learners, and in other situations (interactions with in-video quizzes), their behaviors appear to be disparate. We conclude that future research should be cautious if employing crowdworkers to carry out learning tasks, as the two populations do not behave comparably on all learning-related activities.
- Omar Alonso and Stefano Mizzaro. 2012. Using crowdsourcing for TREC relevance assessment. Information processing & management, Vol. 48, 6 (2012), 1053--1066. Google Scholar
Digital Library
- Omar Alonso, Daniel E Rose, and Benjamin Stewart. 2008. Crowdsourcing for relevance evaluation. In ACM SigIR Forum, Vol. 42. ACM, 9--15. Google Scholar
Digital Library
- Yigal Attali. 2015. Effects of multiple-try feedback and question type during mathematics problem solving on performance in similar problems. Computers & Education, Vol. 86 (2015), 260--267. Google Scholar
Digital Library
- Yigal Attali and Meirav Arieli-Attali. 2015. Gamification in assessment: Do points affect test performance? Computers & Education, Vol. 83 (2015), 57--63.Google Scholar
Cross Ref
- John Bohannon. 2011. Social science for pennies. (2011).Google Scholar
- Anne Cocos, Ting Qian, Chris Callison-Burch, and Aaron J Masino. 2017. Crowd control: Effectively utilizing unscreened crowd workers for biomedical data annotation. Journal of biomedical informatics, Vol. 69 (2017), 86--92. Google Scholar
Digital Library
- Derrick Coetzee, Seongtaek Lim, Armando Fox, Bjorn Hartmann, and Marti A. Hearst. 2015. Structuring interactions for large-scale synchronous peer learning. In CSCW '15. 1139--1152. Google Scholar
Digital Library
- Gabriel Culbertson, Solace Shen, Erik Andersen, and Malte Jung. 2017. Have your Cake and Eat it Too: Foreign Language Learning with a Crowdsourced Video Captioning System. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 286--296. Google Scholar
Digital Library
- Maria Cutumisu and Daniel L Schwartz. 2016. Choosing versus Receiving Feedback: The Impact of Feedback Valence on Learning in an Assessment Game. In EDM '16. 341--346.Google Scholar
- Dan Davis, Guanliang Chen, Claudia Hauff, and Geert-Jan Houben. 2016. Gauging MOOC Learners' Adherence to the Designed Learning Path. In Proceedings of the 9th International Conference on Educational Data Mining. 54--61.Google Scholar
- Dan Davis, Guanliang Chen, Claudia Hauff, and Geert-Jan Houben. 2018. Activating learning at scale: A review of innovations in online learning strategies. Computers & Education, Vol. 125 (2018), 327 -- 344.Google Scholar
Digital Library
- Shayan Doroudi, Ece Kamar, Emma Brunskill, and Eric Horvitz. 2016. Toward a learning science for complex crowdsourcing tasks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2623--2634. Google Scholar
Digital Library
- Ujwal Gadiraju and Stefan Dietze. 2017. Improving learning through achievement priming in crowdsourced information finding microtasks. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference. ACM, 105--114. Google Scholar
Digital Library
- Ujwal Gadiraju, Besnik Fetahu, and Ricardo Kawase. 2015. Training workers for improving performance in crowdsourcing microtasks. In Design for Teaching and Learning in a Networked World. Springer, 100--114.Google Scholar
- Ujwal Gadiraju, Sebastian Möller, Martin Nöllenburg, Dietmar Saupe, Sebastian Egger-Lampl, Daniel Archambault, and Brian Fisher. 2017. Crowdsourcing Versus the Laboratory: Towards Human-Centered Experiments Using the Crowd. In Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments. Springer, 6--26.Google Scholar
- Chase Geigle and ChengXiang Zhai. 2017. Modeling Student Behavior With Two-Layer Hidden Markov Models. Journal of Educational Data Mining, Vol. 9, 1 (2017), 1--24.Google Scholar
- Catherine Grady and Matthew Lease. 2010. Crowdsourcing document relevance assessment with mechanical turk. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon's mechanical turk. Association for Computational Linguistics, 172--179. Google Scholar
Digital Library
- Philip J Guo, Juho Kim, and Rob Rubin. 2014. How video production affects student engagement: An empirical study of mooc videos. In Proceedings of the first ACM conference on Learning@ scale conference. ACM, 41--50. Google Scholar
Digital Library
- Sherif Halawa, Daniel Greene, and John Mitchell. 2014. Dropout prediction in MOOCs using learner activity features. In Experiences and best practices in and around MOOCs, Vol. 7. 3--12.Google Scholar
- Walter W Hauck and Sharon Anderson. 1984. A new statistical procedure for testing equivalence in two-group comparative bioavailability trials. Journal of Pharmacokinetics and Biopharmaceutics, Vol. 12, 1 (1984), 83--91.Google Scholar
Cross Ref
- Andrew Dean Ho, Justin Reich, Sergiy O Nesterko, Daniel Thomas Seaton, Tommy Mullaney, Jim Waldo, and Isaac Chuang. 2014. HarvardX and MITx: The first year of open online courses, fall 2012-summer 2013. Technical Report. Harvard University and Massachusetts Institute of Technology.Google Scholar
- Jeff Howe. 2008. Crowdsourcing: How the power of the crowd is driving the future of business .Random House.Google Scholar
- Gabriella Kazai and Natasa Milic-Frayling. 2009. On the evaluation of the quality of relevance assessments collected through crowdsourcing. In SIGIR 2009 Workshop on the Future of IR Evaluation. 21--22.Google Scholar
- Juho Kim, Philip J Guo, Daniel T Seaton, Piotr Mitros, Krzysztof Z Gajos, and Robert C Miller. 2014. Understanding in-video dropouts and interaction peaks inonline lecture videos. In Proceedings of the first ACM conference on Learning@ scale conference. ACM, 31--40. Google Scholar
Digital Library
- Yea-Seul Kim, Katharina Reinecke, and Jessica Hullman. 2017. Explaining the gap: Visualizing one's predictions improves recall and comprehension of data. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 1375--1386. Google Scholar
Digital Library
- Aniket Kittur, Ed H Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with Mechanical Turk. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 453--456. Google Scholar
Digital Library
- René F Kizilcec, Mar Pérez-Sanagust'in, and Jorge J Maldonado. 2017. Self-regulated learning strategies predict learner behavior and goal attainment in Massive Open Online Courses. Computers & education, Vol. 104 (2017), 18--33. Google Scholar
Digital Library
- Kenneth R Koedinger, Jihee Kim, Julianna Zhuxin Jia, Elizabeth A McLaughlin, and Norman L Bier. 2015. Learning is not a spectator sport: Doing is better than watching for learning from a MOOC. In Proceedings of the second (2015) ACM conference on learning@ scale. ACM, 111--120. Google Scholar
Digital Library
- Geza Kovacs. 2016. Effects of in-video quizzes on MOOC lecture viewing. In Proceedings of the third (2016) ACM conference on Learning@ Scale. ACM, 31--40. Google Scholar
Digital Library
- Bum Chul Kwon and Bongshin Lee. 2016. A Comparative Evaluation on Online Learning Approaches using Parallel Coordinate Visualization. In CHI '16. ACM, 993--997. Google Scholar
Digital Library
- Chappell Lawson, Gabriel S Lenz, Andy Baker, and Michael Myers. 2010. Looking like a winner: Candidate appearance and electoral success in new democracies. World Politics, Vol. 62, 4 (2010), 561--593.Google Scholar
Cross Ref
- Xiao Ma, Megan Cackett, Leslie Park, Eric Chien, and Mor Naaman. 2018. Web-Based VR Experiments Powered by the Crowd. In Proceedings of the 2018 World Wide Web Conference. 33--43. Google Scholar
Digital Library
- Jaclyn K Maass and Philip I Pavlik Jr. 2016. Modeling the Influence of Format and Depth during Effortful Retrieval Practice. In EDM '16. 143--150.Google Scholar
- Thi Thao Duyen T Nguyen, Thomas Garncarz, Felicia Ng, Laura A Dabbish, and Steven P Dow. 2017. Fruitful Feedback: Positive affective language and source anonymity improve critique reception and work outcomes. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 1024--1034. Google Scholar
Digital Library
- Jihyun Park, Kameryn Denaro, Fernando Rodriguez, Padhraic Smyth, and Mark Warschauer. 2017. Detecting changes in student behavior from clickstream data. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference. ACM, 21--30. Google Scholar
Digital Library
- Oleksandra Poquet, Lisa Lim, Negin Mirriahi, and Shane Dawson. 2018. Video and learning: a systematic review (2007--2017). In Proceedings of the 8th International Conference on Learning Analytics and Knowledge. ACM, 151--160. Google Scholar
Digital Library
- Anna N Rafferty, Rachel A Jansen, and Thomas L Griffiths. 2016. Using Inverse Planning for Personalized Feedback. In EDM '16. 472--477.Google Scholar
- Daniel T Seaton, Sergiy Nesterko, Tommy Mullaney, Justin Reich, Andrew Ho, and Isaac Chuang. 2014. Characterizing video use in the catalogue of MITx MOOCs. In European MOOC Stakeholders Summit, Lausanne. 140--146.Google Scholar
- Abdulhadi Shoufan. 2018. Estimating the cognitive value of YouTube's educational videos: A learning analytics approach. Computers in Human Behavior, Vol. -, - (2018), --.Google Scholar
- Tanmay Sinha, Patrick Jermann, Nan Li, and Pierre Dillenbourg. 2014. Your click decides your fate: Inferring Information Processing and Attrition Behavior from MOOC Video Clickstream Interactions. In 2014 Empirical Methods in Natural Language Processing Workshop on Modeling Large Scale Social Interaction in Massively Open Online Courses. 1--12.Google Scholar
Cross Ref
- David J Stanley and Jeffrey R Spence. 2014. Expectations for replications: Are yours realistic? Perspectives on Psychological Science, Vol. 9, 3 (2014), 305--318.Google Scholar
Cross Ref
- Jennifer Tosti-Kharas and Caryn Conley. 2016. Coding psychological constructs in text using Mechanical Turk: A reliable, accurate, and efficient alternative. Frontiers in psychology, Vol. 7 (2016), 741.Google Scholar
- Selen Türkay. 2016. The effects of whiteboard animations on retention and subjective experiences when learning advanced physics topics. Computers & Education, Vol. 98 (2016), 102--114. Google Scholar
Digital Library
- Donna Vakharia and Matthew Lease. 2015. Beyond Mechanical Turk: An analysis of paid crowd work platforms. In Proceedings of the iConference. 1--17.Google Scholar
- Frans Van der Sluis, Jasper Ginn, and Tim Van der Zee. 2016. Explaining Student Behavior at Scale: The influence of video complexity on student dwelling time. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale. ACM, 51--60. Google Scholar
Digital Library
- Tim Van der Zee, Wilfried Admiraal, Fred Paas, Nadira Saab, and Bas Giesbers. 2017. Effects of subtitles, complexity, and language proficiency on learning from online education videos. Journal of Media Psychology: Theories, Methods, and Applications, Vol. 29, 1 (2017), 18.Google Scholar
Cross Ref
- Miaomiao Wen and Carolyn Penstein Rosé. 2014. Identifying latent study habits by mining learner behavior patterns in massive open online courses. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management. ACM, 1983--1986. Google Scholar
Digital Library
- Hadley Wickham. 2013. Bin-summarise-smooth: a framework for visualising large data. Technical Report. had. co. nz, Tech. Rep.Google Scholar
- Joseph Jay Williams, Juho Kim, Anna Rafferty, Samuel Maldonado, Krzysztof Z. Gajos, Walter S. Lasecki, and Neil Heffernan. 2016a. AXIS: Generating Explanations at Scale with Learnersourcing and Machine Learning. In [email protected] '16. 379--388. http://dl.acm.org/citation.cfm?id=2876042 Google Scholar
Digital Library
- Joseph Jay Williams, Tania Lombrozo, Anne Hsu, Bernd Huber, and Juho Kim. 2016b. Revising Learner Misconceptions Without Feedback: Prompting for Reflection on Anomalies. In CHI '16. ACM, 470--474. Google Scholar
Digital Library
Index Terms
Evaluating Crowdworkers as a Proxy for Online Learners in Video-Based Learning Contexts
Recommendations
Using learning analytics to explore help-seeking learner profiles in MOOCs
LAK '17: Proceedings of the Seventh International Learning Analytics & Knowledge ConferenceIn online learning environments, learners are often required to be more autonomous in their approach to learning. In scaled online learning environments, like Massive Open Online Courses (MOOCs), there are differences in the ability of learners to ...
On the role of gamification and localization in an open online learning environment: javala experiences
Koli Calling '15: Proceedings of the 15th Koli Calling Conference on Computing Education ResearchMassive Open Online Courses (MOOCs) have rapidly become an important tool for educational institutes in teaching programming. Nevertheless, high drop-out rates have always been a problem in online learning. As MOOCs have become an important part of ...
Moving Through MOOCS: Pedagogy, Learning Design and Patterns of Engagement
Design for Teaching and Learning in a Networked WorldAbstractMassive open online courses (MOOCs) are part of the lifelong learning experience of people worldwide. Many of these learners participate fully. However, the high levels of dropout on most of these courses are a cause for concern. Previous studies ...






Comments