Abstract
Software Crowdsourcing (SW CS) allows a requester to increase the speed of its software development efforts by submitting a task to be performed by the crowd. SW CS is usually structured around software platforms, which are used by crowd members to identify a task suited for them, gather information about this task, and finally, submit a solution for it. In competitive software crowdsourcing, members of the crowd independently create solutions while competing against each other by monetary rewards for task completion. While competition usually reduces collaboration, in this paper, we investigated how crowd members create a collaborative behavior during programming challenges using online forums to help each other, share useful information, and discuss important documents and artifacts. We also investigated different collaborative behaviours by crowd members and and how this collaboration is associated with crowd members' improved outcome in the challenges. These results are based on analysis of the online forums from Topcoder, one of the largest competitive SW CS platforms
- Sabrina Adamczyk, Angelika Bullinger-Hoffmann, and Kathrin Moeslein. 2012. Innovation Contests: A Review, Classification and Outlook. Creativity and Innovation Management 21 (12 2012). https://doi.org/10.1111/caim.12003Google Scholar
- Nikolay Archak. 2010. Money, Glory and Cheap Talk: Analyzing Strategic Behavior of Contestants in Simultaneous Crowdsourcing Contests on TopCoder.com. 21--30. https://doi.org/10.1145/1772690.1772694 Google Scholar
Digital Library
- Andrew Begel, Jan Bosch, and M.-A Storey. 2013. Social Networking Meets Software Development: Perspectives from GitHub, MSDN, Stack Exchange, and TopCoder. Software, IEEE 30 (01 2013), 52--66. https://doi.org/10.1109/MS.2013.13 Google Scholar
Digital Library
- Kevin Boudreau, Patrick Gaulé, Karim Lakhani, Christoph Riedl, and AnitaWoolley. 2014. From Crowds to Collaborators: Initiating Effort Catalyzing Interactions Among Online Creative Workers. SSRN Electronic Journal (01 2014). https://doi.org/10.2139/ssrn.2384068Google Scholar
- Kevin Boudreau and Karim Lakhani. 2015. "Open" Disclosure of Innovations, Incentives and Follow-on Reuse: Theory on Processes of Cumulative Innovation and a Field Experiment in Computational Biology. Research Policy 44 (02 2015). https://doi.org/10.1016/j.respol.2014.08.001Google Scholar
- Brabham and Daren C. 2008. Crowdsourcing as a Model for Problem Solving: An Introduction and Cases. The International Journal of Research into New Media Technologies 14 (1) (01 2008), 75--90.Google Scholar
- Susan L. Bryant, Andrea Forte, and Amy Bruckman. 2005. Becoming Wikipedian: Transformation of Participation in a Collaborative Online Encyclopedia. In Proceedings of the 2005 International ACM SIGGROUP Conference on Supporting Group Work (GROUP '05). Association for Computing Machinery, New York, NY, USA, 1--10. https://doi.org/10.1145/1099203.1099205 Google Scholar
Digital Library
- Marcelo Cataldo, Patrick A.Wagstrom, James D. Herbsleb, and Kathleen M. Carley. 2006. Identification of Coordination Requirements: Implications for the Design of Collaboration and Awareness Tools. In Proceedings of the 2006 20th Anniversary Conference on Computer Supported Cooperative Work (CSCW '06). Association for Computing Machinery, New York, NY, USA, 353--362. https://doi.org/10.1145/1180875.1180929 Google Scholar
Digital Library
- Lars Christensen. 2014. Practices of Stigmergy in the Building Process. Computer Supported Cooperative Work (CSCW) 23 (02 2014). https://doi.org/10.1007/s10606-012--9181--3 Google Scholar
Digital Library
- Andrew L. Cohen, Debra Cash, and Michael J. Muller. 2000. Designing to Support Adversarial Collaboration. In Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work (CSCW '00). Association for Computing Machinery, New York, NY, USA, 31--39. https://doi.org/10.1145/358916.358948 Google Scholar
Digital Library
- J. Corbin and A. Strauss. 2008. Basics of Qualitative Research (3rd ed.): Techniques and Procedures for Developing Grounded Theory. Sage.Google Scholar
- John W Creswell. 2013. Research design: Qualitative, quantitative, and mixed methods approaches. Sage.Google Scholar
- Cleidson R. B. de Souza, Leticia S. Machado, and Ricardo Rodrigo M. Melo. 2020. On Moderating Software Crowdsourcing Challenges. Proc. ACM Hum.-Comput. Interact. 4, GROUP, Article 14 (Jan. 2020), 22 pages. https://doi.org/10.1145/ 3375194 Google Scholar
Digital Library
- Anhai Doan, Raghu Ramakrishnan, and Alon Y. Halevy. 2011. Crowdsourcing Systems on the World-Wide Web. Commun. ACM 54, 4 (April 2011), 86--96. https://doi.org/10.1145/1924421.1924442 Google Scholar
Digital Library
- Alpana Dubey, Kumar Abhinav, Sakshi Taneja, Gurdeep Virdi, Anurag Dwarakanath, Alex Kass, and Suma Mani. 2016. Dynamics of Software Development Crowdsourcing. https://doi.org/10.1109/ICGSE.2016.13Google Scholar
- Mary L. Gray, Siddharth Suri, Syed Shoaib Ali, and Deepti Kulkarni. 2016. The Crowd is a Collaborative Network. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16). ACM, New York, NY, USA, 134--147. https://doi.org/10.1145/2818048.2819942 Google Scholar
Digital Library
- Rebecca Grinter. 1996. Supporting Articulation Work Using Software Configuration Management Systems. Computer Supported Cooperative Work (CSCW) 5 (12 1996), 447--465. https://doi.org/10.1007/BF00136714 Google Scholar
Digital Library
- Rebecca Grinter. 2003. Recomposition: Coordinating aWeb of Software Dependencies. Computer Supported Cooperative Work 12 (09 2003), 297--327. https://doi.org/10.1023/A:1025012916465 Google Scholar
Digital Library
- Christian Heath and Paul Luff. 1992. Collaboration and Control: Crisis Management and Multimedia Technology in London Underground Line Control Rooms. Computer Supported Cooperative Work - CSCW 1 (03 1992), 69--94. https://doi.org/10.1007/BF00752451Google Scholar
- Erin Hoffman, David McDonald, and Mark Zachry. 2017. Evaluating a Computational Approach to Labeling Politeness: Challenges for the Application of Machine Classification to Social Computing Data. Proceedings of the ACM on Human-Computer Interaction 1 (12 2017), 1--14. https://doi.org/10.1145/3134687 Google Scholar
Digital Library
- Jeff Howe. 2008. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business (1 ed.). Crown Publishing Group, New York, NY, USA. Google Scholar
Digital Library
- Katja Hutter, Julia Hautz, Johann Füller, Julia Mueller-Seeger, and Kurt Matzler. 2011. Communitition: The Tension Between Competition and Collaboration in Community Based Design Contests. Creativity and Innovation Management 20 (02 2011), 3--21. https://doi.org/10.1111/j.1467--8691.2011.00589.xGoogle Scholar
- Evgeny Kaganer, Erran Carmel, Rudy Hirschheim, and Olsen Timothy. 2013. Managing the Human Cloud. MIT Sloan Management Review 54 (12 2013), 23.Google Scholar
- Aniket Kittur. 2010. Crowdsourcing, Collaboration and Creativity. ACM Crossroads 17 (12 2010), 22--26. https://doi.org/10.1145/1869086.1869096 Google Scholar
Digital Library
- Aniket Kittur, Jeffrey Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The Future of Crowd Work. Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW, 1301--1318. https://doi.org/10.1145/2441776.2441923 Google Scholar
Digital Library
- Karim Lakhani, David Garvin, and Eric Lonstein. 2010. TopCoder (A): Developing Software through Crowdsourcing.(01 2010).Google Scholar
- Thomas LaToza, Micky Chen, Luxi Jiang, Mengyao Zhao, and Andre van der Hoek. 2015. Borrowing from the Crowd: A Study of Recombination in Software Design Competitions. https://doi.org/10.1109/ICSE.2015.72 Google Scholar
Digital Library
- Thomas LaToza and Andre van der Hoek. 2016. Crowdsourcing in Software Engineering: Models, Motivations, and Challenges. IEEE Software 33 (01 2016), 74--80. https://doi.org/10.1109/MS.2016.12 Google Scholar
Digital Library
- Thomas D. LaToza, W. Ben Towne, Christian M. Adriano, and André van der Hoek. 2014. Microtask Programming: Building Software with a Crowd. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). Association for Computing Machinery, New York, NY, USA, 43--54. https://doi.org/10.1145/ 2642918.2647349 Google Scholar
Digital Library
- Charlotte P. Lee and Drew Paine. 2015. From The Matrix to a Model of Coordinated Action (MoCA): A Conceptual Framework of and for CSCW. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work Social Computing (CSCW '15). Association for Computing Machinery, New York, NY, USA, 179--194. https://doi.org/ 10.1145/2675133.2675161 Google Scholar
Digital Library
- Benedikt Ley, Thomas Ludwig, Volkmar Pipek, Dave Randall, Christian Reuter, and Torben Wiedenhoefer. 2014. Information and Expertise Sharing in Inter-Organizational Crisis Management. Computer Supported Cooperative Work (CSCW) 23 (12 2014), 347--387. https://doi.org/10.1007/s10606-014--9205--2 Google Scholar
Digital Library
- Leticia Machado, Alexandre Zanatta, Sabrina Marczak, and Rafael Prikladnicki. 2017. The Good, the Bad and the Ugly: An Onboard Journey in Software Crowdsourcing Competitive Model. https://doi.org/10.1109/CSI-SE.2017.6 Google Scholar
Digital Library
- Thomas Malone, Robert Laubacher, and Chrysanthos Dellarocas. 2010. The Collective Intelligence Genome. IEEE Engineering Management Review 38 (03 2010), 38--52. https://doi.org/10.1109/EMR.2010.5559142Google Scholar
Cross Ref
- Ke Mao, Licia Capra, Mark Harman, and Yue Jia. 2016. A Survey of the Use of Crowdsourcing in Software Engineering. Journal of Systems and Software 126 (09 2016). https://doi.org/10.1016/j.jss.2016.09.015Google Scholar
- Tara Matthews, Steve Whittaker, Hernan Badenes, Barton A. Smith, Michael Muller, Kate Ehrlich, Michelle X. Zhou, and Tessa Lau. 2013. Community Insights: Helping Community Leaders Enhance the Value of Enterprise Online Communities. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 513--522. https://doi.org/10.1145/2470654.2470728 Google Scholar
Digital Library
- Sreeja Nag, Ira Heffan, Alvar Saenz-Otero, and Mike Lydon. 2012. SPHERES Zero Robotics Software Development: Lessons on Crowdsourcing and Collaborative Competition. https://doi.org/10.1109/AERO.2012.6187452Google Scholar
- Catherine S. Oh, Jeremy N. Bailenson, and Gregory F. Welch. 2018. A Systematic Review of Social Presence: Definition, Antecedents, and Implications. Frontiers in Robotics and AI 5 (2018), 114. https://doi.org/10.3389/frobt.2018.00114Google Scholar
Cross Ref
- Wanda J. Orlikowski. 1992. Learning from Notes: Organizational Issues in Groupware Implementation. In Proceedings of the 1992 ACM Conference on Computer-supported Cooperative Work (CSCW '92). ACM, New York, NY, USA, 362--369. https://doi.org/10.1145/143457.143549 Google Scholar
Digital Library
- Xin Peng, Muhammad Ali Babar, and Christof Ebert. 2014. Collaborative Software Development Platforms for Crowdsourcing. Software, IEEE 31 (03 2014), 30--36. https://doi.org/10.1109/MS.2014.31Google Scholar
- Razieh Saremi and Ye Yang. 2015. Dynamic Simulation of Software Workers and Task Completion. 17--23. https: //doi.org/10.1109/CSI-SE.2015.11 Google Scholar
Digital Library
- Razieh Saremi, Ye Yang, Guenther Ruhe, and David Messinger. 2017. Leveraging Crowdsourcing For Team Elasticity: An Empirical Evaluation at TopCoder. ICSE 2017 SEIP (05 2017). Google Scholar
Digital Library
- Gregory Saxton, Onook Oh, and Rajiv Kishore. 2013. Rules of Crowdsourcing: Models, Issues, and Systems of Control. Information Systems Management (01 2013). https://doi.org/10.1080/10580530.2013.739883 Google Scholar
Digital Library
- Eric Schenk and Claude Guittard. 2011. Towards a characterization of crowdsourcing practices. Journal of Innovation Economics n°7 (01 2011), 93--107. https://doi.org/10.3917/jie.007.0093Google Scholar
- Klaas-Jan Stol, Bora Caglayan, and Brian Fitzgerald. 2017. Competition-Based Crowdsourcing Software Development: A Multi-Method Study from a Customer Perspective. IEEE Transactions on Software Engineering PP (11 2017). https: //doi.org/10.1109/TSE.2017.2774297 Google Scholar
Digital Library
- Klaas-Jan Stol and Brian Fitzgerald. 2014. Two's Company, Three's a Crowd: A Case Study of Crowdsourcing Software Development. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). ACM, New York, NY, USA, 187--198. https://doi.org/10.1145/2568225.2568249 Google Scholar
Digital Library
- Yla Tausczik and Ping Wang. 2017. To Share, or Not to Share?: Community-Level Collaboration in Open Innovation Contests. Proc. ACM Hum.-Comput. Interact. 1, CSCW, Article 100 (Dec. 2017), 23 pages. https://doi.org/10.1145/3134735 Google Scholar
Digital Library
- Hon Jie Teo and Aditya Johri. 2014. Fast, Functional, and Fitting: Expert Response Dynamics and Response Quality in an Online Newcomer Help Forum. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work amp; Social Computing (CSCW '14). Association for Computing Machinery, New York, NY, USA, 332--341. https://doi.org/10.1145/2531602.2531731 Google Scholar
Digital Library
- TopCoder. 2017. TopCoder. Retrieved March 18, 2019 from https://www.topcoder.comGoogle Scholar
- Luis Vaz, Sabrina Marczak, and Igor Steinmacher. 2018. An empirical study on task documentation in software crowdsourcing: the case of the topcoder platform. 62--71. https://doi.org/10.1145/3266237.3266265 Google Scholar
Digital Library
- Fernanda Viegas, Martin Wattenberg, and Kushal Dave. 2004. Studying Cooperation and Conflict between Authors with History Flow Visualizations. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 6. https://doi.org/10.1145/985692.985765 Google Scholar
Digital Library
- Fernanda B. Viégas, Martin Wattenberg, and Kushal Dave. 2004. Studying Cooperation and Conflict between Authors with History Flow Visualizations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04). Association for Computing Machinery, New York, NY, USA, 575--582. https://doi.org/10.1145/ 985692.985765 Google Scholar
Digital Library
- Yi-ChiaWang, Mahesh Joshi, and Carolyn Rosé. 2008. Investigating the effect of discussion forum interface affordances on patterns of conversational interactions. 555--558. https://doi.org/10.1145/1460563.1460650 Google Scholar
Digital Library
- WenjunWu,Wei-Tek Tsai, andWei Li. 2013. An evaluation framework for software crowdsourcing. Frontiers of Computer Science: Selected Publications from Chinese Universities 7 (10 2013), 694--709. https://doi.org/10.1007/s11704-013--2320--2 Google Scholar
Digital Library
- Ye Yang, Muhammad Rezaul Karim, Razieh Saremi, and Guenther Ruhe. 2016. Who Should Take This Task?: Dynamic Decision Support for Crowd Workers. 1--10. https://doi.org/10.1145/2961111.2962594 Google Scholar
Digital Library
- Ye Yang and Razieh Saremi. 2015. Award vs. Worker Behaviors in Competitive Crowdsourcing Tasks. Empirical Software Engineering and Measurement (ESEM), 2015 ACM/IEEE International Symposium on (10 2015).Google Scholar
- Alexandre Zanatta, Leticia Machado, Graziela Pereira, Rafael Prikladnicki, and Erran Carmel. 2016. Software Crowdsourcing Platforms. IEEE Software 33 (11 2016), 112--116. https://doi.org/10.1109/MS.2016.151 Google Scholar
Digital Library
Index Terms
Collaborative Behavior and Winning Challenges in Competitive Software Crowdsourcing
Recommendations
On Moderating Software Crowdsourcing Challenges
GROUPCrowdsourcing divides a task into small pieces that are carried out by the crowd. In Software Engineering, crowdsourcing divides the software development tasks of to be carried out online by the crow and is simply called Software Crowdsourcing (SW CS). ...
An empirical study on task documentation in software crowdsourcing: the case of the topcoder platform
SBES '18: Proceedings of the XXXII Brazilian Symposium on Software EngineeringSoftware Crowdsourcing, the act of outsourcing software development tasks to a crowd in the form of an open call, happens mediated by a platform and is based on tasks. In the competitive model, the members of the crowd seek for tasks and submit ...
Software crowdsourcing reliability: an empirical study on developers behavior
SWAN 2016: Proceedings of the 2nd International Workshop on Software AnalyticsCrowdsourcing has become an emergent paradigm for software production in recent decades. Its open-call format attracts the participation of hundreds of thousands of developers. To ensure the success of software crowdsourcing, we must accurately measure ...






Comments