skip to main content
research-article

Collaborative Behavior and Winning Challenges in Competitive Software Crowdsourcing

Published:13 July 2021Publication History
Skip Abstract Section

Abstract

Software Crowdsourcing (SW CS) allows a requester to increase the speed of its software development efforts by submitting a task to be performed by the crowd. SW CS is usually structured around software platforms, which are used by crowd members to identify a task suited for them, gather information about this task, and finally, submit a solution for it. In competitive software crowdsourcing, members of the crowd independently create solutions while competing against each other by monetary rewards for task completion. While competition usually reduces collaboration, in this paper, we investigated how crowd members create a collaborative behavior during programming challenges using online forums to help each other, share useful information, and discuss important documents and artifacts. We also investigated different collaborative behaviours by crowd members and and how this collaboration is associated with crowd members' improved outcome in the challenges. These results are based on analysis of the online forums from Topcoder, one of the largest competitive SW CS platforms

References

  1. Sabrina Adamczyk, Angelika Bullinger-Hoffmann, and Kathrin Moeslein. 2012. Innovation Contests: A Review, Classification and Outlook. Creativity and Innovation Management 21 (12 2012). https://doi.org/10.1111/caim.12003Google ScholarGoogle Scholar
  2. Nikolay Archak. 2010. Money, Glory and Cheap Talk: Analyzing Strategic Behavior of Contestants in Simultaneous Crowdsourcing Contests on TopCoder.com. 21--30. https://doi.org/10.1145/1772690.1772694 Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Andrew Begel, Jan Bosch, and M.-A Storey. 2013. Social Networking Meets Software Development: Perspectives from GitHub, MSDN, Stack Exchange, and TopCoder. Software, IEEE 30 (01 2013), 52--66. https://doi.org/10.1109/MS.2013.13 Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Kevin Boudreau, Patrick Gaulé, Karim Lakhani, Christoph Riedl, and AnitaWoolley. 2014. From Crowds to Collaborators: Initiating Effort Catalyzing Interactions Among Online Creative Workers. SSRN Electronic Journal (01 2014). https://doi.org/10.2139/ssrn.2384068Google ScholarGoogle Scholar
  5. Kevin Boudreau and Karim Lakhani. 2015. "Open" Disclosure of Innovations, Incentives and Follow-on Reuse: Theory on Processes of Cumulative Innovation and a Field Experiment in Computational Biology. Research Policy 44 (02 2015). https://doi.org/10.1016/j.respol.2014.08.001Google ScholarGoogle Scholar
  6. Brabham and Daren C. 2008. Crowdsourcing as a Model for Problem Solving: An Introduction and Cases. The International Journal of Research into New Media Technologies 14 (1) (01 2008), 75--90.Google ScholarGoogle Scholar
  7. Susan L. Bryant, Andrea Forte, and Amy Bruckman. 2005. Becoming Wikipedian: Transformation of Participation in a Collaborative Online Encyclopedia. In Proceedings of the 2005 International ACM SIGGROUP Conference on Supporting Group Work (GROUP '05). Association for Computing Machinery, New York, NY, USA, 1--10. https://doi.org/10.1145/1099203.1099205 Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Marcelo Cataldo, Patrick A.Wagstrom, James D. Herbsleb, and Kathleen M. Carley. 2006. Identification of Coordination Requirements: Implications for the Design of Collaboration and Awareness Tools. In Proceedings of the 2006 20th Anniversary Conference on Computer Supported Cooperative Work (CSCW '06). Association for Computing Machinery, New York, NY, USA, 353--362. https://doi.org/10.1145/1180875.1180929 Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Lars Christensen. 2014. Practices of Stigmergy in the Building Process. Computer Supported Cooperative Work (CSCW) 23 (02 2014). https://doi.org/10.1007/s10606-012--9181--3 Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Andrew L. Cohen, Debra Cash, and Michael J. Muller. 2000. Designing to Support Adversarial Collaboration. In Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work (CSCW '00). Association for Computing Machinery, New York, NY, USA, 31--39. https://doi.org/10.1145/358916.358948 Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. J. Corbin and A. Strauss. 2008. Basics of Qualitative Research (3rd ed.): Techniques and Procedures for Developing Grounded Theory. Sage.Google ScholarGoogle Scholar
  12. John W Creswell. 2013. Research design: Qualitative, quantitative, and mixed methods approaches. Sage.Google ScholarGoogle Scholar
  13. Cleidson R. B. de Souza, Leticia S. Machado, and Ricardo Rodrigo M. Melo. 2020. On Moderating Software Crowdsourcing Challenges. Proc. ACM Hum.-Comput. Interact. 4, GROUP, Article 14 (Jan. 2020), 22 pages. https://doi.org/10.1145/ 3375194 Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Anhai Doan, Raghu Ramakrishnan, and Alon Y. Halevy. 2011. Crowdsourcing Systems on the World-Wide Web. Commun. ACM 54, 4 (April 2011), 86--96. https://doi.org/10.1145/1924421.1924442 Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Alpana Dubey, Kumar Abhinav, Sakshi Taneja, Gurdeep Virdi, Anurag Dwarakanath, Alex Kass, and Suma Mani. 2016. Dynamics of Software Development Crowdsourcing. https://doi.org/10.1109/ICGSE.2016.13Google ScholarGoogle Scholar
  16. Mary L. Gray, Siddharth Suri, Syed Shoaib Ali, and Deepti Kulkarni. 2016. The Crowd is a Collaborative Network. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16). ACM, New York, NY, USA, 134--147. https://doi.org/10.1145/2818048.2819942 Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Rebecca Grinter. 1996. Supporting Articulation Work Using Software Configuration Management Systems. Computer Supported Cooperative Work (CSCW) 5 (12 1996), 447--465. https://doi.org/10.1007/BF00136714 Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Rebecca Grinter. 2003. Recomposition: Coordinating aWeb of Software Dependencies. Computer Supported Cooperative Work 12 (09 2003), 297--327. https://doi.org/10.1023/A:1025012916465 Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Christian Heath and Paul Luff. 1992. Collaboration and Control: Crisis Management and Multimedia Technology in London Underground Line Control Rooms. Computer Supported Cooperative Work - CSCW 1 (03 1992), 69--94. https://doi.org/10.1007/BF00752451Google ScholarGoogle Scholar
  20. Erin Hoffman, David McDonald, and Mark Zachry. 2017. Evaluating a Computational Approach to Labeling Politeness: Challenges for the Application of Machine Classification to Social Computing Data. Proceedings of the ACM on Human-Computer Interaction 1 (12 2017), 1--14. https://doi.org/10.1145/3134687 Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Jeff Howe. 2008. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business (1 ed.). Crown Publishing Group, New York, NY, USA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Katja Hutter, Julia Hautz, Johann Füller, Julia Mueller-Seeger, and Kurt Matzler. 2011. Communitition: The Tension Between Competition and Collaboration in Community Based Design Contests. Creativity and Innovation Management 20 (02 2011), 3--21. https://doi.org/10.1111/j.1467--8691.2011.00589.xGoogle ScholarGoogle Scholar
  23. Evgeny Kaganer, Erran Carmel, Rudy Hirschheim, and Olsen Timothy. 2013. Managing the Human Cloud. MIT Sloan Management Review 54 (12 2013), 23.Google ScholarGoogle Scholar
  24. Aniket Kittur. 2010. Crowdsourcing, Collaboration and Creativity. ACM Crossroads 17 (12 2010), 22--26. https://doi.org/10.1145/1869086.1869096 Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Aniket Kittur, Jeffrey Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The Future of Crowd Work. Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW, 1301--1318. https://doi.org/10.1145/2441776.2441923 Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Karim Lakhani, David Garvin, and Eric Lonstein. 2010. TopCoder (A): Developing Software through Crowdsourcing.(01 2010).Google ScholarGoogle Scholar
  27. Thomas LaToza, Micky Chen, Luxi Jiang, Mengyao Zhao, and Andre van der Hoek. 2015. Borrowing from the Crowd: A Study of Recombination in Software Design Competitions. https://doi.org/10.1109/ICSE.2015.72 Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Thomas LaToza and Andre van der Hoek. 2016. Crowdsourcing in Software Engineering: Models, Motivations, and Challenges. IEEE Software 33 (01 2016), 74--80. https://doi.org/10.1109/MS.2016.12 Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Thomas D. LaToza, W. Ben Towne, Christian M. Adriano, and André van der Hoek. 2014. Microtask Programming: Building Software with a Crowd. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). Association for Computing Machinery, New York, NY, USA, 43--54. https://doi.org/10.1145/ 2642918.2647349 Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Charlotte P. Lee and Drew Paine. 2015. From The Matrix to a Model of Coordinated Action (MoCA): A Conceptual Framework of and for CSCW. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work Social Computing (CSCW '15). Association for Computing Machinery, New York, NY, USA, 179--194. https://doi.org/ 10.1145/2675133.2675161 Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Benedikt Ley, Thomas Ludwig, Volkmar Pipek, Dave Randall, Christian Reuter, and Torben Wiedenhoefer. 2014. Information and Expertise Sharing in Inter-Organizational Crisis Management. Computer Supported Cooperative Work (CSCW) 23 (12 2014), 347--387. https://doi.org/10.1007/s10606-014--9205--2 Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Leticia Machado, Alexandre Zanatta, Sabrina Marczak, and Rafael Prikladnicki. 2017. The Good, the Bad and the Ugly: An Onboard Journey in Software Crowdsourcing Competitive Model. https://doi.org/10.1109/CSI-SE.2017.6 Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Thomas Malone, Robert Laubacher, and Chrysanthos Dellarocas. 2010. The Collective Intelligence Genome. IEEE Engineering Management Review 38 (03 2010), 38--52. https://doi.org/10.1109/EMR.2010.5559142Google ScholarGoogle ScholarCross RefCross Ref
  34. Ke Mao, Licia Capra, Mark Harman, and Yue Jia. 2016. A Survey of the Use of Crowdsourcing in Software Engineering. Journal of Systems and Software 126 (09 2016). https://doi.org/10.1016/j.jss.2016.09.015Google ScholarGoogle Scholar
  35. Tara Matthews, Steve Whittaker, Hernan Badenes, Barton A. Smith, Michael Muller, Kate Ehrlich, Michelle X. Zhou, and Tessa Lau. 2013. Community Insights: Helping Community Leaders Enhance the Value of Enterprise Online Communities. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 513--522. https://doi.org/10.1145/2470654.2470728 Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Sreeja Nag, Ira Heffan, Alvar Saenz-Otero, and Mike Lydon. 2012. SPHERES Zero Robotics Software Development: Lessons on Crowdsourcing and Collaborative Competition. https://doi.org/10.1109/AERO.2012.6187452Google ScholarGoogle Scholar
  37. Catherine S. Oh, Jeremy N. Bailenson, and Gregory F. Welch. 2018. A Systematic Review of Social Presence: Definition, Antecedents, and Implications. Frontiers in Robotics and AI 5 (2018), 114. https://doi.org/10.3389/frobt.2018.00114Google ScholarGoogle ScholarCross RefCross Ref
  38. Wanda J. Orlikowski. 1992. Learning from Notes: Organizational Issues in Groupware Implementation. In Proceedings of the 1992 ACM Conference on Computer-supported Cooperative Work (CSCW '92). ACM, New York, NY, USA, 362--369. https://doi.org/10.1145/143457.143549 Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Xin Peng, Muhammad Ali Babar, and Christof Ebert. 2014. Collaborative Software Development Platforms for Crowdsourcing. Software, IEEE 31 (03 2014), 30--36. https://doi.org/10.1109/MS.2014.31Google ScholarGoogle Scholar
  40. Razieh Saremi and Ye Yang. 2015. Dynamic Simulation of Software Workers and Task Completion. 17--23. https: //doi.org/10.1109/CSI-SE.2015.11 Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Razieh Saremi, Ye Yang, Guenther Ruhe, and David Messinger. 2017. Leveraging Crowdsourcing For Team Elasticity: An Empirical Evaluation at TopCoder. ICSE 2017 SEIP (05 2017). Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Gregory Saxton, Onook Oh, and Rajiv Kishore. 2013. Rules of Crowdsourcing: Models, Issues, and Systems of Control. Information Systems Management (01 2013). https://doi.org/10.1080/10580530.2013.739883 Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Eric Schenk and Claude Guittard. 2011. Towards a characterization of crowdsourcing practices. Journal of Innovation Economics n°7 (01 2011), 93--107. https://doi.org/10.3917/jie.007.0093Google ScholarGoogle Scholar
  44. Klaas-Jan Stol, Bora Caglayan, and Brian Fitzgerald. 2017. Competition-Based Crowdsourcing Software Development: A Multi-Method Study from a Customer Perspective. IEEE Transactions on Software Engineering PP (11 2017). https: //doi.org/10.1109/TSE.2017.2774297 Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Klaas-Jan Stol and Brian Fitzgerald. 2014. Two's Company, Three's a Crowd: A Case Study of Crowdsourcing Software Development. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). ACM, New York, NY, USA, 187--198. https://doi.org/10.1145/2568225.2568249 Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Yla Tausczik and Ping Wang. 2017. To Share, or Not to Share?: Community-Level Collaboration in Open Innovation Contests. Proc. ACM Hum.-Comput. Interact. 1, CSCW, Article 100 (Dec. 2017), 23 pages. https://doi.org/10.1145/3134735 Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Hon Jie Teo and Aditya Johri. 2014. Fast, Functional, and Fitting: Expert Response Dynamics and Response Quality in an Online Newcomer Help Forum. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work amp; Social Computing (CSCW '14). Association for Computing Machinery, New York, NY, USA, 332--341. https://doi.org/10.1145/2531602.2531731 Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. TopCoder. 2017. TopCoder. Retrieved March 18, 2019 from https://www.topcoder.comGoogle ScholarGoogle Scholar
  49. Luis Vaz, Sabrina Marczak, and Igor Steinmacher. 2018. An empirical study on task documentation in software crowdsourcing: the case of the topcoder platform. 62--71. https://doi.org/10.1145/3266237.3266265 Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Fernanda Viegas, Martin Wattenberg, and Kushal Dave. 2004. Studying Cooperation and Conflict between Authors with History Flow Visualizations. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 6. https://doi.org/10.1145/985692.985765 Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Fernanda B. Viégas, Martin Wattenberg, and Kushal Dave. 2004. Studying Cooperation and Conflict between Authors with History Flow Visualizations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04). Association for Computing Machinery, New York, NY, USA, 575--582. https://doi.org/10.1145/ 985692.985765 Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Yi-ChiaWang, Mahesh Joshi, and Carolyn Rosé. 2008. Investigating the effect of discussion forum interface affordances on patterns of conversational interactions. 555--558. https://doi.org/10.1145/1460563.1460650 Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. WenjunWu,Wei-Tek Tsai, andWei Li. 2013. An evaluation framework for software crowdsourcing. Frontiers of Computer Science: Selected Publications from Chinese Universities 7 (10 2013), 694--709. https://doi.org/10.1007/s11704-013--2320--2 Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Ye Yang, Muhammad Rezaul Karim, Razieh Saremi, and Guenther Ruhe. 2016. Who Should Take This Task?: Dynamic Decision Support for Crowd Workers. 1--10. https://doi.org/10.1145/2961111.2962594 Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Ye Yang and Razieh Saremi. 2015. Award vs. Worker Behaviors in Competitive Crowdsourcing Tasks. Empirical Software Engineering and Measurement (ESEM), 2015 ACM/IEEE International Symposium on (10 2015).Google ScholarGoogle Scholar
  56. Alexandre Zanatta, Leticia Machado, Graziela Pereira, Rafael Prikladnicki, and Erran Carmel. 2016. Software Crowdsourcing Platforms. IEEE Software 33 (11 2016), 112--116. https://doi.org/10.1109/MS.2016.151 Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Collaborative Behavior and Winning Challenges in Competitive Software Crowdsourcing

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!