skip to main content
research-article
Free Access

Modeling, Enacting, and Integrating Custom Crowdsourcing Processes

Published:13 May 2015Publication History
Skip Abstract Section

Abstract

Crowdsourcing (CS) is the outsourcing of a unit of work to a crowd of people via an open call for contributions. Thanks to the availability of online CS platforms, such as Amazon Mechanical Turk or CrowdFlower, the practice has experienced a tremendous growth over the past few years and demonstrated its viability in a variety of fields, such as data collection and analysis or human computation. Yet it is also increasingly struggling with the inherent limitations of these platforms: each platform has its own logic of how to crowdsource work (e.g., marketplace or contest), there is only very little support for structured work (work that requires the coordination of multiple tasks), and it is hard to integrate crowdsourced tasks into state-of-the-art business process management (BPM) or information systems.

We attack these three shortcomings by (1) developing a flexible CS platform (we call it Crowd Computer, or CC) that allows one to program custom CS logics for individual and structured tasks, (2) devising a BPMN--based modeling language that allows one to program CC intuitively, (3) equipping the language with a dedicated visual editor, and (4) implementing CC on top of standard BPM technology that can easily be integrated into existing software and processes. We demonstrate the effectiveness of the approach with a case study on the crowd-based mining of mashup model patterns.

References

  1. Active Endpoints, Adobe, BEA, IBM, Oracle, SAP. 2007. WS-BPEL Extension for People (BPEL4People) Version 1.0. Technical Report.Google ScholarGoogle Scholar
  2. Salman Ahmad, Alexis Battle, Zahan Malkani, and Sepander Kamvar. 2011. The Jabberwocky programming environment for structured social computing. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST’11). 53--64. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Mohammad Allahbakhsh, Boualem Benatallah, Aleksandar Ignjatovic, Hamid Reza Motahari-Nezhad, Elisa Bertino, and Schahram Dustdar. 2013. Quality control in crowdsourcing systems: Issues and directions. IEEE Internet Computing 17, 2, 76--81. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Daniel W. Barowy, Charlie Curtsinger, Emery D. Berger, and Andrew McGregor. 2012. AutoMan: A platform for integrating human-based and digital computation. ACM SIGPLAN Notices 47, 10, 639--654. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Christian Baun, Marcel Kunze, Jens Nimis, and Stefan Tai. 2011. Cloud Computing: Web-Based Dynamic IT Services. Springer. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Alessandro Bozzon, Marco Brambilla, Stefano Ceri, Andrea Mauri, and Riccardo Volonterio. 2014. Pattern-based specification of crowdsourcing applications. In Web Engineering. Lecture Notes in Computer Science, Vol. 8541. Springer, 218--235.Google ScholarGoogle Scholar
  7. Marco Brambilla, Piero Fraternali, and Carmen Vaca. 2011. A notation for supporting social business process modeling. In Business Process Model and Notation. Lecture Notes in Business Information Processing, Vol. 95. Springer, 88--102.Google ScholarGoogle Scholar
  8. Marco Brambilla, Piero Fraternali, and Carmen Karina Vaca Ruiz. 2012. Combining social Web and BPM for improving enterprise performances: The BPM4People approach to social BPM. In Proceedings of the 21st International Conference Companion on World Wide Web (WWW’12 Companion). 223--226. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Ruggiero Cavallo and Shaili Jain. 2012. Efficient crowdsourcing contests. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems—Volume 2 (AAMAS’12). 677--686. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Crowdsourcing Week. 2014. The 2014 Global Crowdsourcing Pulsecheck: 1st Annual Survey Topline Results. Retrieved April 19, 2015, from http://www.slideshare.net/crowdsourcingweek/2014-global-crowdsourcing-pulsecheck-1st-annual-survey-topline-results.Google ScholarGoogle Scholar
  11. Stephan Curran, Kevin Feeney, Reinhard Schaler, and David Lewis. 2009. The management of crowdsourcing in business processes. In Proceedings of the IFIP/IEEE International Symposium on Integrated Network Management-Workshops, 2009. 77--78.Google ScholarGoogle ScholarCross RefCross Ref
  12. Florian Daniel and Maristella Matera. 2014. Mashups: Concepts, Models and Architectures. Springer. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Jeffrey Dean and Sanjay Ghemawat. 2008. MapReduce: Simplified data processing on large clusters. Communications of the ACM 51, 1, 107--113. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Frank Dengler, Agnes Koschmider, Andreas Oberweis, and Huayu Zhang. 2011. Social software for coordination of collaborative process activities. In Proceedings of the 3rd Workshop on Business Process Management and Social Software. 396--407.Google ScholarGoogle ScholarCross RefCross Ref
  15. Steven Dow, Anand Kulkarni, Scott Klemmer, and Björn Hartmann. 2012. Shepherding the crowd yields better work. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW’12). 1013--1022. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Selim Erol, Michael Granitzer, Simone Happ, Sami Jantunen, Ben Jennings, Paul Johannesson, Agnes Koschmider, Selmin Nurcan, Davide Rossi, and Rainer Schmidt. 2010. Combining BPM and social software: Contradiction or chance? Journal of Software Maintenance and Evolution: Research and Practice 22, 67, 449--476. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Michael J. Franklin, Donald Kossmann, Tim Kraska, Sukriti Ramesh, and Reynold Xin. 2011. CrowdDB: Answering queries with crowdsourcing. In Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data. 61--72. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Matthias Hirth, Tobias Hoßfeld, and Phuoc Tran-Gia. 2013. Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms. Mathematical and Computer Modelling 57, 11--12, 2918--2932.Google ScholarGoogle ScholarCross RefCross Ref
  19. Tobias Hoßfeld, Matthias Hirth, Pavel Korshunov, Philippe Hanhart, Bruno Gardlo, Christian Keimel, and Christian Timmerer. 2014a. Survey of Web-based crowdsourcing frameworks for subjective quality assessment. In Proceedings of the 2014 IEEE 16th International Workshop on Multimedia Signal Processing (MMSP’14). 1--6.Google ScholarGoogle Scholar
  20. Tobias Hoßfeld, Matthias Hirth, and Phuoc Tran-Gia. 2011. Modeling of crowdsourcing platforms and granularity of work organization in Future Internet. In Proceedings of the 2011 23rd International Teletraffic Congress (ITC’11). 142--149. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Tobias Hoßfeld, Christian Keimel, Matthias Hirth, Bruno Gardlo, Julian Habigt, Klaus Diepold, and Phuoc Tran-Gia. 2014b. Best practices for QoE crowdtesting: QoE assessment with crowdsourcing. IEEE Transactions on Multimedia 16, 2, 541--558. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Jeff Howe. 2008. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business. Crown Publishing Group, New York, NY. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Panagiotis G. Ipeirotis. 2010. Analyzing the Amazon Mechanical Turk marketplace. XRDS: Crossroads, the ACM Magazine for Students 17, 2, 16--21. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Panagiotis G. Ipeirotis, Foster Provost, and Jing Wang. 2010. Quality management on Amazon Mechanical Turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP’10). ACM, New York, NY, 64--67. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Paul Johannesson, Birger Andersson, and Petia Wohed. 2009. Business process management with social software systems—a new paradigm for work organisation. In Business Process Management Workshops. Lecture Notes in Business Information Processing. Springer, 659--665.Google ScholarGoogle Scholar
  26. Diane Jordan and John Evdemon. 2007. Web Services Business Process Execution Language Version 2.0. OASIS. http://docs.oasis-open.org/wsbpel/2.0/OS/wsbpel-v2.0-OS.html.Google ScholarGoogle Scholar
  27. Roman Khazankin, Benjamin Satzger, and Schahram Dustdar. 2012. Optimized execution of business processes on crowdsourcing platforms. In Proceedings of the 2012 8th International Conference on Collaborative Computing: Networking, Applications, and Worksharing (CollaborateCom’12). 443--451.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Aniket Kittur, Susheel Khamkar, Paul André, and Robert Kraut. 2012. Crowdweaver: Visually managing complex crowd work. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW’12). ACM, New York, NY, 1033--1036. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Aniket Kittur, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW’13). ACM, New York, NY, 1301--1318. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E. Kraut. 2011. CrowdForge: Crowdsourcing complex work. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST’11). 43--52. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Agnes Koschmider, Minseok Song, and Hajo A. Reijers. 2010. Social software for business process modeling. Journal of Information Technology 25, 3, 308--322.Google ScholarGoogle ScholarCross RefCross Ref
  32. Pavel Kucherbaev, Stefano Tranquillini, Florian Daniel, Fabio Casati, Maurizio Marchese, Marco Brambilla, and Piero Fraternali. 2013. Business processes for the Crowd Computer. In Business Process Management Workshops. Lecture Notes in Business Information Processing, Vol. 132. Springer, 256--267.Google ScholarGoogle Scholar
  33. Anand Kulkarni, Matthew Can, and Björn Hartmann. 2012. Collaboratively crowdsourcing workflows with turkomatic. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW’12). ACM, New York, NY, 1003--1012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Gioacchino La Vecchia and Antonio Cisternino. 2010. Collaborative workforce, business process crowdsourcing as an alternative of BPO. In Proceedings of the 10th International Conference on Current Trends in Web Engineering (ICWE’10). 425--430. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. 2010a. Exploring iterative and parallel human computation processes. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP’10). ACM, New York, NY, 68--76. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. 2010b. TurKit: Human computation algorithms on Mechanical Turk. In Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology (UIST’10). ACM, New York, NY, 57--66. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Massolution. 2013. The Crowd in the Cloud: Exploring the Future of Outsourcing. White Paper. Massolution.Google ScholarGoogle Scholar
  38. Patrick Minder and Abraham Bernstein. 2011. CrowdLang—first steps towards programmable human computers for general computation. In Proceedings of the Workshops at the 25th AAAI Conference on Artificial Intelligence.Google ScholarGoogle Scholar
  39. Patrick Minder and Abraham Bernstein. 2012. CrowdLang: A programming language for the systematic exploration of human computation systems. In Social Informatics. Lecture Notes in Computer Science, Vol. 7710. Springer, 124--137. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. David Oleson, Alexander Sorokin, Greg P. Laughlin, Vaughn Hester, John Le, and Lukas Biewald. 2011. Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. In Proceedings of the Workshops at the 25th AAAI Conference on Artificial Intelligence.Google ScholarGoogle Scholar
  41. Object Management Group. 2011. Business Process Model and Notation (BPMN) Version 2.0. Available at http://www.omg.org.Google ScholarGoogle Scholar
  42. Alexander J. Quinn and Benjamin B. Bederson. 2011. Human computation: A survey and taxonomy of a growing field. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’11). ACM, New York, NY, 1403--1412. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Carlos Rodriguez, Florian Daniel, and Fabio Casati. 2014. Crowd-based mining of reusable process model patterns. In Business Process Management. Lecture Notes in Computer Science, Vol. 8659. Springer, 51--66.Google ScholarGoogle Scholar
  44. Carlos Rodriguez, Eros Zaupa, Florian Daniel, and Fabio Casati. 2013. Crowd-Based Pattern Mining: On the Crowdsourcing of Reusable Knowledge Identification from Mashup Models. UNITN.Google ScholarGoogle Scholar
  45. Soudip Roy Chowdhury, Florian Daniel, and Fabio Casati. 2011. Efficient, interactive recommendation of mashup composition knowledge. In Proceedings of the 9th International Conference on Service Oriented Computing (ICSOC’11). 374--388. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Soudip Roy Chowdhury, Carlos Rodríguez, Florian Daniel, and Fabio Casati. 2012. Baya: Assisted mashup development as a service. In Proceedings of the 21st International Conference Companion on World Wide Web (WWW’12). ACM, New York, NY, 409--412. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Benjamin Satzger, Harald Psaier, Daniel Schall, and Schahram Dustdar. 2013. Auction-based crowdsourcing supporting skill management. Information Systems 38, 4, 547--560. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Daniel Schall, Benjamin Satzger, and Harald Psaier. 2012. Crowdsourcing tasks to social networks in BPEL4People. World Wide Web 17, 1, 1--32. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Daniel Schall, Hong-Linh Truong, and Schahram Dustdar. 2008. Unifying human and software services in web-scale collaborations. IEEE Internet Computing 12, 3, 62--68. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Florian Skopik, Daniel Schall, Harald Psaier, Martin Treiber, and Schahram Dustdar. 2011. Towards social crowd environments using service-oriented architectures. Information Technology 53, 3, 108--116.Google ScholarGoogle Scholar
  51. Mohammad Soleymani and Martha Larson. 2010. Crowdsourcing for affective annotation of video: Development of a viewer-reported boredom corpus. In Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE’10). 4--8.Google ScholarGoogle Scholar
  52. Maja Vukovic. 2009. Crowdsourcing for enterprises. In Proceedings of the 2009 World Conference on Services. 686--692. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Mathias Weske. 2007. Business Process Management: Concepts, Languages, Architectures. Springer. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Modeling, Enacting, and Integrating Custom Crowdsourcing Processes

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          • Published in

            cover image ACM Transactions on the Web
            ACM Transactions on the Web  Volume 9, Issue 2
            May 2015
            150 pages
            ISSN:1559-1131
            EISSN:1559-114X
            DOI:10.1145/2776789
            Issue’s Table of Contents

            Copyright © 2015 ACM

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            • Published: 13 May 2015
            • Accepted: 1 March 2015
            • Revised: 1 December 2014
            • Received: 1 July 2014
            Published in tweb Volume 9, Issue 2

            Permissions

            Request permissions about this article.

            Request Permissions

            Check for updates

            Qualifiers

            • research-article
            • Research
            • Refereed

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!