Abstract
Crowdsourcing (CS) is the outsourcing of a unit of work to a crowd of people via an open call for contributions. Thanks to the availability of online CS platforms, such as Amazon Mechanical Turk or CrowdFlower, the practice has experienced a tremendous growth over the past few years and demonstrated its viability in a variety of fields, such as data collection and analysis or human computation. Yet it is also increasingly struggling with the inherent limitations of these platforms: each platform has its own logic of how to crowdsource work (e.g., marketplace or contest), there is only very little support for structured work (work that requires the coordination of multiple tasks), and it is hard to integrate crowdsourced tasks into state-of-the-art business process management (BPM) or information systems.
We attack these three shortcomings by (1) developing a flexible CS platform (we call it Crowd Computer, or CC) that allows one to program custom CS logics for individual and structured tasks, (2) devising a BPMN--based modeling language that allows one to program CC intuitively, (3) equipping the language with a dedicated visual editor, and (4) implementing CC on top of standard BPM technology that can easily be integrated into existing software and processes. We demonstrate the effectiveness of the approach with a case study on the crowd-based mining of mashup model patterns.
- Active Endpoints, Adobe, BEA, IBM, Oracle, SAP. 2007. WS-BPEL Extension for People (BPEL4People) Version 1.0. Technical Report.Google Scholar
- Salman Ahmad, Alexis Battle, Zahan Malkani, and Sepander Kamvar. 2011. The Jabberwocky programming environment for structured social computing. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST’11). 53--64. Google Scholar
Digital Library
- Mohammad Allahbakhsh, Boualem Benatallah, Aleksandar Ignjatovic, Hamid Reza Motahari-Nezhad, Elisa Bertino, and Schahram Dustdar. 2013. Quality control in crowdsourcing systems: Issues and directions. IEEE Internet Computing 17, 2, 76--81. Google Scholar
Digital Library
- Daniel W. Barowy, Charlie Curtsinger, Emery D. Berger, and Andrew McGregor. 2012. AutoMan: A platform for integrating human-based and digital computation. ACM SIGPLAN Notices 47, 10, 639--654. Google Scholar
Digital Library
- Christian Baun, Marcel Kunze, Jens Nimis, and Stefan Tai. 2011. Cloud Computing: Web-Based Dynamic IT Services. Springer. Google Scholar
Digital Library
- Alessandro Bozzon, Marco Brambilla, Stefano Ceri, Andrea Mauri, and Riccardo Volonterio. 2014. Pattern-based specification of crowdsourcing applications. In Web Engineering. Lecture Notes in Computer Science, Vol. 8541. Springer, 218--235.Google Scholar
- Marco Brambilla, Piero Fraternali, and Carmen Vaca. 2011. A notation for supporting social business process modeling. In Business Process Model and Notation. Lecture Notes in Business Information Processing, Vol. 95. Springer, 88--102.Google Scholar
- Marco Brambilla, Piero Fraternali, and Carmen Karina Vaca Ruiz. 2012. Combining social Web and BPM for improving enterprise performances: The BPM4People approach to social BPM. In Proceedings of the 21st International Conference Companion on World Wide Web (WWW’12 Companion). 223--226. Google Scholar
Digital Library
- Ruggiero Cavallo and Shaili Jain. 2012. Efficient crowdsourcing contests. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems—Volume 2 (AAMAS’12). 677--686. Google Scholar
Digital Library
- Crowdsourcing Week. 2014. The 2014 Global Crowdsourcing Pulsecheck: 1st Annual Survey Topline Results. Retrieved April 19, 2015, from http://www.slideshare.net/crowdsourcingweek/2014-global-crowdsourcing-pulsecheck-1st-annual-survey-topline-results.Google Scholar
- Stephan Curran, Kevin Feeney, Reinhard Schaler, and David Lewis. 2009. The management of crowdsourcing in business processes. In Proceedings of the IFIP/IEEE International Symposium on Integrated Network Management-Workshops, 2009. 77--78.Google Scholar
Cross Ref
- Florian Daniel and Maristella Matera. 2014. Mashups: Concepts, Models and Architectures. Springer. Google Scholar
Digital Library
- Jeffrey Dean and Sanjay Ghemawat. 2008. MapReduce: Simplified data processing on large clusters. Communications of the ACM 51, 1, 107--113. Google Scholar
Digital Library
- Frank Dengler, Agnes Koschmider, Andreas Oberweis, and Huayu Zhang. 2011. Social software for coordination of collaborative process activities. In Proceedings of the 3rd Workshop on Business Process Management and Social Software. 396--407.Google Scholar
Cross Ref
- Steven Dow, Anand Kulkarni, Scott Klemmer, and Björn Hartmann. 2012. Shepherding the crowd yields better work. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW’12). 1013--1022. Google Scholar
Digital Library
- Selim Erol, Michael Granitzer, Simone Happ, Sami Jantunen, Ben Jennings, Paul Johannesson, Agnes Koschmider, Selmin Nurcan, Davide Rossi, and Rainer Schmidt. 2010. Combining BPM and social software: Contradiction or chance? Journal of Software Maintenance and Evolution: Research and Practice 22, 67, 449--476. Google Scholar
Digital Library
- Michael J. Franklin, Donald Kossmann, Tim Kraska, Sukriti Ramesh, and Reynold Xin. 2011. CrowdDB: Answering queries with crowdsourcing. In Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data. 61--72. Google Scholar
Digital Library
- Matthias Hirth, Tobias Hoßfeld, and Phuoc Tran-Gia. 2013. Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms. Mathematical and Computer Modelling 57, 11--12, 2918--2932.Google Scholar
Cross Ref
- Tobias Hoßfeld, Matthias Hirth, Pavel Korshunov, Philippe Hanhart, Bruno Gardlo, Christian Keimel, and Christian Timmerer. 2014a. Survey of Web-based crowdsourcing frameworks for subjective quality assessment. In Proceedings of the 2014 IEEE 16th International Workshop on Multimedia Signal Processing (MMSP’14). 1--6.Google Scholar
- Tobias Hoßfeld, Matthias Hirth, and Phuoc Tran-Gia. 2011. Modeling of crowdsourcing platforms and granularity of work organization in Future Internet. In Proceedings of the 2011 23rd International Teletraffic Congress (ITC’11). 142--149. Google Scholar
Digital Library
- Tobias Hoßfeld, Christian Keimel, Matthias Hirth, Bruno Gardlo, Julian Habigt, Klaus Diepold, and Phuoc Tran-Gia. 2014b. Best practices for QoE crowdtesting: QoE assessment with crowdsourcing. IEEE Transactions on Multimedia 16, 2, 541--558. Google Scholar
Digital Library
- Jeff Howe. 2008. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business. Crown Publishing Group, New York, NY. Google Scholar
Digital Library
- Panagiotis G. Ipeirotis. 2010. Analyzing the Amazon Mechanical Turk marketplace. XRDS: Crossroads, the ACM Magazine for Students 17, 2, 16--21. Google Scholar
Digital Library
- Panagiotis G. Ipeirotis, Foster Provost, and Jing Wang. 2010. Quality management on Amazon Mechanical Turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP’10). ACM, New York, NY, 64--67. Google Scholar
Digital Library
- Paul Johannesson, Birger Andersson, and Petia Wohed. 2009. Business process management with social software systems—a new paradigm for work organisation. In Business Process Management Workshops. Lecture Notes in Business Information Processing. Springer, 659--665.Google Scholar
- Diane Jordan and John Evdemon. 2007. Web Services Business Process Execution Language Version 2.0. OASIS. http://docs.oasis-open.org/wsbpel/2.0/OS/wsbpel-v2.0-OS.html.Google Scholar
- Roman Khazankin, Benjamin Satzger, and Schahram Dustdar. 2012. Optimized execution of business processes on crowdsourcing platforms. In Proceedings of the 2012 8th International Conference on Collaborative Computing: Networking, Applications, and Worksharing (CollaborateCom’12). 443--451.Google Scholar
Digital Library
- Aniket Kittur, Susheel Khamkar, Paul André, and Robert Kraut. 2012. Crowdweaver: Visually managing complex crowd work. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW’12). ACM, New York, NY, 1033--1036. Google Scholar
Digital Library
- Aniket Kittur, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW’13). ACM, New York, NY, 1301--1318. Google Scholar
Digital Library
- Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E. Kraut. 2011. CrowdForge: Crowdsourcing complex work. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST’11). 43--52. Google Scholar
Digital Library
- Agnes Koschmider, Minseok Song, and Hajo A. Reijers. 2010. Social software for business process modeling. Journal of Information Technology 25, 3, 308--322.Google Scholar
Cross Ref
- Pavel Kucherbaev, Stefano Tranquillini, Florian Daniel, Fabio Casati, Maurizio Marchese, Marco Brambilla, and Piero Fraternali. 2013. Business processes for the Crowd Computer. In Business Process Management Workshops. Lecture Notes in Business Information Processing, Vol. 132. Springer, 256--267.Google Scholar
- Anand Kulkarni, Matthew Can, and Björn Hartmann. 2012. Collaboratively crowdsourcing workflows with turkomatic. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW’12). ACM, New York, NY, 1003--1012. Google Scholar
Digital Library
- Gioacchino La Vecchia and Antonio Cisternino. 2010. Collaborative workforce, business process crowdsourcing as an alternative of BPO. In Proceedings of the 10th International Conference on Current Trends in Web Engineering (ICWE’10). 425--430. Google Scholar
Digital Library
- Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. 2010a. Exploring iterative and parallel human computation processes. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP’10). ACM, New York, NY, 68--76. Google Scholar
Digital Library
- Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. 2010b. TurKit: Human computation algorithms on Mechanical Turk. In Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology (UIST’10). ACM, New York, NY, 57--66. Google Scholar
Digital Library
- Massolution. 2013. The Crowd in the Cloud: Exploring the Future of Outsourcing. White Paper. Massolution.Google Scholar
- Patrick Minder and Abraham Bernstein. 2011. CrowdLang—first steps towards programmable human computers for general computation. In Proceedings of the Workshops at the 25th AAAI Conference on Artificial Intelligence.Google Scholar
- Patrick Minder and Abraham Bernstein. 2012. CrowdLang: A programming language for the systematic exploration of human computation systems. In Social Informatics. Lecture Notes in Computer Science, Vol. 7710. Springer, 124--137. Google Scholar
Digital Library
- David Oleson, Alexander Sorokin, Greg P. Laughlin, Vaughn Hester, John Le, and Lukas Biewald. 2011. Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. In Proceedings of the Workshops at the 25th AAAI Conference on Artificial Intelligence.Google Scholar
- Object Management Group. 2011. Business Process Model and Notation (BPMN) Version 2.0. Available at http://www.omg.org.Google Scholar
- Alexander J. Quinn and Benjamin B. Bederson. 2011. Human computation: A survey and taxonomy of a growing field. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’11). ACM, New York, NY, 1403--1412. Google Scholar
Digital Library
- Carlos Rodriguez, Florian Daniel, and Fabio Casati. 2014. Crowd-based mining of reusable process model patterns. In Business Process Management. Lecture Notes in Computer Science, Vol. 8659. Springer, 51--66.Google Scholar
- Carlos Rodriguez, Eros Zaupa, Florian Daniel, and Fabio Casati. 2013. Crowd-Based Pattern Mining: On the Crowdsourcing of Reusable Knowledge Identification from Mashup Models. UNITN.Google Scholar
- Soudip Roy Chowdhury, Florian Daniel, and Fabio Casati. 2011. Efficient, interactive recommendation of mashup composition knowledge. In Proceedings of the 9th International Conference on Service Oriented Computing (ICSOC’11). 374--388. Google Scholar
Digital Library
- Soudip Roy Chowdhury, Carlos Rodríguez, Florian Daniel, and Fabio Casati. 2012. Baya: Assisted mashup development as a service. In Proceedings of the 21st International Conference Companion on World Wide Web (WWW’12). ACM, New York, NY, 409--412. Google Scholar
Digital Library
- Benjamin Satzger, Harald Psaier, Daniel Schall, and Schahram Dustdar. 2013. Auction-based crowdsourcing supporting skill management. Information Systems 38, 4, 547--560. Google Scholar
Digital Library
- Daniel Schall, Benjamin Satzger, and Harald Psaier. 2012. Crowdsourcing tasks to social networks in BPEL4People. World Wide Web 17, 1, 1--32. Google Scholar
Digital Library
- Daniel Schall, Hong-Linh Truong, and Schahram Dustdar. 2008. Unifying human and software services in web-scale collaborations. IEEE Internet Computing 12, 3, 62--68. Google Scholar
Digital Library
- Florian Skopik, Daniel Schall, Harald Psaier, Martin Treiber, and Schahram Dustdar. 2011. Towards social crowd environments using service-oriented architectures. Information Technology 53, 3, 108--116.Google Scholar
- Mohammad Soleymani and Martha Larson. 2010. Crowdsourcing for affective annotation of video: Development of a viewer-reported boredom corpus. In Proceedings of the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE’10). 4--8.Google Scholar
- Maja Vukovic. 2009. Crowdsourcing for enterprises. In Proceedings of the 2009 World Conference on Services. 686--692. Google Scholar
Digital Library
- Mathias Weske. 2007. Business Process Management: Concepts, Languages, Architectures. Springer. Google Scholar
Digital Library
Index Terms
Modeling, Enacting, and Integrating Custom Crowdsourcing Processes
Recommendations
A model-driven method for enacting the design-time QoS analysis of business processes
Business Process Management (BPM) is a holistic approach for describing, analyzing, executing, managing, and improving large enterprise business processes. A business process can be seen as a flow of tasks that are orchestrated to accomplish well-...
Model-driven engineering with domain-specific meta-modelling languages
Domain-specific modelling languages are normally defined through general-purpose meta-modelling languages like the MOF. While this is satisfactory for many model-driven engineering (MDE) projects, several researchers have identified the need for domain-...
On-chain behavior prediction Machine Learning model for blockchain-based crowdsourcing
AbstractIn this paper, we address the problem of behavior prediction for task allocation in blockchain-based crowdsourcing framework. Centralized crowdsourcing frameworks complement workers’ reputations with predicted behavior, through Machine ...
Highlights- A behavior prediction ML model trained Off-chain and deployed on-chain for workers.






Comments