TurKit: human computation algorithms on mechanical turk

ABSTRACT

Mechanical Turk (MTurk) provides an on-demand source of human computation. This provides a tremendous opportunity to explore algorithms which incorporate human computation as a function call. However, various systems challenges make this difficult in practice, and most uses of MTurk post large numbers of independent tasks. TurKit is a toolkit for prototyping and exploring algorithmic human computation, while maintaining a straight-forward imperative programming style. We present the crash-and-rerun programming model that makes TurKit possible, along with a variety of applications for human computation algorithms. We also present case studies of TurKit used for real experiments across different fields.

References

1. }}von Ahn, L. Games With A Purpose. IEEE Computer Magazine, June 2006. Pages 96--98.
2. }}von Ahn, L. and Dabbish, L. Labeling Images with a Computer Game. ACM Conference on Human Factors in Computing Systems, CHI 2004.
3. }}von Ahn, L., Maurer, B., McMillen, C., Abraham, D. and Blum, M. reCAPTCHA: Human-Based Character Recognition via Web Security Measures. Science, September 12, 2008. pp 1465--1468.
4. }}Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., Panovich, K. "Soylent: A Word Processor with a Crowd Inside". UIST 2010.
5. }}Bryant, S. L., et al. Becoming Wikipedian: transformation of participation in a collaborative online encyclopedia. GROUP 2005.
6. }}Cornsweet, T. N. The Staircase-Method in Psychophysics. The American Journal of Psychology, Vol. 75, No. 3 (Sep., 1962), pp. 485--49.
7. }}Dai, P., Mausam, Weld, D. S. Decision-Theoretic Control of Crowd-Sourced Workflows. AAAI 2010.
8. }}Dia, M. A. "On Decision Making in Tandem Networks". M. Eng. Thesis. Massachusetts Institute of Technology. 2009.
9. }}Feldman, S. I. and Brown, C. B. IGOR: a system for program debugging via reversible execution. Proc. ACM SIGPLAN Workshop on Parallel and Distributed Debugging. 1988.
10. }}Heer, J., Bostock, M. Crowdsourcing Graphical Perception: Using MTurk to Assess Visualization Design. CHI 2010.
11. }}Kittur, A., Chi, E. H., and Suh, B. Crowdsourcing user studies with MTurk. CHI 2008.
12. }}Kittur, A. and Kraut, R. E. Harnessing the wisdom of crowds in wikipedia: quality through coordination. CSCW 2008.
13. }}Ko, A. J. and Myers, B. A. Finding causes of program output with the Java Whyline. CHI 2009.
14. }}Kosorukoff A. Human based genetic algorithm. IlliGAL report no. 2001004. UIUC, 2001.
15. }}Little, G., Chilton, L. B., Goldman, M. and Miller, R. C. Exploring Iterative and Parallel Human Computation Processes. KDD-HCOMP 2010.
16. }}Mason, W. and Watts, D. J. Financial incentives and the "performance of crowds". KDD-HCOMP 2009.
17. }}Quinn, A. J., Bederson, B. B. A Taxonomy of Distributed Human Computation. Technical Report HCIL-2009-23. University of Maryland. 2009.
18. }}Russell, B., Torralba, A., Murphy, K., Freeman, W. LabelMe: a database and web-based tool for image annotation. International Journal of Computer Vision, Vol. 77, No. 1 (1 May 2008), pp. 157--173.
19. }}Snow, R., O'Connor, B., Jurafsky, D., and Ng, A. Y. Cheap and fast - but is it good?: evaluating non-expert annotations for natural language tasks. EMNLP 2008.
20. }}Sorokin, A. and D. Forsyth, "Utility data annotation with Amazon MTurk". CVPR 2008.

Index Terms

• Human-centered computing

  • Interaction design

    • Interaction design process and methods

      • Interface design prototyping