Abstract
Surveys can be viewed as programs, complete with logic, control flow, and bugs. Word choice or the order in which questions are asked can unintentionally bias responses. Vague, confusing, or intrusive questions can cause respondents to abandon a survey. Surveys can also have runtime errors: inattentive respondents can taint results. This effect is especially problematic when deploying surveys in uncontrolled settings, such as on the web or via crowdsourcing platforms. Because the results of surveys drive business decisions and inform scientific conclusions, it is crucial to make sure they are correct.
We present SurveyMan, a system for designing, deploying, and automatically debugging surveys. Survey authors write their surveys in a lightweight domain-specific language aimed at end users. SurveyMan statically analyzes the survey to provide feedback to survey authors before deployment. It then compiles the survey into JavaScript and deploys it either to the web or a crowdsourcing platform. SurveyMan's dynamic analyses automatically find survey bugs, and control for the quality of responses. We evaluate SurveyMan's algorithms analytically and empirically, demonstrating its effectiveness with case studies of social science surveys conducted via Amazon's Mechanical Turk.
Supplemental Material
Available for Download
SurveyMan: Programming and Automatically Debugging Surveys
- M. Barclay, W. Lober, and B. Karras. SuML: A survey markup language for generalized survey encoding. In Proceedings of the AMIA Symposium, page 970. American Medical Informatics Association, 2002.Google Scholar
- D. W. Barowy, C. Curtsinger, E. D. Berger, and A. McGregor. AUTOMAN: A platform for integrating human-based and digital computation. In Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications, OOPSLA '12, pages 639--654, New York, NY, USA, 2012. ACM. Google Scholar
Digital Library
- A. J. Berinsky, G. A. Huber, and G. S. Lenz. Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Political Analysis, 20(3):351--368, 2012.Google Scholar
Cross Ref
- G. A. Churchill Jr and D. Iacobucci. Marketing research: methodological foundations. Cengage Learning, 2009.Google Scholar
- A. M. Colman. A dictionary of psychology. Oxford University Press, 2009.Google Scholar
- M. P. Couper. Designing Effective Web Surveys. Cambridge University Press, New York, NY, USA, 1st edition, 2008. Google Scholar
Digital Library
- D. De Vaus. Surveys in social research. Psychology Press, 2002.Google Scholar
- E. W. Dijkstra. Guarded commands, nondeterminacy and formal derivation of programs. Commun. ACM, 18(8):453--457, Aug. 1975. Google Scholar
Digital Library
- D. A. Dillman. Mail and telephone surveys, volume 3. Wiley New York, 1978.Google Scholar
- J. S. Downs, M. B. Holbrook, S. Sheng, and L. F. Cranor. Are your participants gaming the system?: Screening Mechanical Turk workers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '10, pages 2399--2402, New York, NY, USA, 2010. ACM. Google Scholar
Digital Library
- G. Emanuel. Post A Survey On Mechanical Turk And Watch The Results Roll In: All Tech Considered: NPR. http://n.pr/1gqklTx, Mar. 2014.Google Scholar
- I. Google. Google consumer surveys. http://www.google.com/insights/consumersurveys/home, 2013.Google Scholar
- J. J. Horton, D. G. Rand, and R. J. Zeckhauser. The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14(3):399--425, 2011.Google Scholar
Cross Ref
- J. L. Huang, P. G. Curran, J. Keeney, E. M. Poposki, and R. P. DeShon. Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1):99--114, 2012.Google Scholar
Cross Ref
- Instant.ly. Instant.ly. http://instant.ly, 2013.Google Scholar
- P. Ipeirotis. Demographics of Mechanical Turk. Technical Report NYU working paper no. CEDER-10-01, 2010.Google Scholar
- S. C. Kingsley. Personal Communication, August 2014.Google Scholar
- J. A. Krosnick. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied cognitive psychology, 5(3):213--236, 1991.Google Scholar
- M. MacHenry and J. Matthews. Topsl: A domain-specific language for on-line surveys. In O. Shivers and O. Waddell, editors, Proceedings of the Fifth ACM SIGPLAN Workshop on Scheme and Functional Programming, pages 33--39, Snowbird, Utah, Sept. 22, 2004. Technical report TR600, Department of Computer Science, Indiana University. http://www.cs.indiana.edu/cgi-bin/techreports/TRNNN.cgi?trnum=TR600.Google Scholar
- G. MacKerron. Implementation, implementation, implementation: Old and new options for putting surveys and experiments online. Journal of Choice Modelling, 4:20--48, 2011.Google Scholar
Cross Ref
- E. Martin. Survey questionnaire construction. Technical Report Survey Methodology #2006-13, Director's Office, U.S. Census Bureau, 2006.Google Scholar
- W. Mason and S. Suri. Conducting behavioral research on Amazon's Mechanical Turk. Behavior research methods, 44(1):1--23, 2012.Google Scholar
- A. S. McKay. Improving data quality with four short sentences: How an honor code can make the difference during data collection. 2014.Google Scholar
- A. W. Meade and S. B. Craig. Identifying careless responses in survey data. Psychological methods, 17(3):437, 2012.Google Scholar
- S. Netherlands. Blaise: Survey software for professionals. http://www.blaise.com/ShortIntroduction, 2013.Google Scholar
- D. M. Oppenheimer, T. Meyvis, and N. Davidenko. Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4):867--872, 2009.Google Scholar
- Pew Research Center. Question Order | Pew Research Center for the People and the Press. http://www.people-press.org/methodology/questionnaire-design/question-order/, 2014.Google Scholar
- Pew Research Center. Question Wording | Pew Research Center for the People and the Press. http://www.people-press.org/methodology/questionnaire-design/question-wording/, 2014.Google Scholar
- A. Peytchev. Survey breakoff. Public Opinion Quarterly, 73(1):74--97, 2009.Google Scholar
Cross Ref
- P. Pizzo. Personal Communication, August 2014.Google Scholar
- I. Qualtrics. Qualtrics.com. http://qualtrics.com, 2013.Google Scholar
- J. P. Robinson-Cimpian. Inaccurate estimation of disparities due to mischievous responders several suggestions to assess conclusions. Educational Researcher, 43(4):171--185, 2014.Google Scholar
Cross Ref
- SocialSci. Cambridge, ma, usa. http://www.socialsci.com, 2014.Google Scholar
- D. J. Solomon. Conducting web-based surveys, August 2001.Google Scholar
- S. Spencer. Canard question module editor. https://github.com/LegoStormtroopr/canard, 2013.Google Scholar
- S. Spencer. A case against the skip statement, 2013. Unpublished.Google Scholar
- S. Spencer. The simple questionnaire building language, 2013.Google Scholar
- S. S. Stevens. On the theory of scales of measurement. 1946.Google Scholar
Cross Ref
- L. Survey. Lime survey 2.05. https://www.limesurvey.org/en/, 2014.Google Scholar
- L. SurveyGizmo. Surveygizmo. http://http://www.surveygizmo.com/, 2014.Google Scholar
- SurveyMonkey, Inc. Surveymonkey. http://surveymonkey.com, 2013.Google Scholar
- R. Tourangeau, F. Conrad, and M. Couper. The Science of Web Surveys. Oxford University Press, 2013.Google Scholar
Cross Ref
- P. D. Umbach. Web surveys: Best practices. New Directions for Institutional Research, 2004(121):23--38, 2004.Google Scholar
Cross Ref
- U.S. Government Accountability Office. Questionnaire programming language. http://qpl.gao.gov/qpl6ref/01.php, 2009.Google Scholar
- D. Zhu and B. Carterette. An analysis of assessor behavior in crowdsourced preference judgments. In Proceedings of the SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (CSE 2010), pages 21--26, 2010.Google Scholar
Index Terms
SurveyMan: programming and automatically debugging surveys
Recommendations
SurveyMan: programming and automatically debugging surveys
OOPSLA '14: Proceedings of the 2014 ACM International Conference on Object Oriented Programming Systems Languages & ApplicationsSurveys can be viewed as programs, complete with logic, control flow, and bugs. Word choice or the order in which questions are asked can unintentionally bias responses. Vague, confusing, or intrusive questions can cause respondents to abandon a survey. ...
Neverlang and FeatureIDE just married: integrated language product line development environment
SPLC '20: Proceedings of the 24th ACM Conference on Systems and Software Product Line: Volume A - Volume ALanguage development is inherently complex. With the support of a suitable language development environment most computer scientists could develop their own domain-specific language (DSL) with relative ease. Yet, when the DSL is the result of a ...







Comments