Abstract
Online experimentation with volunteers relies on participants' non-financial motivations to complete a study, such as to altruistically support science or to compare oneself to others. Researchers rely on these motivations to attract study participants and often use incentives, like performance comparisons, to encourage participation. Often, these study incentives are advertised using a slogan (e.g., "What is your thinking style?''). Research on framing effects suggests that advertisement slogans attract people with varying demographics and motivations. Could the slogan advertisements for studies risk attracting only specific users? To investigate the existence of potential sample biases, we measured how different slogan frames affected which participants self-selected into studies. We found that slogan frames impact recruitment significantly; changing the slogan frame from a 'supporting science' frame to a 'comparing oneself to others' frame lead to a 9% increase in recruitment for some studies. Additionally, slogans framed as learning more about oneself attract participants significantly more motivated by boredom compared to other slogan frames. We discuss design implications for using frames to improve recruitment and mitigate sources of sample bias in online research with volunteers.
- Adam L Alter and Daniel M Oppenheimer. 2009. Uniting the tribes of fluency to form a metacognitive nation. Personality and Social Psychology Review 13, 3 (2009), 219--235.Google Scholar
Cross Ref
- Judd Antin and Aaron Shaw. 2012. Social Desirability Bias and Self-reports of Motivation: A Study of Amazon Mechanical Turk in the US and India. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 2925--2934. Google Scholar
Digital Library
- Timothy C. Bednall and Liliana L. Bove. 2011. Donating Blood: A Meta-Analytic Review of Self-Reported Motivators and Deterrents. Transfusion Medicine Reviews 25, 4 (Oct 2011), 317--334.Google Scholar
Cross Ref
- Yoav Benjamini and Yosef Hochberg. 1995. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodological) 57 (1995), 289--300.Google Scholar
Cross Ref
- Jonah Berger and Katherine L Milkman. 2012. What Makes Online Content Viral? Journal of Marketing Research 49, 2 (apr 2012), 192--205. arXiv:arXiv:1011.1669v3Google Scholar
Cross Ref
- Adam J. Berinsky, Gregory A. Huber, and Gabriel S. Lenz. 2012. Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk.PoliticalAnalysis20,3(2012),351--368.Google Scholar
- RobertM.Bond,ChristopherJ.Fariss,JasonJ.Jones,AdamD.I.Kramer,CameronMarlow,JaimeE.Settle,andJamesH. Fowler. 2012. A 61-million-person experiment in social in uence and political mobilization. Nature 489, 7415 (sep 2012), 295--298.Google Scholar
Cross Ref
- Amber E Boydstun, Dallas Card, Justin Gross, Philip Resnik, and Noah A Smith. 2014. Tracking the development of media frames within and across policy issues. (2014).Google Scholar
- Paul R Brewer. 2003. Values, Political Knowledge, and Public Opinion about Gay Rights. Public Opinion Quaterly 67 (2003), 173--201.Google Scholar
Cross Ref
- Christopher J Bryan, Gregory M Walton, Todd Rogers, and Carol S Dweck. 2011. Motivating voter turnout by invoking the self. Proceedings of the National Academy of Sciences 108, 31 (August 2011), 12653--12656.Google Scholar
Cross Ref
- Dallas Card, Amber E Boydstun, Justin H Gross, Philip Resnik, and Noah A Smith. 2015. The media frames corpus: Annotations of frames across issues. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) 2, 438--444.Google Scholar
Cross Ref
- Chun Tuan Chang and Yu Kang Lee. 2009. Framing charity advertising: In uences of message framing, image valence, and temporal framing on a charitable appeal. Journal of Applied Social Psychology 39, 12 (2009), 2910--2935.Google Scholar
Cross Ref
- Dennis Chong and James N Druckman. 2007. Framing Theory. Annual Review of Political Science 10 (2007), 103--26.Google Scholar
Cross Ref
- Vickie Curtis. 2015. Motivation to Participate in an Online Citizen Science Game: A Study of Foldit. Science Communi- cation 37, 6 (Dec 2015), 723--746.Google Scholar
- J.M. Digman. 1990. Personality structure: The emergence of the ve-factor model. Annual Review of Psychology 41 (1990), 417--440. arXiv:arXiv:1011.1669v3Google Scholar
Cross Ref
- Robert M Entman. 1993. Framing: Toward clari cation of a fractured paradigm. Journal of communication 43, 4 (1993), 51--58.Google Scholar
Cross Ref
- Ronald Aylmer Fisher et al. 1924. The distribution of the partial correlation coe cient. Metron 3, 3--4 (1924), 329--332.Google Scholar
- Joseph L Fleiss. 1994. Measures of e ect size for categorical data. The handbook of research synthesis (1994), 245--260.Google Scholar
- Laura Germine, Ken Nakayama, Bradley C Duchaine, Christopher F Chabris, Garga Chatterjee, and Jeremy B Wilmer. 2012. Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments. Psychonomic Bulletin & Review 19, 5 (2012), 847--857.Google Scholar
Cross Ref
- Jorge Goncalves, Vassilis Kostakos, Evangelos Karapanos, Mary Barreto, Tiago Camacho, Anthony Tomasic, and John Zimmerman. 2014. Citizen motivation on the go: The role of psychological empowerment. Interacting with Computers 26, 3 (may 2014), 196--207.Google Scholar
Cross Ref
- Anja S Goritz. 2006. Incentives in web studies: Methodological issues and a review. International Journal of Internet Science 1, 1 (2006), 58--70. http://www.ijis.net/ijis1_1/ijis1_1_goeritz.pdfGoogle Scholar
- Donald P. Haider-Markel and Mark R Joslyn. 2001. Gun policy, opinion, tragedy, and blame attribution: The conditional influence of issue frames. Journal of Politics 63, 2 (2001), 520--543.Google Scholar
Cross Ref
- Joshua Hartshorne. 2018. Games With Words. https://www.gameswithwords.org/Google Scholar
- John R. Hauser, Glen L. Urban, Guilherme Liberali, and Michael Braun. 2009. Website Morphing. Marketing Science 28, 2 (mar 2009), 202--223. arXiv:1606.07988Google Scholar
Cross Ref
- Jos Hornikx and Daniel J. O'Keefe. 2009. Adapting Consumer Advertising Appeals to Cultural Values A Meta-Analytic Review of E ects on Persuasiveness and Ad Liking. Annals of the International Communication Association 33, 1 (jan 2009), 39--71.Google Scholar
Cross Ref
- Gary Hsieh and Rafal Kocielnik. 2016. You Get Who You Pay for: The Impact of Incentives on Participation Bias. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM Press, New York, New York, USA, 821--833. Google Scholar
Digital Library
- Susan Jamieson. 2004. Likert scales: how to (ab)use them. Medical Education 38 (2004), 1212--1218.Google Scholar
Cross Ref
- Eric Jones, Travis Oliphant, Pearu Peterson, et al. 2001--. SciPy: Open source scienti c tools for Python. http://www.scipy.org/ Online.Google Scholar
- Eunice Jun, Gary Hsieh, and Katharina Reinecke. 2017. Types of Motivation A ect Study Selection, Attention, and Dropouts in Online Experiments. Proceedings of the ACM on Human-Computer Interaction Archive 1, 1 (2017), 1--15. Google Scholar
Digital Library
- Nicolas Kaufmann, Thimo Schulze, and Daniel Veit. 2011. More than fun and money. Worker Motivation in Crowd-sourcing - A Study on Mechanical Turk. Proceedings of the Seventeenth Americas Conference on Information Systems 4, 2009 (2011), 1--11.Google Scholar
- B J Knowlton, L R Squire, and Mark A Gluck. 1994. Probabilistic classi cation learning in amnesia. Learning and Memory 1, 2 (1994), 106--120.Google Scholar
- Tae Kyoung Lee, Kevin Crowston, Mahboobeh Harandi, Carsten Østerlund, and Grant Miller. 2018. Appealing to di erent motivations in a message to recruit citizen scientists: results of a eld experiment. JCOM, Journal of Science Communication 17, 01 (2018), A02--2.Google Scholar
Cross Ref
- Irwin P Levin, Gj Gaeth, Judy Schreiber, and Marco Lauriola. 2002. A New Look at Framing E ects: Distribution of E ect Sizes, Individual Di erences, and Independence of Types of E ects. Organizational Behavior and Human Decision Processes 88, 1 (2002), 411--429.Google Scholar
Cross Ref
- Irwin P Levin and Gary J Gaeth. 1988. How Consumers are A ected by the Framing of Attribute Information Before and After Consuming the Product. Source Journal of Consumer Research 15, 3 (1988), 374--378. http://www.jstor.org/ stable/2489471Google Scholar
Cross Ref
- Qisheng Li, Krzysztof Z Gajos, Katharina Reinecke, and Paul G Allen. 2018. Volunteer-Based Online Studies With Older Adults and People with Disabilities. ACM Conference on Computers and Accessibility (ASSETS 2018). Google Scholar
Digital Library
- Yang Liu, Roy Chen, Yan Chen, Qiaozhu Mei, and Suzy Salib. 2012. 'I loan because...': understanding motivations for pro-social lending. In Proceedings of the fth ACM international conference on Web search and data mining - WSDM '12. ACM Press, New York, New York, USA, 503. Google Scholar
Digital Library
- Yong Liu, Denzil Ferreira, Jorge Goncalves, Simo Hosio, Pratyush Pandab, and Vassilis Kostakos. 2017. Donating context data to science: The e ects of social signals and perceptions on action-taking. Interacting with Computers 29, 2 (2017), 132--146.Google Scholar
- Winter Mason and Siddharth Suri. 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods 44, 1 (mar 2012), 1--23. arXiv:http://ssrn.com/abstract=1691163Google Scholar
- Merri eld, Colleen. 2014. Toward a Model of Boredom: Investigating the Psychophysiological, Cognitive, and Neural Correlates of Boredom. Ph.D. Dissertation. http://hdl.handle.net/10012/8671Google Scholar
- Geo Norman. 2010. Likert scales, levels of measurement and the "laws" of statistics. Advances in Health Sciences Education 15, 5 (dec 2010), 625--632.Google Scholar
Cross Ref
- Oded Nov, Ofer Arazy, and David Anderson. 2011. Dusting for science. In Proceedings of the 2011 iConference on - iConference '11. ACM Press, New York, New York, USA, 68--74. Google Scholar
Digital Library
- Oded Nov, Ofer Arazy, and David Anderson. 2014. [email protected]: What Drives the Quantity and Quality of Online Citizen Science Participation? PLoS ONE 9, 4 (apr 2014), e90375.Google Scholar
Cross Ref
- M. L. Ohmer. 2007. Citizen Participation in Neighborhood Organizations and Its Relationship to Volunteers' Self- and Collective E cacy and Sense of Community. Social Work Research 31, 2 (jun 2007), 109--120.Google Scholar
Cross Ref
- Godfrey Pell. 2005. Use and misuse of Likert scales. Medical Education 39, 9 (sep 2005), 970--970.Google Scholar
Cross Ref
- ProjectImplicit. 2011. ProjectImplicit. https://www.projectimplicit.net/index.htmlGoogle Scholar
- M Jordan Raddick, Georgia Bracey, Pamela L Gay, Chris J Lintott, Carie Cardamone, Phil Murray, Kevin Schawinski, Alexander S Szalay, and Jan Vandenberg. 2013. Galaxy Zoo: Motivations of citizen scientists. arXiv preprint arXiv:1303.6886 (2013).Google Scholar
- Katharina Reinecke and Krzysztof Z Gajos. 2015. LabintheWild: Conducting large-scale online experiments with uncompensated samples. Proceedings of the 18th ACM conference on Computer Supported Cooperative Work & Social Computing (2015), 1364--1378. Google Scholar
Digital Library
- Amber Marie Reinhart, Heather M Marshall, Thomas Hugh Feeley, and Frank Tutzauer. 2007. The Persuasive E ects of Message Framing in Organ Donation: The Mediating Role of Psychological Reactance. Communication Monographs ISSN:74,2(2007),229--255.Google Scholar
Cross Ref
- UD Reips, B Batinic, W Bandilla, M Bosnjak, and L Graf. 1999. Financial incentives personal information and drop-out rate in online studies. In Current Internet Science Trends, Techniques, Results. Deutsche Gesellschaft fur Online-Forschung.Google Scholar
- Joel Ross, Andrew Zaldivar, Lilly Irani, and Bill Tomlinson. 2010. Who are the Turkers? Worker Demographics in Amazon Mechanical Turk. Human Factors in Computing Systems (CHI) (2010), 2863--2872. Google Scholar
Digital Library
- Josef Ruppenhofer, Michael Ellsworth, Miriam RL Petruck, Christopher R Johnson, and Jan Sche czyk. 2016. FrameNet II: Extended theory and practice. Institut fur Deutsche Sprache, Bibliothek.Google Scholar
- J.S. Seabold and J. Perktold. 2010. Statsmodels: Econometric and Statistical Modeling with Python. In Proceedings of the 9th Python in Science Conference.Google Scholar
- Nihar B Shah and Martin J Wainwright. 2015. Simple, Robust and Optimal Ranking from Pairwise Comparisons. arXiv (2015), 1--17. arXiv:1512.08949 https://arxiv.org/pdf/1512.08949.pdfhttp://arxiv.org/abs/1512.08949Google Scholar
- Steven J. Sherman, Diane M. Mackie, and Denise M. Driscoll. 1990. Priming and the Di erential Use of Dimen- sions in Evaluation. Personality and Social Psychology Bulletin 16, 3 (sep 1990), 405--418. arXiv:0803973233Google Scholar
Cross Ref
- P.M. Sniderman and S.M. Theriault. 2004. The Structure of Political Argument and the Logic of Issue Framing. Studies in Public Opinion: Attitudes, onattitudes, Measurement Error, and Change (jan 2004), 133--164.Google Scholar
- Keiran Snyder. 2017. 1000 di erent people, the same words - Textio Word Nerd. https://textio.ai/1000-di erent-people-the-same-words-6149b5a1f351Google Scholar
- Gail M Sullivan and Richard Feinn. 2012. Using E ect Size-or Why the P Value Is Not Enough. Journal of graduate medical education 4, 3 (sep 2012), 279--82.Google Scholar
Cross Ref
- Chenhao Tan, Lillian Lee, and Bo Pang. 2014. The e ect of wording on message propagation: Topic-and author- controlled natural experiments on Twitter. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL'14) (2014).Google Scholar
- TestMyBrain. 2017. TestMyBrain. http://dx.plos.org/10.1371/journal.pone.0165100Google Scholar
- Amos Tversky and Daniel Kahneman. 1981. The framing of decisions and the psychology of choice. Science 211, 4481 (1981), 453--458.Google Scholar
- Glen L. Urban, Guilherme (Gui) Liberali, Erin MacDonald, Robert Bordley, and John R. Hauser. 2014. Morphing Banner Advertising. Marketing Science 33, 1 (jan 2014), 27--46.Google Scholar
Cross Ref
- VolunteerScience. 2018. Volunteer Science. https://volunteerscience.com/Google Scholar
- Heng Li Yang and Cheng Yu Lai. 2010. Motivations of Wikipedia content contributors. Computers in Human Behavior 26, 6 (nov 2010), 1377--1383. Google Scholar
Digital Library
- Marc A. Zimmerman. 1995. Psychological empowerment: Issues and illustrations. American Journal of Community Psychology 23, 5 (oct 1995), 581--599.Google Scholar
Cross Ref
Index Terms
Framing Effects: Choice of Slogans Used to Advertise Online Experiments Can Boost Recruitment and Lead to Sample Biases
Recommendations
Advertising for Cognitive Interviews
Researchers commonly rely on relatively small convenience samples for cognitive pretesting questionnaires. Methods used to recruit these samples vary depending on the population of interest, study timeline, study budget, and other factors. Over the past ...
Naturalistic decision framing in computer mediated scientific exploration
NDM'09: Proceedings of the 9th Bi-annual international conference on Naturalistic Decision MakingMotivation - This study was conducted to determine how people frame decisions naturally. Research Approach - Decision framing was analyzed in the computer mediated Mars Exploration Rover (MER) mission planning process. Outcomes of proposed actions were ...
Debiasing the framing effect: The effect of warning and involvement
This study reports on an experiment that examined the effect of message framing on the Internet buyers' attitude and purchase intention. The effect of two possible moderators in framing effect including warning about the possibilities of bias and the ...






Comments