skip to main content
research-article

Framing Effects: Choice of Slogans Used to Advertise Online Experiments Can Boost Recruitment and Lead to Sample Biases

Authors Info & Claims
Published:01 November 2018Publication History
Skip Abstract Section

Abstract

Online experimentation with volunteers relies on participants' non-financial motivations to complete a study, such as to altruistically support science or to compare oneself to others. Researchers rely on these motivations to attract study participants and often use incentives, like performance comparisons, to encourage participation. Often, these study incentives are advertised using a slogan (e.g., "What is your thinking style?''). Research on framing effects suggests that advertisement slogans attract people with varying demographics and motivations. Could the slogan advertisements for studies risk attracting only specific users? To investigate the existence of potential sample biases, we measured how different slogan frames affected which participants self-selected into studies. We found that slogan frames impact recruitment significantly; changing the slogan frame from a 'supporting science' frame to a 'comparing oneself to others' frame lead to a 9% increase in recruitment for some studies. Additionally, slogans framed as learning more about oneself attract participants significantly more motivated by boredom compared to other slogan frames. We discuss design implications for using frames to improve recruitment and mitigate sources of sample bias in online research with volunteers.

References

  1. Adam L Alter and Daniel M Oppenheimer. 2009. Uniting the tribes of fluency to form a metacognitive nation. Personality and Social Psychology Review 13, 3 (2009), 219--235.Google ScholarGoogle ScholarCross RefCross Ref
  2. Judd Antin and Aaron Shaw. 2012. Social Desirability Bias and Self-reports of Motivation: A Study of Amazon Mechanical Turk in the US and India. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 2925--2934. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Timothy C. Bednall and Liliana L. Bove. 2011. Donating Blood: A Meta-Analytic Review of Self-Reported Motivators and Deterrents. Transfusion Medicine Reviews 25, 4 (Oct 2011), 317--334.Google ScholarGoogle ScholarCross RefCross Ref
  4. Yoav Benjamini and Yosef Hochberg. 1995. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodological) 57 (1995), 289--300.Google ScholarGoogle ScholarCross RefCross Ref
  5. Jonah Berger and Katherine L Milkman. 2012. What Makes Online Content Viral? Journal of Marketing Research 49, 2 (apr 2012), 192--205. arXiv:arXiv:1011.1669v3Google ScholarGoogle ScholarCross RefCross Ref
  6. Adam J. Berinsky, Gregory A. Huber, and Gabriel S. Lenz. 2012. Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk.PoliticalAnalysis20,3(2012),351--368.Google ScholarGoogle Scholar
  7. RobertM.Bond,ChristopherJ.Fariss,JasonJ.Jones,AdamD.I.Kramer,CameronMarlow,JaimeE.Settle,andJamesH. Fowler. 2012. A 61-million-person experiment in social in uence and political mobilization. Nature 489, 7415 (sep 2012), 295--298.Google ScholarGoogle ScholarCross RefCross Ref
  8. Amber E Boydstun, Dallas Card, Justin Gross, Philip Resnik, and Noah A Smith. 2014. Tracking the development of media frames within and across policy issues. (2014).Google ScholarGoogle Scholar
  9. Paul R Brewer. 2003. Values, Political Knowledge, and Public Opinion about Gay Rights. Public Opinion Quaterly 67 (2003), 173--201.Google ScholarGoogle ScholarCross RefCross Ref
  10. Christopher J Bryan, Gregory M Walton, Todd Rogers, and Carol S Dweck. 2011. Motivating voter turnout by invoking the self. Proceedings of the National Academy of Sciences 108, 31 (August 2011), 12653--12656.Google ScholarGoogle ScholarCross RefCross Ref
  11. Dallas Card, Amber E Boydstun, Justin H Gross, Philip Resnik, and Noah A Smith. 2015. The media frames corpus: Annotations of frames across issues. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) 2, 438--444.Google ScholarGoogle ScholarCross RefCross Ref
  12. Chun Tuan Chang and Yu Kang Lee. 2009. Framing charity advertising: In uences of message framing, image valence, and temporal framing on a charitable appeal. Journal of Applied Social Psychology 39, 12 (2009), 2910--2935.Google ScholarGoogle ScholarCross RefCross Ref
  13. Dennis Chong and James N Druckman. 2007. Framing Theory. Annual Review of Political Science 10 (2007), 103--26.Google ScholarGoogle ScholarCross RefCross Ref
  14. Vickie Curtis. 2015. Motivation to Participate in an Online Citizen Science Game: A Study of Foldit. Science Communi- cation 37, 6 (Dec 2015), 723--746.Google ScholarGoogle Scholar
  15. J.M. Digman. 1990. Personality structure: The emergence of the ve-factor model. Annual Review of Psychology 41 (1990), 417--440. arXiv:arXiv:1011.1669v3Google ScholarGoogle ScholarCross RefCross Ref
  16. Robert M Entman. 1993. Framing: Toward clari cation of a fractured paradigm. Journal of communication 43, 4 (1993), 51--58.Google ScholarGoogle ScholarCross RefCross Ref
  17. Ronald Aylmer Fisher et al. 1924. The distribution of the partial correlation coe cient. Metron 3, 3--4 (1924), 329--332.Google ScholarGoogle Scholar
  18. Joseph L Fleiss. 1994. Measures of e ect size for categorical data. The handbook of research synthesis (1994), 245--260.Google ScholarGoogle Scholar
  19. Laura Germine, Ken Nakayama, Bradley C Duchaine, Christopher F Chabris, Garga Chatterjee, and Jeremy B Wilmer. 2012. Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments. Psychonomic Bulletin & Review 19, 5 (2012), 847--857.Google ScholarGoogle ScholarCross RefCross Ref
  20. Jorge Goncalves, Vassilis Kostakos, Evangelos Karapanos, Mary Barreto, Tiago Camacho, Anthony Tomasic, and John Zimmerman. 2014. Citizen motivation on the go: The role of psychological empowerment. Interacting with Computers 26, 3 (may 2014), 196--207.Google ScholarGoogle ScholarCross RefCross Ref
  21. Anja S Goritz. 2006. Incentives in web studies: Methodological issues and a review. International Journal of Internet Science 1, 1 (2006), 58--70. http://www.ijis.net/ijis1_1/ijis1_1_goeritz.pdfGoogle ScholarGoogle Scholar
  22. Donald P. Haider-Markel and Mark R Joslyn. 2001. Gun policy, opinion, tragedy, and blame attribution: The conditional influence of issue frames. Journal of Politics 63, 2 (2001), 520--543.Google ScholarGoogle ScholarCross RefCross Ref
  23. Joshua Hartshorne. 2018. Games With Words. https://www.gameswithwords.org/Google ScholarGoogle Scholar
  24. John R. Hauser, Glen L. Urban, Guilherme Liberali, and Michael Braun. 2009. Website Morphing. Marketing Science 28, 2 (mar 2009), 202--223. arXiv:1606.07988Google ScholarGoogle ScholarCross RefCross Ref
  25. Jos Hornikx and Daniel J. O'Keefe. 2009. Adapting Consumer Advertising Appeals to Cultural Values A Meta-Analytic Review of E ects on Persuasiveness and Ad Liking. Annals of the International Communication Association 33, 1 (jan 2009), 39--71.Google ScholarGoogle ScholarCross RefCross Ref
  26. Gary Hsieh and Rafal Kocielnik. 2016. You Get Who You Pay for: The Impact of Incentives on Participation Bias. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM Press, New York, New York, USA, 821--833. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Susan Jamieson. 2004. Likert scales: how to (ab)use them. Medical Education 38 (2004), 1212--1218.Google ScholarGoogle ScholarCross RefCross Ref
  28. Eric Jones, Travis Oliphant, Pearu Peterson, et al. 2001--. SciPy: Open source scienti c tools for Python. http://www.scipy.org/ Online.Google ScholarGoogle Scholar
  29. Eunice Jun, Gary Hsieh, and Katharina Reinecke. 2017. Types of Motivation A ect Study Selection, Attention, and Dropouts in Online Experiments. Proceedings of the ACM on Human-Computer Interaction Archive 1, 1 (2017), 1--15. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Nicolas Kaufmann, Thimo Schulze, and Daniel Veit. 2011. More than fun and money. Worker Motivation in Crowd-sourcing - A Study on Mechanical Turk. Proceedings of the Seventeenth Americas Conference on Information Systems 4, 2009 (2011), 1--11.Google ScholarGoogle Scholar
  31. B J Knowlton, L R Squire, and Mark A Gluck. 1994. Probabilistic classi cation learning in amnesia. Learning and Memory 1, 2 (1994), 106--120.Google ScholarGoogle Scholar
  32. Tae Kyoung Lee, Kevin Crowston, Mahboobeh Harandi, Carsten Østerlund, and Grant Miller. 2018. Appealing to di erent motivations in a message to recruit citizen scientists: results of a eld experiment. JCOM, Journal of Science Communication 17, 01 (2018), A02--2.Google ScholarGoogle ScholarCross RefCross Ref
  33. Irwin P Levin, Gj Gaeth, Judy Schreiber, and Marco Lauriola. 2002. A New Look at Framing E ects: Distribution of E ect Sizes, Individual Di erences, and Independence of Types of E ects. Organizational Behavior and Human Decision Processes 88, 1 (2002), 411--429.Google ScholarGoogle ScholarCross RefCross Ref
  34. Irwin P Levin and Gary J Gaeth. 1988. How Consumers are A ected by the Framing of Attribute Information Before and After Consuming the Product. Source Journal of Consumer Research 15, 3 (1988), 374--378. http://www.jstor.org/ stable/2489471Google ScholarGoogle ScholarCross RefCross Ref
  35. Qisheng Li, Krzysztof Z Gajos, Katharina Reinecke, and Paul G Allen. 2018. Volunteer-Based Online Studies With Older Adults and People with Disabilities. ACM Conference on Computers and Accessibility (ASSETS 2018). Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Yang Liu, Roy Chen, Yan Chen, Qiaozhu Mei, and Suzy Salib. 2012. 'I loan because...': understanding motivations for pro-social lending. In Proceedings of the fth ACM international conference on Web search and data mining - WSDM '12. ACM Press, New York, New York, USA, 503. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Yong Liu, Denzil Ferreira, Jorge Goncalves, Simo Hosio, Pratyush Pandab, and Vassilis Kostakos. 2017. Donating context data to science: The e ects of social signals and perceptions on action-taking. Interacting with Computers 29, 2 (2017), 132--146.Google ScholarGoogle Scholar
  38. Winter Mason and Siddharth Suri. 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods 44, 1 (mar 2012), 1--23. arXiv:http://ssrn.com/abstract=1691163Google ScholarGoogle Scholar
  39. Merri eld, Colleen. 2014. Toward a Model of Boredom: Investigating the Psychophysiological, Cognitive, and Neural Correlates of Boredom. Ph.D. Dissertation. http://hdl.handle.net/10012/8671Google ScholarGoogle Scholar
  40. Geo Norman. 2010. Likert scales, levels of measurement and the "laws" of statistics. Advances in Health Sciences Education 15, 5 (dec 2010), 625--632.Google ScholarGoogle ScholarCross RefCross Ref
  41. Oded Nov, Ofer Arazy, and David Anderson. 2011. Dusting for science. In Proceedings of the 2011 iConference on - iConference '11. ACM Press, New York, New York, USA, 68--74. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Oded Nov, Ofer Arazy, and David Anderson. 2014. [email protected]: What Drives the Quantity and Quality of Online Citizen Science Participation? PLoS ONE 9, 4 (apr 2014), e90375.Google ScholarGoogle ScholarCross RefCross Ref
  43. M. L. Ohmer. 2007. Citizen Participation in Neighborhood Organizations and Its Relationship to Volunteers' Self- and Collective E cacy and Sense of Community. Social Work Research 31, 2 (jun 2007), 109--120.Google ScholarGoogle ScholarCross RefCross Ref
  44. Godfrey Pell. 2005. Use and misuse of Likert scales. Medical Education 39, 9 (sep 2005), 970--970.Google ScholarGoogle ScholarCross RefCross Ref
  45. ProjectImplicit. 2011. ProjectImplicit. https://www.projectimplicit.net/index.htmlGoogle ScholarGoogle Scholar
  46. M Jordan Raddick, Georgia Bracey, Pamela L Gay, Chris J Lintott, Carie Cardamone, Phil Murray, Kevin Schawinski, Alexander S Szalay, and Jan Vandenberg. 2013. Galaxy Zoo: Motivations of citizen scientists. arXiv preprint arXiv:1303.6886 (2013).Google ScholarGoogle Scholar
  47. Katharina Reinecke and Krzysztof Z Gajos. 2015. LabintheWild: Conducting large-scale online experiments with uncompensated samples. Proceedings of the 18th ACM conference on Computer Supported Cooperative Work & Social Computing (2015), 1364--1378. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Amber Marie Reinhart, Heather M Marshall, Thomas Hugh Feeley, and Frank Tutzauer. 2007. The Persuasive E ects of Message Framing in Organ Donation: The Mediating Role of Psychological Reactance. Communication Monographs ISSN:74,2(2007),229--255.Google ScholarGoogle ScholarCross RefCross Ref
  49. UD Reips, B Batinic, W Bandilla, M Bosnjak, and L Graf. 1999. Financial incentives personal information and drop-out rate in online studies. In Current Internet Science Trends, Techniques, Results. Deutsche Gesellschaft fur Online-Forschung.Google ScholarGoogle Scholar
  50. Joel Ross, Andrew Zaldivar, Lilly Irani, and Bill Tomlinson. 2010. Who are the Turkers? Worker Demographics in Amazon Mechanical Turk. Human Factors in Computing Systems (CHI) (2010), 2863--2872. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Josef Ruppenhofer, Michael Ellsworth, Miriam RL Petruck, Christopher R Johnson, and Jan Sche czyk. 2016. FrameNet II: Extended theory and practice. Institut fur Deutsche Sprache, Bibliothek.Google ScholarGoogle Scholar
  52. J.S. Seabold and J. Perktold. 2010. Statsmodels: Econometric and Statistical Modeling with Python. In Proceedings of the 9th Python in Science Conference.Google ScholarGoogle Scholar
  53. Nihar B Shah and Martin J Wainwright. 2015. Simple, Robust and Optimal Ranking from Pairwise Comparisons. arXiv (2015), 1--17. arXiv:1512.08949 https://arxiv.org/pdf/1512.08949.pdfhttp://arxiv.org/abs/1512.08949Google ScholarGoogle Scholar
  54. Steven J. Sherman, Diane M. Mackie, and Denise M. Driscoll. 1990. Priming and the Di erential Use of Dimen- sions in Evaluation. Personality and Social Psychology Bulletin 16, 3 (sep 1990), 405--418. arXiv:0803973233Google ScholarGoogle ScholarCross RefCross Ref
  55. P.M. Sniderman and S.M. Theriault. 2004. The Structure of Political Argument and the Logic of Issue Framing. Studies in Public Opinion: Attitudes, onattitudes, Measurement Error, and Change (jan 2004), 133--164.Google ScholarGoogle Scholar
  56. Keiran Snyder. 2017. 1000 di erent people, the same words - Textio Word Nerd. https://textio.ai/1000-di erent-people-the-same-words-6149b5a1f351Google ScholarGoogle Scholar
  57. Gail M Sullivan and Richard Feinn. 2012. Using E ect Size-or Why the P Value Is Not Enough. Journal of graduate medical education 4, 3 (sep 2012), 279--82.Google ScholarGoogle ScholarCross RefCross Ref
  58. Chenhao Tan, Lillian Lee, and Bo Pang. 2014. The e ect of wording on message propagation: Topic-and author- controlled natural experiments on Twitter. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL'14) (2014).Google ScholarGoogle Scholar
  59. TestMyBrain. 2017. TestMyBrain. http://dx.plos.org/10.1371/journal.pone.0165100Google ScholarGoogle Scholar
  60. Amos Tversky and Daniel Kahneman. 1981. The framing of decisions and the psychology of choice. Science 211, 4481 (1981), 453--458.Google ScholarGoogle Scholar
  61. Glen L. Urban, Guilherme (Gui) Liberali, Erin MacDonald, Robert Bordley, and John R. Hauser. 2014. Morphing Banner Advertising. Marketing Science 33, 1 (jan 2014), 27--46.Google ScholarGoogle ScholarCross RefCross Ref
  62. VolunteerScience. 2018. Volunteer Science. https://volunteerscience.com/Google ScholarGoogle Scholar
  63. Heng Li Yang and Cheng Yu Lai. 2010. Motivations of Wikipedia content contributors. Computers in Human Behavior 26, 6 (nov 2010), 1377--1383. Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. Marc A. Zimmerman. 1995. Psychological empowerment: Issues and illustrations. American Journal of Community Psychology 23, 5 (oct 1995), 581--599.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Framing Effects: Choice of Slogans Used to Advertise Online Experiments Can Boost Recruitment and Lead to Sample Biases

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!