Standardizing Reporting of Participant Compensation in HCI: A Systematic Literature Review and Recommendations for the Field

ABSTRACT

The user study is a fundamental method used in HCI. In designing user studies, we often use compensation strategies to incentivize recruitment. However, compensation can also lead to ethical issues, such as coercion. The CHI community has yet to establish best practices for participant compensation. Through a systematic review of manuscripts at CHI and other associated publication venues, we found high levels of variation in the compensation strategies used within the community and how we report on this aspect of the study methods. A qualitative analysis of justifications offered for compensation sheds light into how some researchers are currently contextualizing this practice. This paper provides a description of current compensation strategies and information that can inform the design of compensation strategies in future studies. The findings may be helpful to generate productive discourse in the HCI community towards the development of best practices for participant compensation in user studies.

References

1. 1988. Visualization in Human-Computer Interaction (th interdisciplinary workshop on informatics and psychology, schärding, austria, may 24-27, 1988 ed.). Springer: Lecture Notes in Computer Science.
2. Abdullah X. Ali, Meredith Ringel Morris, and Jacob O. Wobbrock. 2018. Crowdsourcing Similarity Judgments for Agreement Analysis in End-User Elicitation Studies. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST ’18). Association for Computing Machinery, New York, NY, USA, 177–188. https://doi.org/10.1145/3242587.3242621
3. Mashael Yousef Almoqbel and Donghee Yvette Wohn. 2019. Individual and Collaborative Behaviors of Rideshare Drivers in Protecting Their Safety. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 217 (Nov. 2019), 21 pages. https://doi.org/10.1145/3359319
4. Nabil Alshurafa, Jayalakshmi Jain, Rawan Alharbi, Gleb Iakovlev, Bonnie Spring, and Angela Pfammatter. 2018. Is More Always Better? Discovering Incentivized MHealth Intervention Engagement Related to Health Behavior Trends. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 4, Article 153 (Dec. 2018), 26 pages. https://doi.org/10.1145/3287031
5. Monya Baker. 2016, July 28. 1,500 scientists lift the lid on reproducibility. https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970
6. Kathleen Marie Beck. 2019. Academic researcher decision making processes for research participant compensation. Ph.D. Dissertation. University of Iowa.
7. Frank Bentley, Katie Quehl, Jordan Wirfs-Brock, and Melissa Bica. 2019. Understanding Online News Behaviors. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3290605.3300820
8. Judy Van Biljon. 2011. A critical review on the reporting of surveys in transdisciplinary research: A case study in Information Systems. TD: The Journal for Transdisciplinary Research in Southern Africa 7, 2(2011), 337–350.
9. Kelly Caine. 2016. Local Standards for Sample Size at CHI. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 981–992. https://doi.org/10.1145/2858036.2858498
10. Stuart K. Card, Thomas P. Moran, and Allen Newell. 1983. The Psychology of Human-Computer Interaction. CRC Press.
11. Hao-Fei Cheng, Ruotong Wang, Zheng Zhang, Fiona O’Connell, Terrance Gray, F. Maxwell Harper, and Haiyi Zhu. 2019. Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300789
12. Manu Chopra, Indrani Medhi Thies, Joyojeet Pal, Colin Scott, William Thies, and Vivek Seshadri. 2019. Exploring Crowdsourced Work in Low-Resource Settings. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300611
13. Kar-Hai Chu, Jason Colditz, Jaime Sidani, Michael Zimmer, and Brian Primack. 2019. Re-evaluating standards of human subjects protection for sensitive health data in social media networks. Social Networks (2019). https://doi.org/10.1016/j.socnet.2019.10.010
14. Marianela Ciolfi Felice, Sarah Fdili Alaoui, and Wendy E. Mackay. 2018. Knotation: Exploring and Documenting Choreographic Processes(CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3173574.3174022
15. Mayo Clinic. 2020. Clinical Trials - Volunteering. https://www.mayo.edu/research/clinical-trials/deciding-to-volunteer
16. Michael Correll, Dominik Moritz, and Jeffrey Heer. 2018. Value-Suppressing Uncertainty Palettes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3173574.3174216
17. Jean Costa, François Guimbretière, Malte F. Jung, and Tanzeem Choudhury. 2019. BoostMeUp: Improving Cognitive Performance in the Moment by Unobtrusively Regulating Emotions with a Smartwatch. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 2, Article 40 (June 2019), 23 pages. https://doi.org/10.1145/3328911
18. Gordon Darroch. 2013, October 19. How Science Goes Wrong. https://www.economist.com/weeklyedition/2013-10-19
19. Maitraye Das, Darren Gergle, and Anne Marie Piper. 2019. ”It Doesn’t Win You Friends”: Understanding Accessibility in Collaborative Writing for People with Vision Impairments. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 191 (Nov. 2019), 26 pages. https://doi.org/10.1145/3359293
20. Christine Dedding, Nicole S. Goedhart, Jacqueline E.W. Broerse, and Tineke A. Abma. 2020. Exploring the boundaries of ‘good’ Participatory Action Research in times of increasing popularity: dealing with constraints in local policy for digital inclusion. Educational Action Research 0, 0 (2020), 1–17. https://doi.org/10.1080/09650792.2020.1743733
21. Interaction Design. 2020. Human-Computer Interaction (HCI). https://www.interaction-design.org/literature/topics/human-computer-interaction
22. Sanorita Dey, Brittany R.L. Duff, Wai-Tat Fu, and Karrie Karahalios. 2019. VidLyz: An Interactive Approach to Assist Novice Entrepreneurs in Making Persuasive Campaign Videos. 3, CSCW, Article 43 (Nov. 2019), 26 pages. https://doi.org/10.1145/3359145
23. Neal Dickert and Christine Grady. 2012. What’s the Price of a Research Subject? Approaches to Payment for Research Participation. New England Journal of Medicine 341 (2012), 198–203. https://doi.org/10.1056/NEJM199907153410312
24. Jessa Dickinson, Mark Díaz, Christopher A. Le Dantec, and Sheena Erete. 2019. ”The Cavalry Ain’t Coming in to Save Us”: Supporting Capacities and Relationships through Civic Tech. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 123 (Nov. 2019), 21 pages. https://doi.org/10.1145/3359225
25. Tawanna R. Dillahunt, Sylvia Simioni, and Xuecong Xu. 2019. Online Grocery Delivery Services: An Opportunity to Address Food Disparities in Transportation-Scarce Areas. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3290605.3300879
26. Dinora Dominguez, Mandy Jawara, Nicole Martino, Ninet Sinaii, and Christine Grady. 2012. Commonly Performed Procedures in Clinical Research: A Benchmark for Payment. Contemporary Clinical Trials 33 (2012), 860–867. https://doi.org/10.1016/j.cct.2012.05.001
27. Judith Dörrenbächer and Marc Hassenzahl. 2019. Changing Perspective: A Co-Design Approach to Explore Future Possibilities of Divergent Hearing(CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300259
28. Julia R. Fernandez and Jeremy Birnholtz. 2019. ”I Don’t Want Them to Not Know”: Investigating Decisions to Disclose Transgender Identity on Dating Platforms. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 226 (Nov. 2019), 21 pages. https://doi.org/10.1145/3359328
29. Oluwaseyi Feyisetan and Elena Simperl. 2017. Social Incentives in Paid Collaborative Crowdsourcing. ACM Trans. Intell. Syst. Technol. 8, 6, Article 73 (July 2017), 31 pages. https://doi.org/10.1145/3078852
30. Casey Fiesler and Nicholas Proferes. 2018. ’Participant’ perceptions of Twitter research ethics. Social Media+ Society 4, 1 (2018). https://doi.org/10.1177/2056305118763366
31. Andrew T. Fiore, Coye Cheshire, Lindsay Shaw Taylor, and G.A. Mendelsohn. 2014. Incentives to Participate in Online Research: An Experimental Examination of ”Surprise” Incentives. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 3433–3442. https://doi.org/10.1145/2556288.2557418
32. U.S. Food and Drug Administration. 2018, Jan. Payment and Reimbursement to Research Subjects. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/payment-and-reimbursement-research-subjects
33. U.S. Food and Drug Administration. 2018, January. Payment and Reimbursement to Research Subjects. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/payment-and-reimbursement-research-subjects
34. Alliance for Human Research Protection (AHRP). 2015, May 7. 1900: Walter Reed, MD, Yellow Fever. https://ahrp.org/1900-walter-reed-md-yellow-fever/
35. US Office for Human Research Protections. 2020. Vulnerable Populations. https://www.hhs.gov/ohrp/regulations-and-policy/guidance/vulnerable-populations/index.html
36. Diana Freed, Sam Havron, Emily Tseng, Andrea Gallardo, Rahul Chatterjee, Thomas Ristenpart, and Nicola Dell. 2019. ”Is My Phone Hacked?” Analyzing Clinical Computer Security Interventions with Survivors of Intimate Partner Violence. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 202 (Nov. 2019), 24 pages. https://doi.org/10.1145/3359304
37. Diana Freed, Jackeline Palmer, Diana Minchala, Karen Levy, Thomas Ristenpart, and Nicola Dell. 2018. “A Stalker’s Paradise”: How Intimate Partner Abusers Exploit Technology. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3174241
38. Christine Grady. 2005. Payment of clinical research subjects. Journal of Clinical Investigation 115, 7 (2005), 1681–1687. https://doi.org/10.1172/JCI25694
39. Christine Grady, Neal Dickert, Tom Jawetz, Gary Gensler, and Ezekiel Emanuel. 2005. An analysis of US practices of paying research participants. Contemporary Clinical Trials 26, 3 (2005), 365–375. https://doi.org/10.1016/j.cct.2005.02.003
40. M.L. Gray and S. Suri. 2019. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. HMH Books.
41. Ben Green and Yiling Chen. 2019. The Principles and Limits of Algorithm-in-the-Loop Decision Making. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 50 (Nov. 2019), 24 pages. https://doi.org/10.1145/3359152
42. The PRISMA Group. 2015. The PRISMA Statement. http://www.prisma-statement.org/
43. Darren Guinness, Edward Cutrell, and Meredith Ringel Morris. 2018. Caption Crawler: Enabling Reusable Alternative Text Descriptions Using Reverse Image Search. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3173574.3174092
44. S M Taiabul Haque, Pratyasha Saha, Muhammad Sajidur Rahman, and Syed Ishtiaque Ahmed. 2019. Of Ulti, ’Hajano’, and ”Matachetar Otanetak Datam”: Exploring Local Practices of Exchanging Confidential and Sensitive Information in Urban Bangladesh. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 173 (Nov. 2019), 22 pages. https://doi.org/10.1145/3359275
45. Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P. Bigham. 2018. A data-driven analysis of workers’ earnings on Amazon Mechanical Turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montreal, Quebec, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 14.
46. Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Benjamin V. Hanrahan, Jeffrey P. Bigham, and Chris Callison-Burch. 2019. Worker Demographics and Earnings on Amazon Mechanical Turk: An Exploratory Analysis. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3290607.3312970
47. Gail E. Henderson. 2015. The Ethics of HIV “Cure” Research: What Can We Learn from Consent Forms?AIDS Research & Human Retroviruses 31, 56 (2015), 60. https://doi.org/10.1089/AID.2014.0219
48. Paul Hitlin. 2016, July 11. Research in the Crowdsourcing Age, a Case Study. https://www.pewresearch.org/internet/2016/07/11/research-in-the-crowdsourcing-age-a-case-study/
49. Kevin Hu, Snehalkumar ’Neil’ S. Gaikwad, Madelon Hulsebos, Michiel A. Bakker, Emanuel Zgraggen, César Hidalgo, Tim Kraska, Guoliang Li, Arvind Satyanarayan, and Çağatay Demiralp. 2019. VizNet: Towards A Large-Scale Visualization Learning and Benchmarking Repository. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300892
50. Lilly C. Irani and M. Six Silberman. 2013. Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). Association for Computing Machinery, New York, NY, USA, 611–620. https://doi.org/10.1145/2470654.2470742
51. Jason T. Jacques and Per Ola Kristensson. 2019. Crowdworker Economics in the Gig Economy. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/3290605.3300621
52. Ravi Karkar, Rafal Kocielnik, Xiaoyi Zhang, Jasmine Zia, George N. Ioannou, Sean A. Munson, and James Fogarty. 2018. Beacon: Designing a Portable Device for Self-Administering a Measure of Critical Flicker Frequency. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 3, Article 117 (Sept. 2018), 27 pages. https://doi.org/10.1145/3264927
53. Robert Klitzman, Ilene Albala, Joseph Siragusa, Kristen N. Nelson, and Paul S. Appelbaum. 2007. The reporting of monetary compensation in research articles. Journal of Empirical Research on Human Research Ethics 2, 4(2007). https://doi.org/10.1525/jer.2007.2.4.61
54. Noah Klugman, Santiago Correa, Pat Pannuto, Matthew Podolsky, Jay Taneja, and Prabal Dutta. 2019. The Open Incentive Kit (OINK): Standardizing the Generation, Comparison, and Deployment of Incentive Systems. In Proceedings of the Tenth International Conference on Information and Communication Technologies and Development (Ahmedabad, India) (ICTD ’19). Association for Computing Machinery, New York, NY, USA, Article 17, 11 pages. https://doi.org/10.1145/3287098.3287101
55. Emily A. Largent and Holly Fernandez Lynch. 2017. Paying research participants: regulatory uncertainty, conceptual confusion, and a path forward. Yale journal of health policy, law, and ethics 17, 1 (2017).
56. Laura Lascau, Sandy J. J. Gould, Anna L. Cox, Elizaveta Karmannaya, and Duncan P. Brumby. 2019. Monotasking or Multitasking: Designing for Crowdworkers’ Preferences. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300649
57. Jessica Latterman and Jon F. Merz. 2001. How Much Are Subjects Paid to Participate in Research?The American Journal of Bioethics 1, 2 (2001), 45–46. https://doi.org/10.1162/152651601300169040
58. Thomas Reuters Practical Law. 2020. Protected Class. https://content.next.westlaw.com/5-501-5857?transitionType=Default&contextData=(sc.Default)&firstPage=true
59. Susan Lederer. 1995. Subjected to science: human experimentation in America before the Second World War. The Johns Hopkins University Press.
60. Tianyi Li, Chandler J. Manns, Chris North, and Kurt Luther. 2019. Dropping the Baton? Understanding Errors and Bottlenecks in a Crowdsourced Sensemaking Pipeline. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 136 (Nov. 2019), 26 pages. https://doi.org/10.1145/3359238
61. Irene Lopatovska and Ioannis Arapakis. 2011. Theories, methods and current research on emotions in library and information science, information retrieval and human–computer interaction. Processing & Management 47, 4 (2011), 575–592. https://doi.org/10.1016/j.ipm.2010.09.001
62. David Martin, Benjamin V. Hanrahan, Jacki O’Neill, and Neha Gupta. 2014. Being a Turker. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (Baltimore, Maryland, USA) (CSCW ’14). Association for Computing Machinery, New York, NY, USA, 224–235. https://doi.org/10.1145/2531602.2531663
63. Nora McDonald, Benjamin Mako Hill, Rachel Greenstadt, and Andrea Forte. 2019. Privacy, Anonymity, and Perceived Risk in Open Collaboration: A Study of Service Providers. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300901
64. Brad A Meyers. 1998. A Brief History of Human Computer Interaction Technology. ACM Interactions 5, 2 (1998), 44–54.
65. Shayan Mirjafari, Kizito Masaba, Ted Grover, Weichen Wang, Pino Audia, Andrew T. Campbell, Nitesh V. Chawla, Vedant Das Swain, Munmun De Choudhury, Anind K. Dey, Sidney K. D’Mello, Ge Gao, Julie M. Gregg, Krithika Jagannath, Kaifeng Jiang, Suwen Lin, Qiang Liu, Gloria Mark, Gonzalo J. Martinez, Stephen M. Mattingly, Edward Moskal, Raghu Mulukutla, Subigya Nepal, Kari Nies, Manikanta D. Reddy, Pablo Robles-Granda, Koustuv Saha, Anusha Sirigiri, and Aaron Striegel. 2019. Differentiating Higher and Lower Job Performers in the Workplace Using Mobile Sensing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 2, Article 37 (June 2019), 24 pages. https://doi.org/10.1145/3328908
66. D. Moher, A. Liberati, J. Tetzlaff, DG Altman, and The PRISMA Group. 2009. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6, 6 (2009), e1000097. https://doi.org/10.1371/journal.pmed1000097
67. Meredith Ringel Morris, Adam Fourney, Abdullah Ali, and Laura Vonessen. 2018. Understanding the Needs of Searchers with Dyslexia. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3173574.3173609
68. Martez E. Mott and Jacob O. Wobbrock. 2019. Cluster Touch: Improving Touch Accuracy on Smartphones for People with Motor and Situational Impairments. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300257
69. Mohamed Musthag, Andrew Raij, Deepak Ganesan, Santosh Kumar, and Saul Shiffman. 2011. Exploring Micro-Incentive Strategies for Participant Compensation in High-Burden Studies. In Proceedings of the 13th International Conference on Ubiquitous Computing (Beijing, China) (UbiComp ’11). Association for Computing Machinery, New York, NY, USA, 435–444. https://doi.org/10.1145/2030112.2030170
70. Pranathi Mylavarapu, Adil Yalcin, Xan Gregg, and Niklas Elmqvist. 2019. Ranked-List Visualization: A Graphical Perception Study. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300422
71. Amal Nanavati, M. Bernardine Dias, and Aaron Steinfeld. 2018. Speak Up: A Multi-Year Deployment of Games to Motivate Speech Therapy in India. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3173574.3173892
72. Engineering National Academies of Sciencesand Medicine. 2019. Reproducibility and replicability in science. National Academies Press.
73. The U.S. Office of Human Research Protections. 2020. Regulations. https://www.hhs.gov/ohrp/regulations-and-policy/regulations/index.html
74. NYU Department of Psychology. 2020. Participate in Paid Studies. https://as.nyu.edu/psychology/research/participate-in-research/paid-studies.html
75. University of Tronoto. 2020. Compensation & Reimbursement of Research Participants. https://research.utoronto.ca/compensation-reimbursement-research-participants
76. Gary M. Olson and Judith S. Olson. 2003. Human-computer interaction: Psychological aspects of the human use of computing. ACM Interactions 54, 1 (2003), 491–516. https://doi.org/10.1146/annurev.psych.54.101601.145044
77. Steve Oney, Alan Lundgard, Rebecca Krosnick, Michael Nebeling, and Walter S. Lasecki. 2018. Arboretum and Arbility: Improving Web Accessibility Through a Shared Browsing Architecture. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST ’18). Association for Computing Machinery, New York, NY, USA, 937–949. https://doi.org/10.1145/3242587.3242649
78. Jessica A. Pater, Brooke Farrington, Alycia Brown, Lauren E. Reining, Tammy Toscos, and Elizabeth D. Mynatt. 2019. Exploring Indicators of Digital Self-Harm with Eating Disorder Patients: A Case Study. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 84 (Nov. 2019), 26 pages. https://doi.org/10.1145/3359186
79. Shishir G. Patil, Don Kurian Dennis, Chirag Pabbaraju, Nadeem Shaheer, Harsha Vardhan Simhadri, Vivek Seshadri, Manik Varma, and Prateek Jain. 2019. GesturePod: Enabling On-Device Gesture-Based Interaction for White Cane Users. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology(New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 403–415. https://doi.org/10.1145/3332165.3347881
80. Simon T. Perrault and Weiyu Zhang. 2019. Effects of Moderation and Opinion Heterogeneity on Attitude towards the Online Deliberation Experience. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300247
81. Susanne Poeller, Max V. Birk, Nicola Baumann, and Regan L. Mandryk. 2018. Let Me Be Implicit: Using Motive Disposition Theory to Predict and Explain Behaviour in Digital Games. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3173574.3173764
82. Lisa Posch, Arnim Bleier, Clemens M. Lechner, Daniel Danner, Fabian Flöck, and Markus Strohmaier. 2019. Measuring Motivations of Crowdworkers: The Multidimensional Crowdworker Motivation Scale. Trans. Soc. Comput. 2, 2, Article 8 (Sept. 2019), 34 pages. https://doi.org/10.1145/3335081
83. Jenny Preece, Yvonne Rogers, Helen Sharp, David Benyon, Simon Holland, and Tom Carey. 1994. Human-Computer Interaction. Addison-Wesley Longman Ltd.
84. Elizabeth Ripley, Francis Macrina adn Monkia Markowitz, and Chris Gennings. 2011. Why Do We Pay? A National Survey of Investigators and IRB Chairpersons. J Empirical Res on Hum Res Ethics 43, 5 (2011), 54. https://doi.org/10.1525/jer.2010.5.3.43
85. Jessica Schroeder, Chelsey Wilkes, Kael Rowan, Arturo Toledo, Ann Paradiso, Mary Czerwinski, Gloria Mark, and Marsha M. Linehan. 2018. Pocket Skills: A Conversational Mobile Web App To Support Dialectical Behavioral Therapy. (2018), 1–15. https://doi.org/10.1145/3173574.3173972
86. Valentin Schwind, Pascal Knierim, Nico Haas, and Niels Henze. 2019. Using Presence Questionnaires in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300590
87. Gözel Shakeri, Alexander Ng, John H. Williamson, and Stephen A. Brewster. 2016. Evaluation of Haptic Patterns on a Steering Wheel. In Proceedings of the 8th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (Ann Arbor, MI, USA) (Automotive’UI 16). Association for Computing Machinery, New York, NY, USA, 129–136. https://doi.org/10.1145/3003715.3005417
88. M. S. Silberman, B. Tomlinson, R. LaPlante, J. Ross, L. Irani, and A. Zaldivar. 2018. Responsible Research with Crowds: Pay Crowdworkers at Least Minimum Wage. Commun. ACM 61, 3 (Feb. 2018), 39–41. https://doi.org/10.1145/3180492
89. Six Silberman. 2015. Stop Citing Ross et al.2010, ”Who are the Crowdworkers?”. https://medium.com/@silberman/stop-citing-ross-et-al-2010-who-are-the-crowdworkers-b3b9b1e8d300
90. Rachel Charlotte Smith and Claus Bossen adn Anne Marie Kanstrup. 2017. Participatory design in an era of participation. International Journal of CoCreation in Design and the Arts 13, 2(2017). https://doi.org/10.1080/15710882.2017.1310466
91. Lisa Stähli, David Rudi, and Martin Raubal. 2018. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST ’18). Association for Computing Machinery, New York, NY, USA, 299–311. https://doi.org/10.1145/3242587.3242624
92. David M. Swanson and Rebecca A. Betensky. 2015. Research participant compensation: a matter of statistical inference as well as ethics. Contemporary clinical trials 45 (2015). https://doi.org/10.1016/j.cct.2015.08.0141
93. Jerry O. Talton, Krishna Dusad, Konstantinos Koiliaris, and Ranjitha S. Kumar. 2019. How Do People Sort by Ratings?. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/3290605.3300535
94. Jonathan A. Tran, Katie S. Yang, Katie Davis, and Alexis Hiniker. 2019. Modeling the Engagement-Disengagement Cycle of Compulsive Phone Use(CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300542
95. Berkeley University California. 2020. Compensation of Research Participants. https://cphs.berkeley.edu/compensation.pdf
96. Irvine University California. 2020. Payments to Research Participants. https://www.accounting.uci.edu/support/guides/research-participants.html
97. Niels van Berkel, Jorge Goncalves, Danula Hettiachchi, Senuri Wijenayake, Ryan M. Kelly, and Vassilis Kostakos. 2019. Crowdsourcing Perceptions of Fair Predictors for Machine Learning: A Recidivism Case Study. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 28 (Nov. 2019), 21 pages. https://doi.org/10.1145/3359130
98. Aditya Vashistha, Abhinav Garg, and Richard Anderson. 2019. ReCall: Crowdsourcing on Basic Phones to Financially Sustain Voice Forums. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300399
99. Sukrit Venkatagiri, Jacob Thebault-Spieker, Rachel Kohler, John Purviance, Rifat Sabbir Mansur, and Kurt Luther. 2019. GroundTruth: Augmenting Expert Image Geolocation with Crowdsourcing and Shared Representations. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 107 (Nov. 2019), 30 pages. https://doi.org/10.1145/3359209
100. John Vines, Rachel Clarke, Peter Wright, John McCarthy, and Patrick Olivier. 2013. Configuring Participation: On How We Involve People in Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). Association for Computing Machinery, New York, NY, USA, 429–438. https://doi.org/10.1145/2470654.2470716
101. Jessica Vitak, Yuting Liao, Mega Subramaniam, and Priya Kumar. 2018. ’I Knew It Was Too Good to Be True”: The Challenges Economically Disadvantaged Internet Users Face in Assessing Trustworthiness, Avoiding Scams, and Developing Self-Efficacy Online. Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 176 (Nov. 2018), 25 pages. https://doi.org/10.1145/3274445
102. Jessica Vitak, Katie Shilton, and Zahra Ashktorab. 2016. Beyond the Belmont Principles: Ethical Challenges, Practices, and Beliefs in the Online Data Research Community. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (San Francisco, California, USA) (CSCW ’16). Association for Computing Machinery, New York, NY, USA, 941–953. https://doi.org/10.1145/2818048.2820078
103. Isaac Wang, Jesse Smith, and Jaime Ruiz. 2019. Exploring Virtual Agents for Augmented Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300511
104. Helen C. Wauck, Elisa D. Mekler, and Wai-Tat Fu. 2019. A Player-Centric Approach to Designing Spatial Skill Training Games. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300296
105. Gavin Wood, Kiel Long, Tom Feltwell, Scarlett Rowland, Phillip Brooker, Jamie Mahoney, John Vines, Julie Barnett, and Shaun Lawson. 2018. Rethinking Engagement with Online News through Social and Visual Co-Annotation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3173574.3174150
106. Ziang Xiao, Po-Shiun Ho, Xinran Wang, Karrie Karahalios, and Hari Sundaram. 2019. Should We Use an Abstract Comic Form to Persuade? Experiments with Online Charitable Donation. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 75 (Nov. 2019), 28 pages. https://doi.org/10.1145/3359177
107. Lillian Yang, Brennan Jones, Carman Neustaedter, and Samarth Singhal. 2018. Shopping Over Distance through a Telepresence Robot. Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 191 (Nov. 2018), 18 pages. https://doi.org/10.1145/3274460
108. Min Zhang, Corina Sas, Zoe Lambert, and Masitah Ahmad. 2019. Designing for the Infrastructure of the Supply Chain of Malay Handwoven Songket in Terengganu. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3290605.3300716

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)