10.1145/3351095.3372859acmconferencesArticle/Chapter ViewAbstractPublication PagesfatConference Proceedings
research-article
Free Access

Studying up: reorienting the study of algorithmic fairness around issues of power

ABSTRACT

Research within the social sciences and humanities has long characterized the work of data science as a sociotechnical process, comprised of a set of logics and techniques that are inseparable from specific social norms, expectations and contexts of development and use. Yet all too often the assumptions and premises underlying data analysis remain unexamined, even in contemporary debates about the fairness of algorithmic systems. This blindspot exists in part because the methodological toolkit used to evaluate the fairness of algorithmic systems remains limited to a narrow set of computational and legal modes of analysis. In this paper, we expand on Elish and Boyd's [17] call for data scientists to develop more robust frameworks for understanding their work as situated practice by examining a specific methodological debate within the field of anthropology, frequently referred to as the practice of "studying up". We reflect on the contributions that the call to "study up" has made in the field of anthropology before making the case that the field of algorithmic fairness would similarly benefit from a reorientation "upward". A case study from our own work illustrates what it looks like to reorient one's research questions "up" in a high-profile debate regarding the fairness of an algorithmic system - namely, pretrial risk assessment in American criminal law. We discuss the limitations of contemporary fairness discourse with regard to pretrial risk assessment before highlighting the insights gained when we reframe our research questions to focus on those who inhabit positions of power and authority within the U.S. court system. Finally, we reflect on the challenges we have encountered in implementing data science projects that "study up". In the process, we surface new insights and questions about what it means to ethically engage in data science work that directly confronts issues of power and authority.

References

  1. 1987. United States v. Salerno, 481 U.S. 739, 755. (1987).Google ScholarGoogle Scholar
  2. Julia Angwin et al. 2016. Machine Bias. (2016).Google ScholarGoogle Scholar
  3. Chelsea Barabas. 2019. Beyond Bias: Re-imagining the Terms of. Criminal Law (April 25, 2019) (2019).Google ScholarGoogle Scholar
  4. Allen J Beck and Darrell K Gilliard. 1995. Bureau of Jus-tice Statistics, Prisoners in 1994. NCJ 151654 (1995).Google ScholarGoogle Scholar
  5. Ruha Benjamin. 2016. Catching Our Breath: Critical Race STS and the Carceral Imagination. Engaging Science, Technology, and Society 2 (2016), 145--156.Google ScholarGoogle ScholarCross RefCross Ref
  6. Ruha Benjamin, Troy Duster, Ron Eglash, Nettrice Gaskins, Anthony Ryan Hatch, Andrea Miller, Alondra Nelson, Tamara K Nopper, Christopher Perreira, Winifred R Poster, et al. [n. d.]. Captivating Technology: Race, Carceral Technoscience, and Liberatory Imagination in Everyday Life. ([n. d.]).Google ScholarGoogle Scholar
  7. Gerald D Berreman. 1968. Is anthropology alive? Social responsibility in social anthropology. Current Anthropology 9, 5, Part 1 (1968), 391--396.Google ScholarGoogle Scholar
  8. Kum-Kum Bhavnani et al. 1991. Talking politics: A psychological framing of views from youth in Britain. Cambridge University Press.Google ScholarGoogle Scholar
  9. Angéle Christin. 2017. Algorithms in practice: Comparing web journalism and criminal justice. Big Data & Society 4, 2 (2017).Google ScholarGoogle Scholar
  10. Sasha Costanza-Chock. 2018. Design Justice: towards an intersectional feminist framework for design theory and practice. Proceedings of the Design Research Society (2018).Google ScholarGoogle ScholarCross RefCross Ref
  11. Matthew DeMichele, Peter Baumgartner, Michael Wenger, Kelle Barrick, Megan Comfort, and Shilpi Misra. 2018. The public safety assessment: A re-validation and assessment of predictive utility and differential prediction by race and gender in kentucky. Available at SSRN 3168452 (2018).Google ScholarGoogle Scholar
  12. Stephen Demuth and Darrell Steffensmeier. 2004. The impact of gender and race-ethnicity in the pretrial release process. Social Problems 51, 2 (2004), 222--242.Google ScholarGoogle ScholarCross RefCross Ref
  13. Nicholas Diakopoulos, Sorelle Friedler, Marcelo Arenas, Solon Barocas, Michael Hay, Bill Howe, HV Jagadish, Kris Unsworth, Arnaud Sahuguet, Suresh Venkatasubramanian, et al. 2017. Principles for accountable algorithms and a social impact statement for algorithms. FAT/ML (2017).Google ScholarGoogle Scholar
  14. Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference. ACM, 214--226.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Catherine DâĂŹIgnazio and Lauren Klein. 2019. Data feminism. (2019).Google ScholarGoogle Scholar
  16. Mark Easterby-Smith, Richard Thorpe, and Paul R Jackson. 2012. Management Research. Sage.Google ScholarGoogle Scholar
  17. Madeleine Clare Elish and Danah Boyd. 2018. Situating methods in the magic of Big Data and AI. Communication monographs 85, 1 (2018), 57--80.Google ScholarGoogle Scholar
  18. Virginia Eubanks. 2018. Automating inequality: How high-tech tools profile, police, and punish the poor. St. Martin's Press.Google ScholarGoogle Scholar
  19. Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and removing disparate impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 259--268.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. J Fitz and D Halpin. 1994. Ministers and mandarins: Educational research in elite settings. Researching the powerful in education. G. Walford. (1994).Google ScholarGoogle Scholar
  21. Gutorm Gjessing. 1968. The social responsibility of the social scientist. Current Anthropology 9, 5, Part 1 (1968), 397--402.Google ScholarGoogle Scholar
  22. Kathleen Gough. 1968. New proposals for anthropologists. Current anthropology 9, 5, Part 1 (1968), 403--435.Google ScholarGoogle Scholar
  23. Lauryn P Gouldin. 2018. Defining Flight Risk. U. Chi. L. Rev. 85 (2018), 677.Google ScholarGoogle Scholar
  24. Daniel Greene, Anna Lauren Hoffmann, and Luke Stark. 2019. Better, nicer, clearer, fairer: A critical assessment of the movement for ethical artificial intelligence and machine learning. In Proceedings of the 52nd Hawaii International Conference on System Sciences.Google ScholarGoogle ScholarCross RefCross Ref
  25. Hugh Gusterson. 1997. Studying up revisited. PoLAR: Political and Legal Anthropology Review 20, 1 (1997), 114--119.Google ScholarGoogle ScholarCross RefCross Ref
  26. Jessica Guynn. 2015. Google photos labeled black people 'gorillas'. USA Today 1 (2015).Google ScholarGoogle Scholar
  27. Eric Price Hardt, Moritz and Nati Srebro. 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems (2016).Google ScholarGoogle Scholar
  28. Rosanna Hertz and Jonathan B Imber. 1995. Studying elites using qualitative methods. Vol. 175. Sage Publications.Google ScholarGoogle Scholar
  29. Kashmir Hill. 2012. How Target figured out a teen girl was pregnant before her father did. Forbes (2012).Google ScholarGoogle Scholar
  30. Anna Lauren Hoffmann. 2019. Where fairness fails: data, algorithms, and the limits of antidiscrimination discourse. Information, Communication & Society 22, 7 (2019), 900--915.Google ScholarGoogle ScholarCross RefCross Ref
  31. Sheila Jasanoff. 1994. Learning from disaster: risk management after Bhopal. University of Pennsylvania Press.Google ScholarGoogle Scholar
  32. Niels Kerssens. [n. d.]. De-Agentializing Data Practices: The Shifting Power of Metaphor in 1990s Discourses on Data Mining. ([n. d.]).Google ScholarGoogle Scholar
  33. Bruno Lepri, Nuria Oliver, Emmanuel Letouzé, Alex Pentland, and Patrick Vinck. 2018. Fair, transparent, and accountable algorithmic decision-making processes. Philosophy & Technology 31, 4 (2018), 611--627.Google ScholarGoogle ScholarCross RefCross Ref
  34. George E Marcus. 1983. Elites, ethnographic issues. Univ of New Mexico Pr.Google ScholarGoogle Scholar
  35. Carole McGranahan. 2016. Theorizing refusal: An introduction. Cultural Anthropology 31, 3 (2016), 319--325.Google ScholarGoogle ScholarCross RefCross Ref
  36. Todd D Minton and Zhen Zeng. 2015. Jail inmates at midyear 2014. NCJ 248629 (2015).Google ScholarGoogle Scholar
  37. Khalil Gibran Muhammad. 2011. The condemnation of blackness. Harvard University Press.Google ScholarGoogle Scholar
  38. Laura Nader. 1972. Up the anthropologist: perspectives gained from studying up. (1972).Google ScholarGoogle Scholar
  39. Peter Novick. 1988. That noble dream: The 'objectivity question' and the American historical profession. Vol. 13. Cambridge University Press.Google ScholarGoogle Scholar
  40. Rodrigo Ochigame. 2019. The Illusion of algorithmic fairness. (2019).Google ScholarGoogle Scholar
  41. Teresa Odendahl, Aileen M Shaw, et al. 2002. Interviewing elites. Handbook of interview research: Context and method (2002), 299--316.Google ScholarGoogle Scholar
  42. Sherry B Ortner. 1995. Resistance and the problem of ethnographic refusal. Comparative studies in society and history 37, 1 (1995), 173--193.Google ScholarGoogle Scholar
  43. Samir Passi and Solon Barocas. 2019. Problem formulation and fairness. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 39--48.Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Rebecca Warne Peters and Claire Wendland. 2016. Up the Africanist: the possibilities and problems of 'studying up'in Africa. Critical African Studies 8, 3 (2016), 239--254.Google ScholarGoogle ScholarCross RefCross Ref
  45. Rob Price. 2016. Microsoft is deleting its AI chatbotâĂŹs incredibly racist tweets. Business Insider (2016).Google ScholarGoogle Scholar
  46. Esther Priyadharshini. 2003. Coming unstuck: Thinking otherwise about âĂIJs-tudying upâĂİ. Anthropology & education quarterly 34, 4 (2003), 420--437.Google ScholarGoogle Scholar
  47. Edward Said. 2014. Orientalism. In Geopolitics. Routledge.Google ScholarGoogle Scholar
  48. Andrew D Selbst and Solon Barocas. 2018. The intuitive appeal of explainable machines. Fordham L. Rev. 87 (2018), 1085.Google ScholarGoogle Scholar
  49. Andrew D Selbst, Danah Boyd, Sorelle A Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. Fairness and abstraction in sociotechnical systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 59--68.Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Sara Selwood. 2002. The politics of data collection: Gathering, analysing and using data about the subsidised cultural sector in England. Cultural trends 12, 47 (2002), 13--84.Google ScholarGoogle Scholar
  51. Audra Simpson. 2007. On ethnographic refusal: Indigeneity, âĂŸvoiceâĂŹand colonial citizenship. Junctures: The Journal for Thematic Dialogue 9 (2007).Google ScholarGoogle Scholar
  52. Audra Simpson. 2014. Mohawk interruptus: Political life across the borders of settler states. Duke University Press.Google ScholarGoogle Scholar
  53. Mark Solovey. 2001. Science and the state during the Cold War: Blurred boundaries and a contested legacy. (2001).Google ScholarGoogle Scholar
  54. Daniel Souleles. 2018. How to Study People Who Do Not Want to be Studied: Practical Reflections on Studying Up. PoLAR: Political and Legal Anthropology Review 41, S1 (2018), 51--68.Google ScholarGoogle ScholarCross RefCross Ref
  55. Luke Stark and Anna Lauren Hoffmann. [n. d.]. Data Is the New What? Popular Metaphors & Professional Ethics in Emerging Data Culture. ([n. d.]).Google ScholarGoogle Scholar
  56. Luke Stark and Anna Lauren Hoffmann. [n. d.]. Data Is the New What? Popular Metaphors & Professional Ethics in Emerging Data Culture. Journal ISSN 2371 ([n. d.]), 4549.Google ScholarGoogle Scholar
  57. Eve Tuck and K Wayne Yang. 2014. R-words: Refusing research. Humanizing research: Decolonizing qualitative inquiry with youth and communities (2014), 223--248.Google ScholarGoogle Scholar
  58. P Wagner and W Sawyer. 2018. Mass incarceration: The whole pie. The Prison Policy Initiative. (2018).Google ScholarGoogle Scholar
  59. Ed Yong. 2017. A popular algorithm is no better at predicting crimes than random people. The Atlantic (2017).Google ScholarGoogle Scholar

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Sign in
  • Article Metrics

    • Downloads (Last 12 months)256
    • Downloads (Last 6 weeks)256

    Other Metrics

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader
About Cookies On This Site

We use cookies to ensure that we give you the best experience on our website.

Learn more

Got it!