ABSTRACT
Research within the social sciences and humanities has long characterized the work of data science as a sociotechnical process, comprised of a set of logics and techniques that are inseparable from specific social norms, expectations and contexts of development and use. Yet all too often the assumptions and premises underlying data analysis remain unexamined, even in contemporary debates about the fairness of algorithmic systems. This blindspot exists in part because the methodological toolkit used to evaluate the fairness of algorithmic systems remains limited to a narrow set of computational and legal modes of analysis. In this paper, we expand on Elish and Boyd's [17] call for data scientists to develop more robust frameworks for understanding their work as situated practice by examining a specific methodological debate within the field of anthropology, frequently referred to as the practice of "studying up". We reflect on the contributions that the call to "study up" has made in the field of anthropology before making the case that the field of algorithmic fairness would similarly benefit from a reorientation "upward". A case study from our own work illustrates what it looks like to reorient one's research questions "up" in a high-profile debate regarding the fairness of an algorithmic system - namely, pretrial risk assessment in American criminal law. We discuss the limitations of contemporary fairness discourse with regard to pretrial risk assessment before highlighting the insights gained when we reframe our research questions to focus on those who inhabit positions of power and authority within the U.S. court system. Finally, we reflect on the challenges we have encountered in implementing data science projects that "study up". In the process, we surface new insights and questions about what it means to ethically engage in data science work that directly confronts issues of power and authority.
References
- 1987. United States v. Salerno, 481 U.S. 739, 755. (1987).Google Scholar
- Julia Angwin et al. 2016. Machine Bias. (2016).Google Scholar
- Chelsea Barabas. 2019. Beyond Bias: Re-imagining the Terms of. Criminal Law (April 25, 2019) (2019).Google Scholar
- Allen J Beck and Darrell K Gilliard. 1995. Bureau of Jus-tice Statistics, Prisoners in 1994. NCJ 151654 (1995).Google Scholar
- Ruha Benjamin. 2016. Catching Our Breath: Critical Race STS and the Carceral Imagination. Engaging Science, Technology, and Society 2 (2016), 145--156.Google Scholar
Cross Ref
- Ruha Benjamin, Troy Duster, Ron Eglash, Nettrice Gaskins, Anthony Ryan Hatch, Andrea Miller, Alondra Nelson, Tamara K Nopper, Christopher Perreira, Winifred R Poster, et al. [n. d.]. Captivating Technology: Race, Carceral Technoscience, and Liberatory Imagination in Everyday Life. ([n. d.]).Google Scholar
- Gerald D Berreman. 1968. Is anthropology alive? Social responsibility in social anthropology. Current Anthropology 9, 5, Part 1 (1968), 391--396.Google Scholar
- Kum-Kum Bhavnani et al. 1991. Talking politics: A psychological framing of views from youth in Britain. Cambridge University Press.Google Scholar
- Angéle Christin. 2017. Algorithms in practice: Comparing web journalism and criminal justice. Big Data & Society 4, 2 (2017).Google Scholar
- Sasha Costanza-Chock. 2018. Design Justice: towards an intersectional feminist framework for design theory and practice. Proceedings of the Design Research Society (2018).Google Scholar
Cross Ref
- Matthew DeMichele, Peter Baumgartner, Michael Wenger, Kelle Barrick, Megan Comfort, and Shilpi Misra. 2018. The public safety assessment: A re-validation and assessment of predictive utility and differential prediction by race and gender in kentucky. Available at SSRN 3168452 (2018).Google Scholar
- Stephen Demuth and Darrell Steffensmeier. 2004. The impact of gender and race-ethnicity in the pretrial release process. Social Problems 51, 2 (2004), 222--242.Google Scholar
Cross Ref
- Nicholas Diakopoulos, Sorelle Friedler, Marcelo Arenas, Solon Barocas, Michael Hay, Bill Howe, HV Jagadish, Kris Unsworth, Arnaud Sahuguet, Suresh Venkatasubramanian, et al. 2017. Principles for accountable algorithms and a social impact statement for algorithms. FAT/ML (2017).Google Scholar
- Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference. ACM, 214--226.Google Scholar
Digital Library
- Catherine DâĂŹIgnazio and Lauren Klein. 2019. Data feminism. (2019).Google Scholar
- Mark Easterby-Smith, Richard Thorpe, and Paul R Jackson. 2012. Management Research. Sage.Google Scholar
- Madeleine Clare Elish and Danah Boyd. 2018. Situating methods in the magic of Big Data and AI. Communication monographs 85, 1 (2018), 57--80.Google Scholar
- Virginia Eubanks. 2018. Automating inequality: How high-tech tools profile, police, and punish the poor. St. Martin's Press.Google Scholar
- Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and removing disparate impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 259--268.Google Scholar
Digital Library
- J Fitz and D Halpin. 1994. Ministers and mandarins: Educational research in elite settings. Researching the powerful in education. G. Walford. (1994).Google Scholar
- Gutorm Gjessing. 1968. The social responsibility of the social scientist. Current Anthropology 9, 5, Part 1 (1968), 397--402.Google Scholar
- Kathleen Gough. 1968. New proposals for anthropologists. Current anthropology 9, 5, Part 1 (1968), 403--435.Google Scholar
- Lauryn P Gouldin. 2018. Defining Flight Risk. U. Chi. L. Rev. 85 (2018), 677.Google Scholar
- Daniel Greene, Anna Lauren Hoffmann, and Luke Stark. 2019. Better, nicer, clearer, fairer: A critical assessment of the movement for ethical artificial intelligence and machine learning. In Proceedings of the 52nd Hawaii International Conference on System Sciences.Google Scholar
Cross Ref
- Hugh Gusterson. 1997. Studying up revisited. PoLAR: Political and Legal Anthropology Review 20, 1 (1997), 114--119.Google Scholar
Cross Ref
- Jessica Guynn. 2015. Google photos labeled black people 'gorillas'. USA Today 1 (2015).Google Scholar
- Eric Price Hardt, Moritz and Nati Srebro. 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems (2016).Google Scholar
- Rosanna Hertz and Jonathan B Imber. 1995. Studying elites using qualitative methods. Vol. 175. Sage Publications.Google Scholar
- Kashmir Hill. 2012. How Target figured out a teen girl was pregnant before her father did. Forbes (2012).Google Scholar
- Anna Lauren Hoffmann. 2019. Where fairness fails: data, algorithms, and the limits of antidiscrimination discourse. Information, Communication & Society 22, 7 (2019), 900--915.Google Scholar
Cross Ref
- Sheila Jasanoff. 1994. Learning from disaster: risk management after Bhopal. University of Pennsylvania Press.Google Scholar
- Niels Kerssens. [n. d.]. De-Agentializing Data Practices: The Shifting Power of Metaphor in 1990s Discourses on Data Mining. ([n. d.]).Google Scholar
- Bruno Lepri, Nuria Oliver, Emmanuel Letouzé, Alex Pentland, and Patrick Vinck. 2018. Fair, transparent, and accountable algorithmic decision-making processes. Philosophy & Technology 31, 4 (2018), 611--627.Google Scholar
Cross Ref
- George E Marcus. 1983. Elites, ethnographic issues. Univ of New Mexico Pr.Google Scholar
- Carole McGranahan. 2016. Theorizing refusal: An introduction. Cultural Anthropology 31, 3 (2016), 319--325.Google Scholar
Cross Ref
- Todd D Minton and Zhen Zeng. 2015. Jail inmates at midyear 2014. NCJ 248629 (2015).Google Scholar
- Khalil Gibran Muhammad. 2011. The condemnation of blackness. Harvard University Press.Google Scholar
- Laura Nader. 1972. Up the anthropologist: perspectives gained from studying up. (1972).Google Scholar
- Peter Novick. 1988. That noble dream: The 'objectivity question' and the American historical profession. Vol. 13. Cambridge University Press.Google Scholar
- Rodrigo Ochigame. 2019. The Illusion of algorithmic fairness. (2019).Google Scholar
- Teresa Odendahl, Aileen M Shaw, et al. 2002. Interviewing elites. Handbook of interview research: Context and method (2002), 299--316.Google Scholar
- Sherry B Ortner. 1995. Resistance and the problem of ethnographic refusal. Comparative studies in society and history 37, 1 (1995), 173--193.Google Scholar
- Samir Passi and Solon Barocas. 2019. Problem formulation and fairness. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 39--48.Google Scholar
Digital Library
- Rebecca Warne Peters and Claire Wendland. 2016. Up the Africanist: the possibilities and problems of 'studying up'in Africa. Critical African Studies 8, 3 (2016), 239--254.Google Scholar
Cross Ref
- Rob Price. 2016. Microsoft is deleting its AI chatbotâĂŹs incredibly racist tweets. Business Insider (2016).Google Scholar
- Esther Priyadharshini. 2003. Coming unstuck: Thinking otherwise about âĂIJs-tudying upâĂİ. Anthropology & education quarterly 34, 4 (2003), 420--437.Google Scholar
- Edward Said. 2014. Orientalism. In Geopolitics. Routledge.Google Scholar
- Andrew D Selbst and Solon Barocas. 2018. The intuitive appeal of explainable machines. Fordham L. Rev. 87 (2018), 1085.Google Scholar
- Andrew D Selbst, Danah Boyd, Sorelle A Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. Fairness and abstraction in sociotechnical systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 59--68.Google Scholar
Digital Library
- Sara Selwood. 2002. The politics of data collection: Gathering, analysing and using data about the subsidised cultural sector in England. Cultural trends 12, 47 (2002), 13--84.Google Scholar
- Audra Simpson. 2007. On ethnographic refusal: Indigeneity, âĂŸvoiceâĂŹand colonial citizenship. Junctures: The Journal for Thematic Dialogue 9 (2007).Google Scholar
- Audra Simpson. 2014. Mohawk interruptus: Political life across the borders of settler states. Duke University Press.Google Scholar
- Mark Solovey. 2001. Science and the state during the Cold War: Blurred boundaries and a contested legacy. (2001).Google Scholar
- Daniel Souleles. 2018. How to Study People Who Do Not Want to be Studied: Practical Reflections on Studying Up. PoLAR: Political and Legal Anthropology Review 41, S1 (2018), 51--68.Google Scholar
Cross Ref
- Luke Stark and Anna Lauren Hoffmann. [n. d.]. Data Is the New What? Popular Metaphors & Professional Ethics in Emerging Data Culture. ([n. d.]).Google Scholar
- Luke Stark and Anna Lauren Hoffmann. [n. d.]. Data Is the New What? Popular Metaphors & Professional Ethics in Emerging Data Culture. Journal ISSN 2371 ([n. d.]), 4549.Google Scholar
- Eve Tuck and K Wayne Yang. 2014. R-words: Refusing research. Humanizing research: Decolonizing qualitative inquiry with youth and communities (2014), 223--248.Google Scholar
- P Wagner and W Sawyer. 2018. Mass incarceration: The whole pie. The Prison Policy Initiative. (2018).Google Scholar
- Ed Yong. 2017. A popular algorithm is no better at predicting crimes than random people. The Atlantic (2017).Google Scholar




Comments