To Live in Their Utopia: Why Algorithmic Systems Create Absurd Outcomes

ABSTRACT

The promise AI’s proponents have made for decades is one in which our needs are predicted, anticipated, and met - often before we even realize it. Instead, algorithmic systems, particularly AIs trained on large datasets and deployed to massive scales, seem to keep making the wrong decisions, causing harm and rewarding absurd outcomes. Attempts to make sense of why AIs make wrong calls in the moment explain the instances of errors, but how the environment surrounding these systems precipitate those instances remains murky. This paper draws from anthropological work on bureaucracies, states, and power, translating these ideas into a theory describing the structural tendency for powerful algorithmic systems to cause tremendous harm. I show how administrative models and projections of the world create marginalization, just as algorithmic models cause representational and allocative harm. This paper concludes with a recommendation to avoid the absurdity algorithmic systems produce by denying them power.

References

1. J. Khadijah Abdurahman. 2019. FAT* Be Wilin’. https://upfromthecracks.medium.com/fat-be-wilin-deb56bf92539
2. Ali Alkhatib and Michael Bernstein. 2019. Street-Level Algorithms: A Theory at the Gaps Between Policy and Decisions. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland UK) (CHI 2019). ACM, New York, NY, USA, Article 530, 13 pages. https://doi.org/10.1145/3290605.3300760
3. Ali Alkhatib, Michael S. Bernstein, and Margaret Levi. 2017. Examining Crowd Work and Gig Work Through The Historical Lens of Piecework. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI 2017). ACM, New York, NY, USA, 4599–4616. https://doi.org/10.1145/3025453.3025974
4. Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). ACM, New York, NY, USA, Article 3, 13 pages. https://doi.org/10.1145/3290605.3300233
5. Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine Bias. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
6. Anthony G. Attrino. 2020. He spent 10 days in jail after facial recognition software led to the arrest of the wrong man, lawsuit says. https://www.nj.com/middlesex/2020/12/he-spent-10-days-in-jail-after-facial-recognition-software-led-to-the-arrest-of-the-wrong-man-lawsuit-says.html
7. Daniel W. Barowy, Charlie Curtsinger, Emery D. Berger, and Andrew McGregor. 2012. AutoMan: A Platform for Integrating Human-Based and Digital Computation. In Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications (Tucson, Arizona, USA) (OOPSLA ’12). Association for Computing Machinery, New York, NY, USA, 639–654. https://doi.org/10.1145/2384616.2384663
8. Eric P.S. Baumer and M. Six Silberman. 2011. When the Implication is Not to Design (Technology). In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA, 2271–2274. https://doi.org/10.1145/1978942.1979275
9. Marcus Bell. 2017. Criminalization of Blackness: Systemic racism and the reproduction of racial inequality in the US criminal justice system. In Systemic Racism. Springer, 163–183.
10. Ruha Benjamin. 2019. Race After Technology: Abolitionist Tools for the New Jim Code. John Wiley & Sons.
11. Cynthia L. Bennett and Daniela K. Rosner. 2019. The Promise of Empathy: Design, Disability, and Knowing the ”Other”. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300528
12. Shlomo Berkovsky, Ronnie Taib, Irena Koprinska, Eileen Wang, Yucheng Zeng, Jingjie Li, and Sabina Kleitman. 2019. Detecting Personality Traits Using Eye-Tracking Data. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). ACM, New York, NY, USA, Article 221, 12 pages. https://doi.org/10.1145/3290605.3300451
13. Chris Bopp, Ellie Harmon, and Amy Voida. 2017. Disempowered by Data: Nonprofits, Social Enterprises, and the Consequences of Data-Driven Work. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). ACM, New York, NY, USA, 3608–3619. https://doi.org/10.1145/3025453.3025694
14. Geoffrey C Bowker and Susan Leigh Star. 2000. Sorting Things Out: Classification and its Consequences. MIT press.
15. Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency(Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, New York, NY, USA, 77–91. http://proceedings.mlr.press/v81/buolamwini18a.html
16. Lilah Burke. 2020. The Death and Life of an Admissions Algorithm. https://www.insidehighered.com/admissions/article/2020/12/14/u-texas-will-stop-using-controversial-algorithm-evaluate-phd
17. Carrie J. Cai, Shamsi T. Iqbal, and Jaime Teevan. 2016. Chain Reactions: The Impact of Order on Microtask Chains. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems(CHI ’16). ACM, 3143–3154. https://doi.org/10.1145/2858036.2858237
18. Robyn Caplan and Tarleton Gillespie. 2020. Tiered Governance and Demonetization: The Shifting Terms of Labor and Compensation in the Platform Economy. Social Media + Society 6, 2 (2020), 2056305120936636. https://doi.org/10.1177/2056305120936636
19. Ngai Keung Chan and Lee Humphreys. 2018. Mediatization of social space and the case of Uber drivers. Media and Communication 6, 2 (2018), 29–38.
20. Pablo Chapablanco. 2018. Traveling While Hispanic: Border Patrol Immigration Investigatory Stops at TSA Checkpoints and Hispanic Appearance. Cornell L. Rev. 104(2018), 1401.
21. Nicholas (Nick) Chown. 2010. ‘Do You Have Any Difficulties That I May Not Be Aware of?’ A Study of Autism Awareness and Understanding in the UK Police Service. International Journal of Police Science & Management 12, 2(2010), 256–273. https://doi.org/10.1350/ijps.2010.12.2.174
22. Angèle Christin. 2017. Algorithms in practice: Comparing web journalism and criminal justice. Big Data & Society 4, 2 (2017), 2053951717718855. https://doi.org/10.1177/2053951717718855
23. M. Cifor, P. Garcia, T.L. Cowan, J. Rault, T. Sutherland, A. Chan, J. Rode, A.L. Hoffmann, N. Salehi, and L. Nakamura. 2019. Feminist Data Manifest-No. https://www.manifestno.com/
24. Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. 2017. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 797–806.
25. Sasha Costanza-Chock. 2018. Design justice, AI, and escape from the matrix of domination. (2018).
26. Florian Daniel, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, and Mohammad Allahbakhsh. 2018. Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques, and Assurance Actions. ACM Comput. Surv. 51, 1, Article 7 (Jan. 2018), 40 pages. https://doi.org/10.1145/3148148
27. Pierre Dragicevic, Yvonne Jansen, Abhraneel Sarma, Matthew Kay, and Fanny Chevalier. 2019. Increasing the Transparency of Research Papers with Explorable Multiverse Analyses. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3290605.3300295
28. Darren Edge, Jonathan Larson, and Christopher White. 2018. Bringing AI to BI: Enabling Visual Analytics of Unstructured Data in a Modern Business Intelligence Platform. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI EA ’18). Association for Computing Machinery, New York, NY, USA, 1–9. https://doi.org/10.1145/3170427.3174367
29. Motahhare Eslami, Sneha R. Krishna Kumaran, Christian Sandvig, and Karrie Karahalios. 2018. Communicating Algorithmic Process in Online Behavioral Advertising. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). ACM, New York, NY, USA, Article 432, 13 pages. https://doi.org/10.1145/3173574.3174006
30. Motahhare Eslami, Aimee Rickman, Kristen Vaccaro, Amirhossein Aleyasen, Andy Vuong, Karrie Karahalios, Kevin Hamilton, and Christian Sandvig. 2015. “I Always Assumed That I Wasn’t Really That Close to [Her]”: Reasoning About Invisible Algorithms in News Feeds. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). ACM, New York, NY, USA, 153–162. https://doi.org/10.1145/2702123.2702556
31. Motahhare Eslami, Kristen Vaccaro, Min Kyung Lee, Amit Elazari Bar On, Eric Gilbert, and Karrie Karahalios. 2019. User Attitudes Towards Algorithmic Opacity and Transparency in Online Reviewing Platforms. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). ACM, New York, NY, USA, Article 494, 14 pages. https://doi.org/10.1145/3290605.3300724
32. Megan Farokhmanesh. 2018. YouTube is still restricting and demonetizing LGBT videos — and adding anti-LGBT ads to some. https://www.theverge.com/2018/6/4/17424472/youtube-lgbt-demonetization-ads-algorithm
33. Jo Freeman. 1972. The tyranny of structurelessness. Berkeley Journal of Sociology(1972), 151–164.
34. Sandy J. J. Gould, Anna L. Cox, and Duncan P. Brumby. 2016. Diminished Control in Crowdsourcing: An Investigation of Crowdworker Multitasking Behavior. ACM Trans. Comput.-Hum. Interact. 23, 3, Article 19 (June 2016), 29 pages. https://doi.org/10.1145/2928269
35. David Graeber. 2015. The utopia of rules: On technology, stupidity, and the secret joys of bureaucracy. Melville House.
36. Kevin Hamilton, Karrie Karahalios, Christian Sandvig, and Motahhare Eslami. 2014. A Path to Understanding the Effects of Algorithm Awareness. In CHI ’14 Extended Abstracts on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI EA ’14). Association for Computing Machinery, New York, NY, USA, 631–642. https://doi.org/10.1145/2559206.2578883
37. Anna Lauren Hoffmann. 2020. Terms of inclusion: Data, discourse, violence. new media & society(2020), 1461444820958725.
38. Yasmin Ibrahim. 2010. The breastfeeding controversy and Facebook: Politicization of image, privacy and protest. International Journal of E-Politics (IJEP) 1, 2 (2010), 16–28.
39. Lilly C. Irani and M. Six Silberman. 2013. Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’13). ACM, 611–620. https://doi.org/10.1145/2470654.2470742
40. John L Jackson Jr. 2013. Thin Description. Harvard University Press.
41. Maurice Jakesch, Megan French, Xiao Ma, Jeffrey T. Hancock, and Mor Naaman. 2019. AI-Mediated Communication: How the Perception That Profile Text Was Written by AI Affects Trustworthiness. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). ACM, New York, NY, USA, Article 239, 13 pages. https://doi.org/10.1145/3290605.3300469
42. Akilah Johnson. 2017. BOSTON. RACISM. IMAGE. REALITY.: The Spotlight Team takes on our hardest question. https://apps.bostonglobe.com/spotlight/boston-racism-image-reality/series/image/
43. Erik Kain. 2017. YouTube Wants Content Creators To Appeal Demonetization, But It’s Not Always That Easy. https://www.forbes.com/sites/erikkain/2017/09/18/adpocalypse-2017-heres-what-you-need-to-know-about-youtubes-demonetization-troubles/#31b00bb96c26
44. Janin Koch, Andrés Lucero, Lena Hegemann, and Antti Oulasvirta. 2019. May AI? Design Ideation with Cooperative Contextual Bandits. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300863
45. Rafal Kocielnik, Saleema Amershi, and Paul N. Bennett. 2019. Will You Accept an Imperfect AI?: Exploring Designs for Adjusting End-user Expectations of AI Systems. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). ACM, New York, NY, USA, Article 411, 14 pages. https://doi.org/10.1145/3290605.3300641
46. Yubo Kou and Xinning Gui. 2018. Entangled with Numbers: Quantified Self and Others in a Team-Based Online Game. Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 93 (Nov. 2018), 25 pages. https://doi.org/10.1145/3274362
47. Anand Kulkarni, Matthew Can, and Björn Hartmann. 2012. Collaboratively Crowdsourcing Workflows with Turkomatic. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (Seattle, Washington, USA) (CSCW ’12). Association for Computing Machinery, New York, NY, USA, 1003–1012. https://doi.org/10.1145/2145204.2145354
48. Juhi Kulshrestha, Motahhare Eslami, Johnnatan Messias, Muhammad Bilal Zafar, Saptarshi Ghosh, Krishna P. Gummadi, and Karrie Karahalios. 2017. Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (Portland, Oregon, USA) (CSCW ’17). Association for Computing Machinery, New York, NY, USA, 417–432. https://doi.org/10.1145/2998181.2998321
49. Min Kyung Lee, Daniel Kusbit, Evan Metsky, and Laura Dabbish. 2015. Working with Machines: The Impact of Algorithmic and Data–Driven Management on Human Workers. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems(CHI ’15). ACM, 1603–1612. https://doi.org/10.1145/2702123.2702548
50. Sarah Lewis. 2019. The Racial Bias Built Into Photography. nytimes.com/2019/04/25/lens/sarah-lewis-racial-bias-photography.html
51. Q. Vera Liao, Daniel Gruen, and Sarah Miller. 2020. Questioning the AI: Informing Design Practices for Explainable AI User Experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3313831.3376590
52. Michael Lipsky. 1980. Street-Level Bureaucracy: The Dilemmas of the Individual in Public Service. Russell Sage Foundation.
53. Zachary Mack. 2019. How streaming affects the lengths of songs. https://www.theverge.com/2019/5/28/18642978/music-streaming-spotify-song-length-distribution-production-switched-on-pop-vergecast-interview
54. Shannon Mattern. 2017. The City Is Not a Computer.Places Journal 18(2017).
55. Brian McInnis, Dan Cosley, Chaebong Nam, and Gilly Leshed. 2016. Taking a HIT: Designing Around Rejection, Mistrust, Risk, and Workers’ Experiences in Amazon Mechanical Turk. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems(CHI ’16). ACM, 2271–2282. https://doi.org/10.1145/2858036.2858539
56. Sharon E Moore, Michael A Robinson, Dewey M Clayton, A Christson Adedoyin, Daniel A Boamah, Eric Kyere, and Dana K Harmon. 2018. A critical race perspective of police shooting of unarmed black males in the United States: Implications for social work. Urban Social Work 2, 1 (2018), 33–47.
57. Safiya Umoja Noble. 2018. Algorithms of oppression: How search engines reinforce racism. nyu Press.
58. Ihudiya Finda Ogbonnaya-Ogburu, Angela D.R. Smith, Alexandra To, and Kentaro Toyama. 2020. Critical Race Theory for HCI. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–16. https://doi.org/10.1145/3313831.3376392
59. Azzurra Pini, Jer Hayes, Connor Upton, and Medb Corcoran. 2019. AI Inspired Recipes: Designing Computationally Creative Food Combos. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3290607.3312948
60. Nicole E Pottinger. 2018. Don’t Forget to Subscribe: Regulation of Online Advertising Evaluated Through YouTube’s Monetization Problem. Ky. LJ 107(2018), 515.
61. Rida Qadri. 2020. Delivery Platform Algorithms Don’t Work Without Drivers’ Deep Local Knowledge. https://slate.com/technology/2020/12/gojek-grab-indonesia-delivery-platforms-algorithms.html
62. A. Jamie Saris. 2007. Culture, inequality and the bureaucratic imagination: states and subjects for a new millenium. Irish Journal of Anthropology (Special Edition) 10, 2 (2007), 54–60. http://mural.maynoothuniversity.ie/1123/
63. James C Scott. 1998. Seeing Like A State: How Certain Schemes to Improve the Human Condition Have Failed. Yale University Press.
64. CalvinJohn Smiley and David Fakunle. 2016. From “brute” to “thug:” The demonization and criminalization of unarmed Black male victims in America. Journal of human behavior in the social environment 26, 3-4(2016), 350–366.
65. Jacob Snow. 2018. Amazon’s Face Recognition Falsely Matched 28 Members of Congress With Mugshots. https://www.aclu.org/blog/privacy-technology/surveillance-technologies/amazons-face-recognition-falsely-matched-28
66. Luke Stark. 2019. Facial Recognition is the Plutonium of AI. XRDS 25, 3 (April 2019), 50–55. https://doi.org/10.1145/3313129
67. Miriam E Sweeney and Kelsea Whaley. 2019. Technically white: emoji skin-tone modifiers as American technoculture. First Monday (2019).
68. Yonathan Tesfai. [n.d.]. YouTube and the LGBT+ Community: Demonitizing Pride. ([n. d.]).
69. Jasper Tran O’Leary, Sara Zewde, Jennifer Mankoff, and Daniela K. Rosner. 2019. Who Gets to Future? Race, Representation, and Design Methods in Africatown. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300791
70. Kristen Vaccaro, Dylan Huang, Motahhare Eslami, Christian Sandvig, Kevin Hamilton, and Karrie Karahalios. 2018. The Illusion of Control: Placebo Effects of Control Settings. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3173590
71. Sara Wachter-Boettcher. 2017. Technically wrong: Sexist apps, biased algorithms, and other threats of toxic tech. WW Norton & Company.
72. Austin Waters and Risto Miikkulainen. 2014. GRADE: Machine learning support for graduate admissions. AI Magazine 35, 1 (2014), 64–64.
73. Sijia Xiao, Danaë Metaxa, Joon Sung Park, Karrie Karahalios, and Niloufar Salehi. 2020. Random, Messy, Funny, Raw: Finstas as Intimate Reconfigurations of Social Media. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376424
74. Qian Yang, Aaron Steinfeld, and John Zimmerman. 2019. Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processes. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). ACM, New York, NY, USA, Article 238, 11 pages. https://doi.org/10.1145/3290605.3300468

Index Terms

• Applied computing

• Computing methodologies

• Social and professional topics