10.1145/3351095.3372833acmconferencesArticle/Chapter ViewAbstractPublication PagesfatConference Proceedings
research-article
Free Access

What to account for when accounting for algorithms: a systematic literature review on algorithmic accountability

ABSTRACT

As research on algorithms and their impact proliferates, so do calls for scrutiny/accountability of algorithms. A systematic review of the work that has been done in the field of 'algorithmic accountability' has so far been lacking. This contribution puts forth such a systematic review, following the PRISMA statement. 242 English articles from the period 2008 up to and including 2018 were collected and extracted from Web of Science and SCOPUS, using a recursive query design coupled with computational methods. The 242 articles were prioritized and ordered using affinity mapping, resulting in 93 'core articles' which are presented in this contribution. The recursive search strategy made it possible to look beyond the term 'algorithmic accountability'. That is, the query also included terms closely connected to the theme (e.g. ethics and AI, regulation of algorithms). This approach allows for a perspective not just from critical algorithm studies, but an interdisciplinary overview drawing on material from data studies to law, and from computer science to governance studies. To structure the material, Bovens's widely accepted definition of accountability serves as a focal point. The material is analyzed on the five points Bovens identified as integral to accountability: its arguments on (1) the actor, (2) the forum, (3) the relationship between the two, (3) the content and criteria of the account, and finally (5) the consequences which may result from the account. The review makes three contributions. First, an integration of accountability theory in the algorithmic accountability discussion. Second, a cross-sectoral overview of the that same discussion viewed in light of accountability theory which pays extra attention to accountability risks in algorithmic systems. Lastly, it provides a definition of algorithmic accountability based on accountability theory and algorithmic accountability literature.

References

  1. ACM. 2018. ACM Code of Ethics and Professional Conduct. https://www.acm.org/code-of-ethicsGoogle ScholarGoogle Scholar
  2. Amina Adadi and Mohammed Berrada. 2018. Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI). IEEE Access 6 (2018), 52138--52160. Google ScholarGoogle ScholarCross RefCross Ref
  3. Optimity advisors and Future of privacy forum. 2018. algo:aware: Raising awareness on algorithms. Technical Report. Washington.Google ScholarGoogle Scholar
  4. AI Now. 2018. Algorithmic Accountability Policy Toolkit. Technical Report. AI Now / New York University, New York.Google ScholarGoogle Scholar
  5. AI Now. 2018. Litigating algorithms: challenging government use of algorithmic decision systems. Technical Report. AI Now/New York University, New York.Google ScholarGoogle Scholar
  6. Madeleine Akrich and Bruno Latour. 1992. A summary of a convenient vocabulary for the semiotics of human and nonhuman assemblies. In Shaping technology / building society: studies in sociotechnical change, Wiebe E. Bijker and John Law (Eds.). MIT Press, Cambridge/London, Chapter 9, 259--264.Google ScholarGoogle Scholar
  7. Mike Ananny. 2015. Toward an Ethics of Algorithms. Science, Technology, & Human Values 41, 1 (2015), 93--117. Google ScholarGoogle ScholarCross RefCross Ref
  8. Mike Ananny. 2016. Toward an Ethics of Algorithms: Convening, Observation, Probability, and Timeliness. Science Technology and Human Values 41, 1 (2016), 93--117. arXiv:arXiv:1011.1669v3 Google ScholarGoogle ScholarCross RefCross Ref
  9. Mike Ananny and Kate Crawford. 2018. Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability. New Media and Society 20, 3 (2018), 973--989. Google ScholarGoogle ScholarCross RefCross Ref
  10. Leighton Andrews. 2018. Public administration, public leadership and the construction of public value in the age of the algorithm and 'big data'. Public Administration (2018). Google ScholarGoogle ScholarCross RefCross Ref
  11. Theo Araujo, Claes De Vreese, Natali Helberger, Sanne Kruikemeier, Julia Van Weert, Bol, Nadine, Daniel Oberski, Mykola Pechenizkiy, Gabi Schaap, and Linnet Taylor. 2018. Automated Decision-Making Fairness in an AI-driven world: Public perceptions, hopes, and concerns. Technical Report. University of Amsterdam, Amsterdam.Google ScholarGoogle Scholar
  12. Thomas Arnold and Matthias Scheutz. 2018. The "big red button" is too late: an alternative model for the ethical evaluation of AI systems. Ethics and Information Technology 20, 1 (2018), 59--69. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Karen Barad. 2007. Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning. Duke University Press, Durham/London.Google ScholarGoogle ScholarCross RefCross Ref
  14. Mathieu Bastian, Sebastien Heymann, and Mathieu Jacomy. 2009. Gephi: an open source software for exploring and manipulating networks. In Proceedings of the Third International ICWSM Conference. http://www.aaai.org/ocs/index.php/ICWSM/09/paper/download/154/1009Google ScholarGoogle Scholar
  15. Seth D. Baum. 2017. Social choice ethics in artificial intelligence. AI and Society (2017), 1--12. Google ScholarGoogle ScholarCross RefCross Ref
  16. Beall's List. 2018. Beall's list of Predatory Journals and Publishers. https://beallslist.weebly.com/Google ScholarGoogle Scholar
  17. Lyria Bennett Moses and Janet Chan. 2018. Algorithmic prediction in policing: assumptions, evaluation, and accountability. Policing and Society 28, 7 (2018), 806--822. Google ScholarGoogle ScholarCross RefCross Ref
  18. Bij Voorbaat Verdacht. 2018. Wat is SyRI? https://bijvoorbaatverdacht.nl/watis-syri/Google ScholarGoogle Scholar
  19. Bij Voorbaat Verdacht. 2019. Missie. https://bijvoorbaatverdacht.nl/missie/Google ScholarGoogle Scholar
  20. Reuben Binns. 2017. Algorithmic Accountability and Public Reason. Philosophy & Technology (2017). arXiv:arXiv:1702.08608 Google ScholarGoogle ScholarCross RefCross Ref
  21. Reuben Binns. 2018. What can political philosophy teach us about algorithmic fairness? IEEE Security & Privacy, 2018 16, 3 (2018), 73--80.Google ScholarGoogle ScholarCross RefCross Ref
  22. Reuben Binns, Max Van Kleek, Michael Veale, Ulrik Lyngs, Jun Zhao, and Nigel Shadbolt. 2018. 'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions. In CHI '18. arXiv:1801.10408 Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Vincent D. Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. 2008. Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment 2008, 10 (2008), 1--12. arXiv:0803.0476 Google ScholarGoogle ScholarCross RefCross Ref
  24. Mark Bovens. 2007. Analysing and Assessing Accountability: A Conceptual Framework. European Law Journal 13, 4 (2007), 447--468. arXiv:1468-0386 Google ScholarGoogle ScholarCross RefCross Ref
  25. Mark Bovens. 2010. Two concepts of accountability: Accountability as a virtue and as a Mechanism. West European Politics 33, 5 (2010), 946--967. arXiv:arXiv:1011.1669v3 Google ScholarGoogle ScholarCross RefCross Ref
  26. Mark Bovens, Thomas Schillemans, and Paul T. Hart. 2008. Does public accountability work? An assessment tool. Public Administration 86, 1 (2008), 225--242. Google ScholarGoogle ScholarCross RefCross Ref
  27. Mark Bovens and Stavros Zouridis. 2002. From to System Level -Level Bureaucracies: How Information and Communication Technology Is Transforming Administrative Discretion and Constitutional Control. Public Administration Review 62, 2 (2002), 174--184.Google ScholarGoogle ScholarCross RefCross Ref
  28. Engin Bozdag. 2013. Bias in algorithmic filtering and personalization. Ethics and Information Technology 15, 3 (2013), 209--227. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Louis Brandeis. 1914. What publicity can do. In Other people's money and how the bankers use it. Frederick A. Srokes, New York. http://louisville.edu/law/library/special-collections/the-louis-d.-brandeis-collection/other-peoples-money-chapter-vGoogle ScholarGoogle Scholar
  30. Gijs Jan Brandsma and Thomas Schillemans. 2013. The accountability cube: Measuring accountability. Journal of Public Administration Research and Theory 23, 4 (2013), 953--975. Google ScholarGoogle ScholarCross RefCross Ref
  31. Kiel Brennan-Marquez. 2016. Plausible Cause: Explanatory Standards in the Age of Powerful Machines. Ssrn 1 (2016). Google ScholarGoogle ScholarCross RefCross Ref
  32. Dennis Broeders, Erik Schrijvers, Bart van der Sloot, Rosamunde van Brakel, Josta de Hoog, and Ernst Hirsch Ballin. 2017. Big Data and security policies: Towards a framework for regulating the phases of analytics and use of Big Data. Computer Law and Security Review 33, 3 (2017), 309--323. Google ScholarGoogle ScholarCross RefCross Ref
  33. Joanna Bryson and Alan Winfield. 2017. Standardizing Ethical Design for Artificial Intelligence and Autonomous Systems. Computer 50, 5 (2017), 116--119. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Andrea Bunt, Matthew Lount, and Catherine Lauzon. 2012. Are explanations always important?: a study of deployed, low-cost intelligent interactive systems. Proceedings of the ACM Conference on Intelligent User Interfaces (2012), 169--178. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Jenna Burrell. 2015. How the Machine 'Thinks:' Understanding Opacity in Machine Learning Algorithms. Ssrn June (2015), 1--12. arXiv:arXiv:1307.4531v1 Google ScholarGoogle ScholarCross RefCross Ref
  36. Robyn Caplan and danah Boyd. 2018. Isomorphism through algorithms: Institutional dependencies in the case of Facebook. Big Data & Society 5, 1 (2018), 205395171875725. Google ScholarGoogle ScholarCross RefCross Ref
  37. Corinne Cath, Sandra Wachter, Brent Mittelstadt, Mariarosaria Taddeo, and Luciano Floridi. 2018. Artificial Intelligence and the 'Good Society': the US, EU, and UK approach. Science and Engineering Ethics 24, 2 (2018), 505--528. Google ScholarGoogle ScholarCross RefCross Ref
  38. Raja Chatila, Kay Firth-Butterflied, John C. Havens, and Konstantinos Karachalios. 2017. The IEEE Global Initiative for Ethical Considerations in Artificial Intelligence and Autonomous Systems. IEEE Robotics and Automation Magazine 24, 1 (2017), 110. Google ScholarGoogle ScholarCross RefCross Ref
  39. Kenneth Ward Church. 2017. Emerging trends: I did it, I did it, I did it, but. Natural Language Engineering 23, 3 (2017), 473--480. Google ScholarGoogle ScholarCross RefCross Ref
  40. Danielle Citron and Frank Pasquale. 2014. The Scored Society: Due Process for Automated Predictions. Washington Law Review 89, 1 (2014), 1--34. arXiv:arXiv:1011.1669v3 Google ScholarGoogle ScholarCross RefCross Ref
  41. Cary Coglianese and David Lehr. 2017. Regulating by robot: Administrative decision making in the Machine-learning era. Georgetown Law Journal 105, 5 (2017), 1147--1223. Google ScholarGoogle ScholarCross RefCross Ref
  42. Kate Crawford. 2016. Can an Algorithm be Agonistic? Ten Scenes from Life in Calculated Publics. Science Technology and Human Values 41, 1 (2016), 77--92. arXiv:arXiv:1011.1669v3 Google ScholarGoogle ScholarCross RefCross Ref
  43. K. Crawford and T. Gillespie. 2014. What is a flag for? Social media reporting tools and the vocabulary of complaint. New Media & Society (2014), 1461444814543163-. Google ScholarGoogle ScholarCross RefCross Ref
  44. Andrej Dameski. 2018. A comprehensive ethical framework for AI entities: Foundations. In 11th International Conference Artificial General Intelligence 2018. 42--51. Google ScholarGoogle ScholarCross RefCross Ref
  45. John Danaher. 2016. The Threat of Algocracy: Reality, Resistance and Accommodation. Philosophy and Technology 29, 3 (2016), 245--268. Google ScholarGoogle ScholarCross RefCross Ref
  46. John Danaher, Michael J Hogan, Chris Noone, Rónán Kennedy, Anthony Behan, Aisling De Paor, Heike Felzmann, Muki Haklay, Su-Ming Khoo, John Morison, Maria Helen Murphy, Niall O'Brolchain, Burkhard Schafer, and Kalpana Shankar. 2017. Algorithmic governance: Developing a research agenda through the power of collective intelligence. Big Data & Society 4, 2 (2017), 205395171772655. Google ScholarGoogle ScholarCross RefCross Ref
  47. Jeffrey Dastin. 2018. Amazon scraps secret AI recruiting tool that showed bias against women. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08GGoogle ScholarGoogle Scholar
  48. Paul B. de Laat. 2017. Algorithmic Decision-Making Based on Machine Learning from Big Data: Can Transparency Restore Accountability? Philosophy & Technology (2017). Google ScholarGoogle ScholarCross RefCross Ref
  49. S Dekker. 2018. Transparantie van algoritmes in gebruik bij de overheid.Google ScholarGoogle Scholar
  50. Lina Dencik, Arne Hintz, Joanna Redden, and Harry Warne. 2018. Data Scores as Governance: Investigating uses of citizen scoring in public services. Technical Report. Data Justice Lab, Cardiff. https://datajustice.files.wordpress.com/2018/12/data-scores-as-governance-project-report2.pdfGoogle ScholarGoogle Scholar
  51. Nicholas Diakopoulos. 2015. Accountability in Algorithmic Decision-making: A view from computational journalism. ACM Queue december (2015), 1--24.Google ScholarGoogle Scholar
  52. Nicholas Diakopoulos. 2015. Algorithmic Accountability. Digital Journalism 3, 3 (2015), 398--415. Google ScholarGoogle ScholarCross RefCross Ref
  53. Nicholas Diakopoulos. 2015. Algorithmic Accountability: Journalistic investigation of computational power structures. Digital Journalism 3, 3 (2015), 398--415. Google ScholarGoogle ScholarCross RefCross Ref
  54. Nicholas Diakopoulos. 2016. Accountability in Algorithmic Decision-making: A view from computational journalism. Commun. ACM 59, 2 (2016), 56--62. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Ezekiel Dixon-Román. 2016. Algo-Ritmo: More-Than-Human Performative Acts and the Racializing Assemblages of Algorithmic Architectures. Cultural Studies <-> Critical Methodologies 16, 5 (2016), 482--490. Google ScholarGoogle ScholarCross RefCross Ref
  56. J.E. Dobson. 2015. Can An Algorithm Be Disturbed?: Machine Learning, Intrinsic Criticism, and the Digital Humanities. College Literature: A Journal of Critical Literay Studies 42, 4 (2015), 543--564. Google ScholarGoogle ScholarCross RefCross Ref
  57. Danilo Doneda and Virgilio A.F. Almeida. 2016. What Is Algorithm Governance? IEEE Internet Computing 20, 4 (2016), 60--63. Google ScholarGoogle ScholarCross RefCross Ref
  58. Cat Drew. 2016. Data science ethics in government. Philosophical Transactions of the Royal Society 374, 2083 (2016), 20160119. Google ScholarGoogle ScholarCross RefCross Ref
  59. Marina Drosou, H.V. Jagadish, Evaggelia Pitoura, and Julia Stoyanovich. 2017. Diversity in Big Data: A Review. Big Data 5, 2 (2017), 73--84. Google ScholarGoogle ScholarCross RefCross Ref
  60. ECP Platform voor de InformatieSamenleving. 2018. Artificial Intelligence Impact Assessment.Google ScholarGoogle Scholar
  61. Lilian Edwards and Michael Veale. 2017. Enslaving the algorithm : from a 'right to an explanation' to a 'right to better decisions'? (2017), 12 pages.Google ScholarGoogle Scholar
  62. Amitai Etzioni and Oren Etzioni. 2017. Incorporating Ethics into Artificial Intelligence. Journal of Ethics 21, 4 (2017), 403--418. Google ScholarGoogle ScholarCross RefCross Ref
  63. Virginia Eubanks. 2018. Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. St. Martin's Press, New York.Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. European Union. 2016. Regulation 2016/679 of the European parliament and the Council of the European Union. arXiv:arXiv:1011.1669v3Google ScholarGoogle Scholar
  65. Katherine Fink. 2018. Opening the government's black boxes: freedom of information and algorithmic accountability. Information Communication and Society 21, 10 (2018), 1453--1471. Google ScholarGoogle ScholarCross RefCross Ref
  66. Batya Friedman and Helen Nissenbaum. 1996. Bias in computer systems. ACM SIGCHI Bulletin 14, 3 (1996), 330 --347. Google ScholarGoogle ScholarDigital LibraryDigital Library
  67. Urs Gasser and Virgilio A.F. Almeida. 2017. A Layered Model for AI Governance. IEEE Internet Computing 21, 6 (2017), 58--62. Google ScholarGoogle ScholarCross RefCross Ref
  68. Gemeente Rotterdam. 2018. Betalingsregeling gemeentelijke belastingschuld. https://www.rotterdam.nl/loket/betalingsregeling-belastingschuld/Google ScholarGoogle Scholar
  69. Jen Jack Gieseking. 2018. Operating anew: Queering GIS with good enough software. Canadian Geographer 62, 1 (2018), 55--66. Google ScholarGoogle ScholarCross RefCross Ref
  70. A. Goffey. 2008. Algorithm. In Software Studies: a lexicon, Matthew Fuller (Ed.). MIT Press, Cambridge/London, 15--20.Google ScholarGoogle Scholar
  71. Mika Gröndahl, Keith Collins, and James Glanz. 2019. The dangerous flaws in Boeing's automated system. https://www.nytimes.com/interactive/2019/03/29/business/boeing-737-max-8-flaws.htmlGoogle ScholarGoogle Scholar
  72. Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Dino Pedreschi, and Fosca Giannotti. 2018. A Survey Of Methods For Explaining Black Box Models. Comput. Surveys 51, 5 (2018). arXiv:1802.01933 http://arxiv.org/abs/1802.01933Google ScholarGoogle Scholar
  73. Andrew J. Hawkins. 2019. Deadly Boeing crashes raise questions about airplane automation. https://www.theverge.com/2019/3/15/18267365/boeing-737-max-8-crash-autopilot-automationGoogle ScholarGoogle Scholar
  74. Paul Henman. 2017. The computer says 'DEBT': Towards a critical sociology of algorithms and algorithmic governance. In Data for Policy 2017: Government by Algorithm? London. Google ScholarGoogle ScholarCross RefCross Ref
  75. Paul Henman. 2019. Of algorithms, Apps and advice: digital social policy and service delivery. Journal of Asian Public Policy 12, 1 (2019), 71--89. Google ScholarGoogle ScholarCross RefCross Ref
  76. Marc Hijink. 2018. Algoritme voorspelt wie fraude pleegt bij bijstandsuitkering., 8--10 pages. https://www.nrc.nl/nieuws/2018/04/08/algoritme-voorspelt-wie-fraude-pleegt-bij-bijstandsuitkering-a1598669Google ScholarGoogle Scholar
  77. Russell Hotten. 2015. Volkswagen: The scandal explained. https://www.bbc.com/news/business-34324772Google ScholarGoogle Scholar
  78. Husson University. 2019. What is the software development life cycle? https://online.husson.edu/software-development-cycle/Google ScholarGoogle Scholar
  79. IEEE. [n. d.]. IEEE Code of Ethics. https://www.ieee.org/about/corporate/governance/p7-8.htmlGoogle ScholarGoogle Scholar
  80. Lucas D Introna. 2016. Algorithms, Governance, and Governmentality : On Governing Academic Writing. Science Technology and Human Values 41, 1 (2016), 17--49. Google ScholarGoogle ScholarCross RefCross Ref
  81. Mathieu Jacomy, Tommaso Venturini, Sebastien Heymann, and Mathieu Bastian. 2014. ForceAtlas2, a continuous graph layout algorithm for handy network visualization designed for the Gephi software. PLoS ONE 9, 6 (2014), 1--12. arXiv:arXiv:1209.0748v1 Google ScholarGoogle ScholarCross RefCross Ref
  82. Marijn Janssen and George Kuk. 2016. The challenges and limits of big data algorithms in technocratic governance. Government Information Quarterly 33, 3 (2016), 371--377. Google ScholarGoogle ScholarCross RefCross Ref
  83. Deborah G. Johnson and Helen Nissenbaum. 1995. Computers, ethics & social values. Prentice-Hall, Upper Saddle River.Google ScholarGoogle Scholar
  84. Natascha Just and Michael Latzer. 2017. Governance by algorithms: reality construction by algorithmic selection on the Internet. Media, Culture and Society 39, 2 (2017), 238--258. Google ScholarGoogle ScholarCross RefCross Ref
  85. Christian Katzenbach. 2012. Technologies as Institutions: Rethinking the Role of Technology in Media Governance Constellations. In Trends in Communication Policy Research, New Theories, Methods & Subjects, Manuel Puppis and Natascha Just (Eds.). Intellect Books, Bristol, 117--138. arXiv:arXiv:1011.1669v3 Google ScholarGoogle ScholarCross RefCross Ref
  86. Jakko Kemper and Daan Kolkman. 2018. Transparent to whom? No algorithmic accountability without a critical audience. Information Communication and Society (2018), 1--16. Google ScholarGoogle ScholarCross RefCross Ref
  87. Rob Kitchin. 2019. The ethics of smart cities. https://www.rte.ie/brainstorm/2019/0425/1045602-the-ethics-of-smart-cities/Google ScholarGoogle Scholar
  88. Donald E. Knuth. 1984. Literate Programming. Computers and Chemical Engineering 22, 12 (1984), 1745--1747. arXiv:arXiv:1011.1669v3 Google ScholarGoogle ScholarCross RefCross Ref
  89. Utku Köse. 2017. Are We Safe Enough in the Future of Artificial Intelligence? A Discussion on Machine Ethics and Artificial Intelligence Safety. In Scientific Methods in Academic Research and Teaching International Conference. 184--197.Google ScholarGoogle Scholar
  90. Felicitas Kraemer, Kees Van Overveld, and Martin Peterson. 2011. Is there an ethics of algorithms ? Ethics and Information Technology 13, 3 (2011), 251--260. Google ScholarGoogle ScholarDigital LibraryDigital Library
  91. Joshua A. Kroll, Solon Barocas, Edward W. Felten, Joel R. Reidenberg, David G. Robinson, and Harlan Yu. 2017. Accountable Algorithms. University of Pennsylvania Law Review 165 (2017), 633--705. arXiv:arXiv:1011.1669v3 Google ScholarGoogle ScholarCross RefCross Ref
  92. Leer- en Expertisepunt Open Overheid. 2019. Actieplan Open Overheid 2018-2020. https://www.open-overheid.nl/actieplan-open-overheid-2018-2020-open-moet-het-zijn/Google ScholarGoogle Scholar
  93. Bruno Lepri, Nuria Oliver, Emmanuel Letouzé, Alex Pentland, and Patrick Vinck. 2017. Fair, Transparent, and Accountable Algorithmic Decision-making Processes. Philosophy & Technology (2017). Google ScholarGoogle ScholarCross RefCross Ref
  94. Lawrence Lessig. 1999. Code: and other laws of cyberspace. Basic Books, New York.Google ScholarGoogle Scholar
  95. Alessandro Liberati, Douglas G. Altman, Jennifer Tetzlaff, Cynthia Mulrow, Peter C. Gøtzsche, John P.A. Ioannidis, Mike Clarke, P. J. Devereaux, Jos Kleijnen, and David Moher. 2009. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. PLoS Medicine 6, 7 (2009). arXiv:arXiv:1011.1669v3 Google ScholarGoogle ScholarCross RefCross Ref
  96. Michael Lipsky. 1980. Street-level bureaucracy: dilemmas of the individual in public services. Russell Sage Foundation, New York.Google ScholarGoogle Scholar
  97. Katharina Loeber. 2018. Big Data, Algorithmic Regulation, and the History of the Cybersyn Project in Chile, 1971--1973. Social Sciences 7, 4 (2018), 65. Google ScholarGoogle ScholarCross RefCross Ref
  98. Caitlin Lustig, Katie Pine, Bonnie Nardi, Lilly Irani, Min Kyung Lee, Dawn Nafus, and Christian Sandvig. 2016. Algorithmic Authority: The Ethics, Politics, and Economics of Algorithms that Interpret, Decide, and Manage. Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA '16 (2016), 1057--1062. Google ScholarGoogle ScholarDigital LibraryDigital Library
  99. Lei Ma, Zhongqiu Zhang, and Nana Zhang. 2018. Ethical Dilemma of Artificial Intelligence and its Research Progress. In IOP Conference Series: Materials Science and Engineering, Vol. 392. Google ScholarGoogle ScholarCross RefCross Ref
  100. Vidushi Marda. 2018. Artificial intelligence policy in India: A framework for engaging the limits of data-driven decision-making. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences (2018). Google ScholarGoogle ScholarCross RefCross Ref
  101. Kirsten Martin. 2018. Ethical Implications and Accountability of Algorithms. Journal of Business Ethics 0, 0 (2018), 1--16. Google ScholarGoogle ScholarCross RefCross Ref
  102. James McGrath and Ankur Gupta. 2018. Writing a Moral Code: Algorithms for Ethical Reasoning by Humans and Machines. Religions 9, 8 (2018), 240--259.Google ScholarGoogle ScholarCross RefCross Ref
  103. Dan McQuillan. 2018. People's Councils for Ethical Machine Learning. Social Media and Society 4, 2 (2018), 1--10. Google ScholarGoogle ScholarCross RefCross Ref
  104. Eden Medina. 2015. Rethinking algorithmic regulation. Kybernetes 44, 6-7 (2015), 1005--1019. Google ScholarGoogle ScholarCross RefCross Ref
  105. Thomas Metzinger. 2019. Ethics washing made in Europe. https://www.tagesspiegel.de/politik/eu-guidelines-ethics-washing-made-in-europe/24195496.htmlGoogle ScholarGoogle Scholar
  106. Brent Daniel Mittelstadt, Patrick Allo, Mariarosaria Taddeo, Sandra Wachter, and Luciano Floridi. 2016. The ethics of algorithms: Mapping the debate. Big Data & Society 3, 2 (2016), 205395171667967. Google ScholarGoogle ScholarCross RefCross Ref
  107. Annemarie Mol. 2002. The body multiple: ontology in medical practice. Duke University Press, Durham.Google ScholarGoogle Scholar
  108. Joshua New and Daniel Castro. 2018. How Policymakers can foster Algorithmic Accountability. Technical Report. Center for Data Innovation, Washington.Google ScholarGoogle Scholar
  109. New York City. 2018. Automated Decision Systems Task Force. https://www1.nyc.gov/site/adstaskforce/index.pageGoogle ScholarGoogle Scholar
  110. Daniel Neyland. 2016. Bearing Account-able Witness to the Ethical Algorithmic System. Science, Technology, & Human Values 41, 1 (2016), 50--76. Google ScholarGoogle ScholarCross RefCross Ref
  111. Daniel Neyland and Norma Möllers. 2017. Algorithmic IF ... THEN rules and the conditions and consequences of power. Information Communication and Society 20, 1 (2017), 45--62. Google ScholarGoogle ScholarCross RefCross Ref
  112. Helen Nissenbaum. 1994. Computing and accountability. Commun. ACM 37, 1 (1994), 72--80. Google ScholarGoogle ScholarDigital LibraryDigital Library
  113. Dietmar Offenhuber. 2017. Waste is information: infrastructure legibility and governance. MIT Press, Cambridge/London.Google ScholarGoogle Scholar
  114. Partnership on AI. 2019. About. https://www.partnershiponai.org/about/Google ScholarGoogle Scholar
  115. Partnership on AI. 2019. Partnership on AI. https://www.partnershiponai.org/Google ScholarGoogle Scholar
  116. Frank Pasquale. 2015. Digital Star Chamber. https://aeon.co/essays/judge-jury-and-executioner-the-unaccountable-algorithmGoogle ScholarGoogle Scholar
  117. David J. Pauleen, David Rooney, and Ali Intezari. 2017. Big data, little wisdom: trouble brewing? Ethical implications for the information systems discipline. Social Epistemology 31, 4 (2017), 400--416. Google ScholarGoogle ScholarCross RefCross Ref
  118. Wolter Pieters. 2011. Explanation and trust: What to tell the user in security and AI? Ethics and Information Technology 13, 1 (2011), 53--64. Google ScholarGoogle ScholarDigital LibraryDigital Library
  119. Craig Plain. 2007. Build an Affinity for K-J Method. Quality Progress 40, 3 (2007), 88.Google ScholarGoogle Scholar
  120. Iyad Rahwan. 2018. Society-in-the-loop: programming the algorithmic social contract. Ethics and Information Technology 20, 1 (2018), 5--14. arXiv:1707.07232 Google ScholarGoogle ScholarDigital LibraryDigital Library
  121. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. In KDD '16 Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, San Francisco, 1135--1144. arXiv:1602.04938 http://arxiv.org/abs/1602.04938Google ScholarGoogle ScholarDigital LibraryDigital Library
  122. Rijksoverheid. 2014. Besluit SUWI. https://wetten.overheid.nl/BWBR0013267/2019-01-01{#}Hoofdstuk5aGoogle ScholarGoogle Scholar
  123. Alex Rosenblat, Tamara Kneese, and Danah Boyd. 2014. Algorithmic Accountability. In The Social, Cultural & Ethical Dimensions of "Big Data". Data & Society Research Institute, New York. https://datasociety.net/pubs/2014-0317/AlgorithmicAccountabilityPrimer.pdfGoogle ScholarGoogle Scholar
  124. Florian Saurwein, Natascha Just, and Michael Latzer. 2015. Governance of algorithms: Options and limitations. Info 17, 6 (2015), 35--49. Google ScholarGoogle ScholarCross RefCross Ref
  125. Nick Seaver. 2017. Algorithms as culture: Some tactics for the ethnography of algorithmic systems. Big Data & Society 4, 2 (2017), 205395171773810. Google ScholarGoogle ScholarCross RefCross Ref
  126. Security.nl. 2018. Kamervragen over algoritmen voor opsporen bijstandsfraude. https://www.security.nl/posting/557836/Kamervragen+over+algoritmen+voor+opsporen+bijstandsfraude?channel=rssGoogle ScholarGoogle Scholar
  127. Bernd Carsten Stahl and David Wright. 2018. Ethics and Privacy in AI and Big Data: Implementing Responsible Research and Innovation. IEEE Security and Privacy 16, 3 (2018), 26--33. Google ScholarGoogle ScholarCross RefCross Ref
  128. Stats NZ. 2018. Algorithm Assessment Report. Technical Report. Stats NZ, Wellington.Google ScholarGoogle Scholar
  129. Daniel Susser. 2019. Ethics Alone Can't Fix Big Tech. https://slate.com/technology/2019/04/ethics-board-google-ai.htmlGoogle ScholarGoogle Scholar
  130. Jim Torresen. 2018. A Review of Future and Ethical Perspectives of Robotics and AI. Frontiers in Robotics and AI 4, January (2018). Google ScholarGoogle ScholarCross RefCross Ref
  131. Totta Data Lab. 2019. Gemeentelijke Fraudedetectie. https://www.tottadatalab.nl/portfolio-item/gemeentelijke-fraudedetectie/Google ScholarGoogle Scholar
  132. T. Van Ark. 2018. Kamervraag/vragen van het lid Buitenweg (GroenLinks).Google ScholarGoogle Scholar
  133. Iris Van der Tuin and Rick Dolphijn. 2012. "Matter feels, converses, suffers, desires, yearns and remembers" Interview with Karen Barad. In New Materialism: Interviews & Cartographies. Open Humanities Press, Ann Arbor, 48--70.Google ScholarGoogle Scholar
  134. Michael Veale, Max Van Kleek, and Reuben Binns. 2018. Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making. (2018), 1--14. arXiv:1802.01029 Google ScholarGoogle ScholarDigital LibraryDigital Library
  135. Anton Vedder and Laurens Naudts. 2017. Accountability for the use of algorithms in a big data environment. International Review of Law, Computers and Technology 31, 2 (2017), 206--224. Google ScholarGoogle ScholarDigital LibraryDigital Library
  136. Ben Wagner. 2016. Algorithmic regulation and the global default: Shifting norms in Internet technology. Etikk i Praksis 10, 1 (2016), 5--13. Google ScholarGoogle ScholarCross RefCross Ref
  137. Ben Wagner. 2018. Ethics as an Escape from Regulation: From 'ethics-washing' to ethics-shopping? In Being Profiled, Cogitas Ergo Sum, E. Bayamliogu, I. Baraliuc, L.A.W. Janssens, and M. Hildebrandt (Eds.). Amsterdam University Press, Amsterdam, 84--90.Google ScholarGoogle Scholar
  138. Meredith Whittaker, Kate Crawford, Roel Dobbe, Genevieve Fried, Elizabeth Kaziunas, Varoon Mathur, Sarah Myers West, Rashida Richardson, Jason Schultz, and Oscar Schwartz. 2018. AI Now Report 2018. Technical Report. AI Now/New York University, New York.Google ScholarGoogle Scholar
  139. Ben Williamson. 2015. Governing software: networks, databases and algorithmic power in the digital governance of public education. Learning, Media and Technology 40, 1 (2015), 83--105. Google ScholarGoogle ScholarCross RefCross Ref
  140. Karen Yeung. 2017. Algorithmic regulation: A critical interrogation. Regulation & Governance April (2017), 1--19. Google ScholarGoogle ScholarCross RefCross Ref
  141. Karen Yeung. 2017. 'Hypernudge': Big Data as a mode of regulation by design. Information, Communication and Society 20, 1 (2017), 118--136. Google ScholarGoogle ScholarCross RefCross Ref
  142. Han Yu, Zhiqi Shen, Chunyan Miao, Cyril Leung, Victor R. Lesser, and Qiang Yang. 2018. Building ethics into artificial intelligence. In IJCAI International Joint Conference on Artificial Intelligence, Vol. 2018-July. 5527--5533. arXiv:1812.02953v1 Google ScholarGoogle ScholarCross RefCross Ref
  143. Tal Zarsky. 2016. The Trouble with Algorithmic Decisions : An Analytic Road Map to Examine Efficiency and Fairness in Automated and Opaque Decision Making. Science Technology and Human Values 41, 1 (2016), 118--132. Google ScholarGoogle ScholarCross RefCross Ref
  144. Malte Ziewitz. 2016. Governing Algorithms: Myth, Mess, and Methods. Science Technology and Human Values 41, 1 (2016), 3--16. Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. What to account for when accounting for algorithms

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in
          • Article Metrics

            • Downloads (Last 12 months)371
            • Downloads (Last 6 weeks)371

            Other Metrics

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!