Bias in computer systems

Reviewer: Darin Chardin Savage

Friedman and Nissenbaum present a fascinating overview of bias within computer systems. The variety of systems surveyed—banking, commerce, computer science, education, medicine, and law—allows for both a broad-ranging and poignant discussion of bias, which, if undetected, may have serious and unfair consequences. The rapid dissemination and ready acceptance of computer algorithms means that biases may easily affect a large number of unsuspecting people. The difficulty, the authors observe, is in identifying and describing the nature of the bias, for this issue has not been comprehensively addressed in the computer literature. Most experts of social bias and discrimination approach the issue from a legal or philosophical background and may not be equipped to interpret and translate issues from a technological standpoint. This places the onus of responsibility on the technological professions themselves, and this paper provides a very thoughtful and thorough framework for meeting this responsibility. From the variety of cases, the authors are able to identify three forms of bias—preexisting, technical, and emergent. The descriptions of how these biases are integral to the cases seem right on target. My only wish as a reader was for a greater quantification of the actual impact of the bias, whether in economic, institutional, or social terms. The authors do give one example that alludes to a quantification of impact by noting that 90 percent of airline reservations are pulled from the first screen of the database. Therefore, those airlines listed only on subsequent screens would be at a great disadvantage. I wanted to know, however, to what extent the businesses are economically disadvantaged by this arrangement and how many people were affected by the BNAP immigration program or the National Resident Match Program. Perhaps the data are not available or are difficult to determine, but the issues raised in the paper evoke an expectation for this kind of data, which, if presented, would highlight the significance of bias. The authors do show unequivocally that bias exists in the case examples, and where and how it exists within the systems. Their rigorous definition of what constitutes bias and their call for standards of unbiased programming that meet clearly defined criteria offer a useful platform from which to address the complex issues of ethics and equity in the information age.