Data is only one aspect of AI bias- fixating on improving the training obscures other forms of algorithmic violence

(Annette Zimmermann is Lecturer in Philosophy at the University of York and Technology & Human Rights Fellow at the Carr Center for Human Rights Policy at the Harvard Kennedy School. 7-21)

A third obstacle is a too limited conception of the ways AI can be harmful or unjust. In many familiar examples of algorithmic injustice, accuracy is distributed unequally across different demographic groups. Criminal recidivism risk prediction tools, for instance, have been shown to have significantly higher false positive rates for Black defendants than for white defendants. Such examples have elicited significant ethical reflection and controversy, helping to call attention to the risks of AI. But we must also recognize that AI tools can be unjust even if they do not rely on biased training data or suffer from disparate distributions of error rates across demographic groups. For one thing, even if developers are well intentioned, the consequences of implementing a particular algorithmic solution in a specific social context may be unjust, because algorithmic outputs reflect and exacerbate social biases and inequalities. It may also be that the goal of an AI tool is simply not just to begin with, regardless—or even indeed because of—the tool’s accuracy. Consider Megvii, a Chinese company that used its facial recognition technology in collaboration with Huawei, the tech giant, to test a “Uighur alarm” tool designed to recognize the faces of members of the Uighur minority and alert the police. Here it is the very goal of the technology that fails to be morally legitimate. A related problem is that human decision-makers, prone to automation bias, may fail to scrutinize algorithmic classifications, taking ostensibly neutral and objective algorithmic outputs as a given instead of interrogating them critically. In still other cases, it may be the logic of the technology that is objectionable, leading to what philosophers call “expressive harm”: the use of particular categories and classifications in AI tools can convey a demeaning, harmful message, which becomes unjust in light of prevalent social norms, assumptions, and experiences. Tools that attempt to deduce sexual orientation or other personality traits from one’s physical appearance, for example, may contribute to reinforcing the harmful message not only that it is possible to “look like a criminal,” or to “look gay,” but also that it is valid to infer personal characteristics and future behavior from the way a person looks. The upshot of these various examples is that the potential harms of AI range far beyond datasets and error rates.