Disability can’t be resisted through “legal measures”, instead we must reshape the normative assumption behind algorithms of fairness.
Tilmes 22 – Nicholas Tilmes is in the field of bioethics and researches the intersection of philosophy, psychology, and biology at Cornell University. He is passionate about applying philosophical insights to concrete ethical dilemmas posed by emerging technologies, such as genetic engineering and functional brain imaging, and the relationship between official medical knowledge and the lived experiences of patients. (Nicholas Tilmes, “Disability, fairness, and algorithmic bias in AI recruitment”, Ethics of Information Technology, Springer Link, Article No. 21, 19 April 2022, https://doi-org.proxy.lib.umich.edu/10.1007/s10676-022-09633-2 ) || PZ
Abstraction and fairness Despite our formal knowledge of the sources of algorithmic bias, it can be difficult to identify or take legal measures to combat it since the ways that machine learning AI analyze inputs are hidden in a black box (Sánchez-Monedero et al., ). The black box often frustrates efforts to correct inaccurate models, as it can be unclear if an AI system granted undue weight to non-causal factors or correctly identified a pattern we have yet to grasp. While seeking explanations may illuminate broad strokes of their processing, these algorithms’ inferences might simply be too complex for humans to make sense of (Selbst & Barocas, ). This inscrutability raises the broader question of whether we should turn to AI for inferences about deeply normative, difficult-to-quantify issues such as built inaccessibility and disability status. In the face of concerns about differing notions of bias, legal uncertainty, and the black box, vendors operationalized demands that AI delivers fair outcomes for protected classes. These fair-ML methods seek to translate bias into easily quantifiable terms that algorithms can understand such as error rates (Agrawal, ). For instance, by defining fairness as a mathematical constraint and optimizing for it or modifying training data until outputs conform to fairness standards, fair-ML can help mitigate disparities stemming from technical errors such as flawed input–output relations or training data. While fairness is operationalized in countless ways, most fall into one of three approaches (Corbett-Davies & Goel, ). First, anti-classification, also called fairness through unawareness, omits data about membership in protected classes as well as any traits associated with them (Zafar et al., ). For instance, one might design an AI that explicitly cannot access information about applicants’ disability status. Second, calibration holds that people who an AI scores similarly ought to end up having similar outcomes (Dieterich et al., ). For example, imagine an AI that equalizes false omission rates and negative predictive values so applicants with similar scores are equally likely to become poor employees, whether they are disabled or not. Third, classification parity defines fairness as a matter of equalizing a predictive measure across groups (Chouldechova et al., ). For example, an AI might equalize error rates such that applicants who end up succeeding are equally likely to be selected, independent of disability status. Fair-ML’s understanding of fairness reflects neither procedural nor normative concepts of the term but instead is a formal, mathematical criteria that AI outputs should not stray from (Ochigame, ). On this view, algorithms are biased if they violate a certain standard, the satisfaction of which is taken to indicate fairness (Mulligan et al., ). While conforming to these standards often is valuable, doing so ought not be taken as sufficient for securing fairness in the normative sense. Instead, fair-ML methods are best understood as pragmatic ways to render bias tractable despite the black box by replacing messy legal and normative issues with formal rules (Bennett et al., ; Green & Viljoen, ). As such, these methods can be effective at addressing mathematical errors, such as inaccurate training datasets and input–output relations. Yet, while fair-ML can reduce certain outcome disparities, it lacks a principled way of detecting bias. The gap between formal and normative concepts of fairness was brought to light when researchers at ProPublica reported that COMPAS, an algorithm which predicted recidivism risk, discriminated against black defendants (Angwin et al., ). Specifically, it had lower false negative and higher false positive rates for black defendants than for white ones, which violated classification parity. COMPAS’ developers, Northpointe (now Equivant), replied by arguing that error rates are flawed measures of fairness and noting that COMPAS exhibited equal predictive values for black and white defendants, thus ensuring calibration (Dieterich et al., ). In this dispute, whether COMPAS was racially biased seemingly hinged neither on legal procedure nor normative principle but on various fair-ML criteria. While anti-classification, calibration, and classification parity all aim to operationalize distinct notions of fairness, it is mathematically impossible to satisfy each of these definitions simultaneously in the real world (Jacobs & Wallach, ). Research has found that these standards could be reconciled only if the recidivism rate were uniform across groups, the AI was entirely accurate, or some groups never faced negative outcomes (Barocas et al., ; Chouldechova, ). Similarly, in the context of hiring, if disabled people performed worse than average on a given pre-employment assessment, calibrating employability predictions would shoulder them with higher false negative and lower false positive rates. This suggests that algorithms inevitably are unfair in that they will violate at least one fair-ML approach and reflects the conceptual tensions between the normative notions of fairness that underpin each of these approaches. Although fair-ML treats bias as the product of inaccurate input–output relations and training data, decisions about how to reduce bias are normative issues, not technical ones. Choices about which kind of algorithmic fairness to implement—anti-classification, calibration, classification parity, and so on—are intrinsically value-laden because they shape what AI designers consider to be a biased outcome. As such, it is unclear how to assess competing fair-ML strategies without appealing to normative reasons. For instance, since letting some criminals go is preferable to punishing innocents, data scientists predicting recidivism risk should be more concerned with false positives than false negatives. Yet, equalizing false negatives seems more important for AI hiring because approved candidates still face additional interviews while those screened out have little redress. At the same time, post-hoc measures for uncovering false negatives in hiring tests, such as approving a random set of lower-ranked applicants, would be much more difficult to justify in the context of recidivism. Narrow focus on abstract proxies for equality denies one the information required to evaluate the merits of different fairness standards. To define fairness as a formal constraint divorced from normative commitments and social context is to commit a category error. The difficulty of using solely technical means to secure normative ends points to broader issues with fair-ML, raising questions about how we should understand disability and fairness in light of structural ableism.