The impact is Algorithmic Ablism – AI will always be indebted to the structures of ablism which influence ethical considerations behind AI development.
Tilmes 22 – Nicholas Tilmes is in the field of bioethics and researches the intersection of philosophy, psychology, and biology at Cornell University. He is passionate about applying philosophical insights to concrete ethical dilemmas posed by emerging technologies, such as genetic engineering and functional brain imaging, and the relationship between official medical knowledge and the lived experiences of patients. (Nicholas Tilmes, “Disability, fairness, and algorithmic bias in AI recruitment”, Ethics of Information Technology, Springer Link, Article No. 21, 19 April 2022, https://doi-org.proxy.lib.umich.edu/10.1007/s10676-022-09633-2 ) || PZ
Algorithmic ableism To optimize for fairness and make data portable despite the black box, disability status often is modelled as a one-dimensional variable. This tendency to abstract away environmental and social context artificially limits categories of disability and obscures the process of measuring it. In doing so, AI design unintentionally can reinforce both the form and content of the medical model of disability. By omitting the value-laden decisions involved in operationalizing disability, fair-ML flattens variations in the intensities and kinds of people’s disabilities and further marginalizes those at the intersections. Analyzing the normative commitments of this approach highlights the importance of considering a wider array of ethical concerns beyond training data and input–output relations. When data scientists tackle open-ended goals such as diversity in hiring, they must translate contested models of disability into a single target that can be measured and adjusted in the black box (Selbst et al., ). As such, bias mitigation requires rendering concepts like protected class status amenable to quantification so discrete weights can be ascribed to them. For example, Pymetrics adjusts its AI to satisfy the 4/5ths rule by assessing if scores differ based on neatly defined categories of gender, race, and age (Yoo, ). HireVue, which uses natural language processing to analyze the content of applicants’ speech during automated interviews and gamified tasks, makes similar claims (Zuloaga, ). As part of this process, HireVue explicitly designs their questions in order to elicit readily quantifiable responses (Larsen, ). AI vendors often idealize abstraction since removing context-specific data can help algorithms function more accurately across a variety of settings (Selbst et al., ). However, accuracy is hampered by the fact that most data about disabled people are not comprehensive, without comparability, and low quality (Altman & Rasch, ; Maritz et al., ). As a result, gathering more disability data and designing a shared system for classifying it plays an important role in bias reduction. Since optimizing for fairness can reduce accuracy, collecting high-quality disability data could help mitigate bias stemming from unrepresentative datasets (Chen et al., ). While better representation in training data is necessary to secure just outcomes, disclosing one’s disability status can lead to data misuse and expose one to further discrimination. However, even if people could provide information about their disabilities without fear of repercussion, what it means to identify as disabled is not always straightforward. It is not simply that some disabled people are understandably reluctant to disclose, but that if and how they identify depends on the model of disability and type of measurement at hand. Researchers have found that how people define their disabilities—if they identify as disabled at all—is predicted by a wide array of factors, such as the type of inquiry (Brown & Broido, ), intensity and stigma (Bogart et al., ), and environmental context (Santuzzi & Waltz, ). A host of other factors, such access to an official diagnosis, internalized self-doubt, and whether one’s disability is (in)visible also plausibly affect self-identification. Other studies suggest that whether and how one identifies impacts the types of bias encountered (Nario-Redmond et al., ), methods used to navigate them (Nario-Redmond et al., ), and levels of self-esteem (Bogart, ). These different understandings of disability not only call for distinct types of measurement but also are correlated with distinct outcomes. For example, official medical diagnoses seem to be good predictors of access to health care while self-identification likely is a better proxy for group and political affiliation. Data scientists cannot operationalize disability status as a single, binary variable because the category’s nature and boundaries are not self-evident even among people who seem to fit within it. While definitions of disability are fiercely contested, the ways in which disability data are collected and become legible to AI lend disproportionate weight to the medical model of disability. Given that disclosing disability status can open one up to further discrimination, disability is often quantified only when the situation demands proof of it, e.g., to make an insurance claim, secure a diagnosis, or receive treatment. Aspects of other models of disability, such as social attitudes and built environment, are difficult to measure and, when people have a chance to self-report, they typically are limited to the binary categories of census and actuarial forms. As a result, data scientists seeking to make use of what little disability data exists often turn to information gathered and stored in a narrow range of contexts and formats. In turn, the processes of selecting proxies, generating labels, and curating datasets tend to translate disability into concepts amenable to the medical model. That is, algorithms end up assessing disabled people through a lens that defines disability as a biological dysfunction isolated from structural ableism. Abstraction deprives AI designers of the information required to operationalize disability status as anything but a biomedical dysfunction. One need not deny disability’s material aspects to recognize that adopting an entirely medical understanding of disability crowds out analysis of the socio-political processes that inform it. After all, that disability becomes legible to AI in terms of measurable diagnoses does not entail that AI is objective but highlights its alignment with the medical model. For example, if disabled people score lower on a given pre-employment test, to operationalize disability without accounting for the ableist structures at the root of that performance gap cannot help but attribute that gap to bodily failures. Similarly, once social context is omitted, one can no longer appeal to hiring bias to explain why irregular eye contact adversely impacts employability, shouldering disabled applicants with that responsibility instead. Because disability cannot be reduced to biological facts, abstracting away information about one’s built environment and social context is inaccurate and renders structural ableism benign. However, efforts to remove context from or quantify aspects of disability sheds light on the value-laden nature of ostensibly objective AI design. After all, the notion that one’s competence can be understood apart from the setting it occurs in is hardly a neutral one. For instance, some researchers studying performance assessments argue competence can only be assessed in situ since it is intrinsically contextual and shifts in response to dynamic, complex workplace environments (Cate & Regehr, ; Govaerts & van der Vleuten, ). To focus solely on quantitative aspects of performance risks omitting information regarding, e.g., interactions between chronic fatigue and irregular work schedules, or moldy buildings and asthma flare ups. That such factors are often removed from AI models while more quantifiable traits are treated as legitimate markers of ‘fit’ is not objective but shaped by ableist norms. For example, while studies show that disabled people perform similarly on many aspects of employability (Houtenville & Kalargyrou, ; Lindsay et al., ), automated interviews often place great stock in easily measured factors such as a measured tone and eye contact (Whittaker et al., ). As such, assessing applicants primarily in terms of abstract, quantitative factors does not preserve objectivity so much as obscure the underlying assumptions of AI designers. By evaluating hiring outcomes for disabled people based on factors that do not resist quantification, fair-ML omits more nebulous aspects of disability and makes it difficult to identify the ways in which structural ableism shapes data. Consider that while sentencing decisions hinge on many factors, recidivism risk has been measured disproportionately. Researchers have noted that this focus disposes algorithms such as COMPAS to assess sentences in ways that prioritize incapacitation over, e.g., retribution and rehabilitation (Green, ; Selbst et al., ). Similarly, imagine a hiring algorithm that predicts employability based entirely on quantifiable traits, like employment gaps, requests for medical leave, or expected cost of reasonable accommodations. Even if designers omitted explicit references to disability, such an AI would subtly emphasize correlations to disability status without providing important context or explanation. By prioritizing quantifiable aspects of disability over those not so easily measured, fair-ML configures and restricts our understanding of and responses to algorithmic bias. While inclusive data collection practices might better capture the contexts disabled people navigate, they would struggle to assign static values to them due to disability’s dynamic and shifting nature. Given that AI designers assign and modify discrete weights to features in their models, algorithms evaluate populations in terms of several snapshots taken at certain times instead of gradually emerging outcomes, they can end up entrenching injustice. For example, hiring AI that select applicants based on similarity to previous employees may not adapt to increases in workplace diversity, thus perpetuating bias. Moreover, since disabilities such as chronic pain, asthma, depression, and so on tend to wax and wane day-by-day or over one’s life, modelling people’s capacities in terms of a single binary dimension imposes artificial fixity. One’s performance on a pre-employment assessment cannot be understood in isolation from whether, to give an example, their chronic pain is flaring up that week. It is not just that experiences of disability are inseparable from their contexts, but that disabilities are embodied in shifting ways that cannot be captured by one variable. Although two people might receive diagnoses for the same disability, how they experience that disability and the degree to which it impacts their lives might vary immensely. By removing context and assigning discrete values to disability status, algorithms flatten differences between and within disabled people. The difficulty of operationalizing disability in AI raises similar concerns about membership in other protected classes, many of which also are modelled as binary variables to facilitate de-biasing. Although the sheer multitude of disabilities and historical shifts in their boundaries shed light on the category’s instability, disability is deeply entangled with other aspects of identity. Race, gender, class, and so on intersect and overlap with disability in ways that generate distinct forms of structural discrimination that cannot be reduced to their component parts (Crenshaw, ). Crucially, intersectionality does not entail that disability is neatly analogous with other identities but that the institutions, structures, and norms shaping them are co-constitutive and mutually reinforcing (Erevelles, ). Given that fair-ML optimizes for fairness with respect to protected attributes in isolation, it artificially separates intertwined aspects of identity, which disproportionately impacts those at their intersections. For example, research has found that facial recognition algorithms exhibit higher error rates when classifying the faces of darker-skinned women than could be explained by either race or gender alone (Buolamwini & Gebru, ). Initiatives promoting more inclusive datasets and fair outcomes must reckon with the fact that classifications and the authority to shape them are mediated by discriminatory structures. Indeed, whether one receives legal recognition as disabled is impacted by intersections with these structures, as multiply marginalized disabled people may struggle to secure access to the category itself. For example, given the price of healthcare, the ability to obtain a diagnosis from the medical institutions that AI researchers often collect data from is unevenly distributed. Similarly, certification of disability status is mediated by other biases, such as many physicians’ undue skepticism about women’s experiences of chronic pain (Mingus, ) and the false belief that black people have an unusually high pain tolerance (Erevelles, ). Given that intersecting axes of oppression impact one’s ability to claim disability status, the few disabled people officially recognized as such in training datasets seem likely to be disproportionately wealthy, white, and male. Data scientists cannot merely tack disability on to a list of isolated protected classes since their intersections affect who ‘counts’ as disabled. Instead, they should attend to the ways in which subjective decisions and structural hierarchies impact AI design, situating experiences of disability in their context.