Tilmes, 2022

Although fair-ML treats bias as the product of inaccurate input–output relations and training data, decisions about how to reduce bias are normative issues, not technical ones. Choices about which kind of algorithmic fairness to implement—anti-classification, calibration, classification parity, and so on—are intrinsically value-laden because they shape what AI designers consider to be a biased outcome. As such, it is unclear how to assess competing fair-ML strategies without appealing to normative reasons. For instance, since letting some criminals go is preferable to punishing innocents, data scientists predicting recidivism risk should be more concerned with false positives than false negatives. Yet, equalizing false negatives seems more important for AI hiring because approved candidates still face additional interviews while those screened out have little redress. At the same time, post-hoc measures for uncovering false negatives in hiring tests, such as approving a random set of lower-ranked applicants, would be much more difficult to justify in the context of recidivism. Narrow focus on abstract proxies for equality denies one the information required to evaluate the merits of different fairness standards. To define fairness as a formal constraint divorced from normative commitments and social context is to commit a category error. The difficulty of using solely technical means to secure normative ends points to broader issues with fair-ML, raising questions about how we should understand disability and fairness in light of structural ableism. Algorithmic ableism To optimize for fairness and make data portable despite the black box, disability status often is modelled as a one-dimensional variable. This tendency to abstract away environmental and social context artificially limits categories of disability and obscures the process of measuring it. In doing so, AI design unintentionally can reinforce both the form and content of the medical model of disability. By omitting the value-laden decisions involved in operationalizing disability, fair-ML flattens variations in the intensities and kinds of people’s disabilities and further marginalizes those at the intersections. Analyzing the normative commitments of this approach highlights the importance of considering a wider array of ethical concerns beyond training data and input–output relations. When data scientists tackle open-ended goals such as diversity in hiring, they must translate contested models of disability into a single target that can be measured and adjusted in the black box (Selbst et al., ). As such, bias mitigation requires rendering concepts like protected class status amenable to quantification so discrete weights can be ascribed to them. For example, Pymetrics adjusts its AI to satisfy the 4/5ths rule by assessing if scores differ based on neatly defined categories of gender, race, and age (Yoo, ). HireVue, which uses natural language processing to analyze the content of applicants’ speech during automated interviews and gamified tasks, makes similar claims (Zuloaga, ). As part of this process, HireVue explicitly designs their questions in order to elicit readily quantifiable responses (Larsen, ). AI vendors often idealize abstraction since removing context-specific data can help algorithms function more accurately across a variety of settings (Selbst et al., ). However, accuracy is hampered by the fact that most data about disabled people are not comprehensive, without comparability, and low quality (Altman & Rasch, ; Maritz et al., ). As a result, gathering more disability data and designing a shared system for classifying it plays an important role in bias reduction. Since optimizing for fairness can reduce accuracy, collecting high-quality disability data could help mitigate bias stemming from unrepresentative datasets (Chen et al., ). While better representation in training data is necessary to secure just outcomes, disclosing one’s disability status can lead to data misuse and expose one to further discrimination. However, even if people could provide information about their disabilities without fear of repercussion, what it means to identify as disabled is not always straightforward. It is not simply that some disabled people are understandably reluctant to disclose, but that if and how they identify depends on the model of disability and type of measurement at hand. Researchers have found that how people define their disabilities—if they identify as disabled at all—is predicted by a wide array of factors, such as the type of inquiry (Brown & Broido, ), intensity and stigma (Bogart et al., ), and environmental context (Santuzzi & Waltz, ). A host of other factors, such access to an official diagnosis, internalized self-doubt, and whether one’s disability is (in)visible also plausibly affect self-identification. Other studies suggest that whether and how one identifies impacts the types of bias encountered (Nario-Redmond et al., ), methods used to navigate them (Nario-Redmond et al., ), and levels of self-esteem (Bogart, ). These different understandings of disability not only call for distinct types of measurement but also are correlated with distinct outcomes. For example, official medical diagnoses seem to be good predictors of access to health care while self-identification likely is a better proxy for group and political affiliation. Data scientists cannot operationalize disability status as a single, binary variable because the category’s nature and boundaries are not self-evident even among people who seem to fit within it. While definitions of disability are fiercely contested, the ways in which disability data are collected and become legible to AI lend disproportionate weight to the medical model of disability. Given that disclosing disability status can open one up to further discrimination, disability is often quantified only when the situation demands proof of it, e.g., to make an insurance claim, secure a diagnosis, or receive treatment. Aspects of other models of disability, such as social attitudes and built environment, are difficult to measure and, when people have a chance to self-report, they typically are limited to the binary categories of census and actuarial forms. As a result, data scientists seeking to make use of what little disability data exists often turn to information gathered and stored in a narrow range of contexts and formats. In turn, the processes of selecting proxies, generating labels, and curating datasets tend to translate disability into concepts amenable to the medical model. That is, algorithms end up assessing disabled people through a lens that defines disability as a biological dysfunction isolated from structural ableism. Abstraction deprives AI designers of the information required to operationalize disability status as anything but a biomedical dysfunction. One need not deny disability’s material aspects to recognize that adopting an entirely medical understanding of disability crowds out analysis of the socio-political processes that inform it. After all, that disability becomes legible to AI in terms of measurable diagnoses does not entail that AI is objective but highlights its alignment with the medical model. For example, if disabled people score lower on a given pre-employment test, to operationalize disability without accounting for the ableist structures at the root of that performance gap cannot help but attribute that gap to bodily failures. Similarly, once social context is omitted, one can no longer appeal to hiring bias to explain why irregular eye contact adversely impacts employability, shouldering disabled applicants with that responsibility instead. Because disability cannot be reduced to biological facts, abstracting away information about one’s built environment and social context is inaccurate and renders structural ableism benign. However, efforts to remove context from or quantify aspects of disability sheds light on the value-laden nature of ostensibly objective AI design. After all, the notion that one’s competence can be understood apart from the setting it occurs in is hardly a neutral one. For instance, some researchers studying performance assessments argue competence can only be assessed in situ since it is intrinsically contextual and shifts in response to dynamic, complex workplace environments (Cate & Regehr, ; Govaerts & van der Vleuten, ). To focus solely on quantitative aspects of performance risks omitting information regarding, e.g., interactions between chronic fatigue and irregular work schedules, or moldy buildings and asthma flare ups. That such factors are often removed from AI models while more quantifiable traits are treated as legitimate markers of ‘fit’ is not objective but shaped by ableist norms. For example, while studies show that disabled people perform similarly on many aspects of employability (Houtenville & Kalargyrou, ; Lindsay et al., ), automated interviews often place great stock in easily measured factors such as a measured tone and eye contact (Whittaker et al., ). As such, assessing applicants primarily in terms of abstract, quantitative factors does not preserve objectivity so much as obscure the underlying assumptions of AI designers. By evaluating hiring outcomes for disabled people based on factors that do not resist quantification, fair-ML omits more nebulous aspects of disability and makes it difficult to identify the ways in which structural ableism shapes data. Consider that while sentencing decisions hinge on many factors, recidivism risk has been measured disproportionately. Researchers have noted that this focus disposes algorithms such as COMPAS to assess sentences in ways that prioritize incapacitation over, e.g., retribution and rehabilitation (Green, ; Selbst et al., ). Similarly, imagine a hiring algorithm that predicts employability based entirely on quantifiable traits, like employment gaps, requests for medical leave, or expected cost of reasonable accommodations. Even if designers omitted explicit references to disability, such an AI would subtly emphasize correlations to disability status without providing important context or explanation. By prioritizing quantifiable aspects of disability over those not so easily measured, fair-ML configures and restricts our understanding of and responses to algorithmic bias. While inclusive data collection practices might better capture the contexts disabled people navigate, they would struggle to assign static values to them due to disability’s dynamic and shifting nature. Given that AI designers assign and modify discrete weights to features in their models, algorithms evaluate populations in terms of several snapshots taken at certain times instead of gradually emerging outcomes, they can end up entrenching injustice. For example, hiring AI that select applicants based on similarity to previous employees may not adapt to increases in workplace diversity, thus perpetuating bias. Moreover, since disabilities such as chronic pain, asthma, depression, and so on tend to wax and wane day-by-day or over one’s life, modelling people’s capacities in terms of a single binary dimension imposes artificial fixity. One’s performance on a pre-employment assessment cannot be understood in isolation from whether, to give an example, their chronic pain is flaring up that week. It is not just that experiences of disability are inseparable from their contexts, but that disabilities are embodied in shifting ways that cannot be captured by one variable. Although two people might receive diagnoses for the same disability, how they experience that disability and the degree to which it impacts their lives might vary immensely. By removing context and assigning discrete values to disability status, algorithms flatten differences between and within disabled people. The difficulty of operationalizing disability in AI raises similar concerns about membership in other protected classes, many of which also are modelled as binary variables to facilitate de-biasing. Although the sheer multitude of disabilities and historical shifts in their boundaries shed light on the category’s instability, disability is deeply entangled with other aspects of identity. Race, gender, class, and so on intersect and overlap with disability in ways that generate distinct forms of structural discrimination that cannot be reduced to their component parts (Crenshaw, ). Crucially, intersectionality does not entail that disability is neatly analogous with other identities but that the institutions, structures, and norms shaping them are co-constitutive and mutually reinforcing (Erevelles, ). Given that fair-ML optimizes for fairness with respect to protected attributes in isolation, it artificially separates intertwined aspects of identity, which disproportionately impacts those at their intersections. For example, research has found that facial recognition algorithms exhibit higher error rates when classifying the faces of darker-skinned women than could be explained by either race or gender alone (Buolamwini & Gebru, ). Initiatives promoting more inclusive datasets and fair outcomes must reckon with the fact that classifications and the authority to shape them are mediated by discriminatory structures. Indeed, whether one receives legal recognition as disabled is impacted by intersections with these structures, as multiply marginalized disabled people may struggle to secure access to the category itself. For example, given the price of healthcare, the ability to obtain a diagnosis from the medical institutions that AI researchers often collect data from is unevenly distributed. Similarly, certification of disability status is mediated by other biases, such as many physicians’ undue skepticism about women’s experiences of chronic pain (Mingus, ) and the false belief that black people have an unusually high pain tolerance (Erevelles, ). Given that intersecting axes of oppression impact one’s ability to claim disability status, the few disabled people officially recognized as such in training datasets seem likely to be disproportionately wealthy, white, and male. Data scientists cannot merely tack disability on to a list of isolated protected classes since their intersections affect who ‘counts’ as disabled. Instead, they should attend to the ways in which subjective decisions and structural hierarchies impact AI design, situating experiences of disability in their context. Disability justice Rather than attempting to promote fairness solely through technical adjustments to training data and input–output relations, data scientists should take up a disability justice approach. This view acknowledges that factors such as error rates and accuracy are necessary but insufficient to understand, identify, and address algorithmic bias. Disability justice serves as a framework for reasoning about how ableist structures and norms subtly configure and restrict the ostensibly objective aspects of AI design. A deeper analysis of disability and capacity not only draws attention to the value-laden nature of design processes but illuminates how data analytics and machine learning help to define and redefine those concepts. Instead, a disability justice approach suggests drawing a wider circle of considerations relevant to AI ethics; centering analysis of assumptions and values in design; closing gaps between stakeholders; and pursuing policies that empower further activism. Disability justice calls into question fair-ML’s dogged quest for objectivity and failure to confront broader socio-political norms, demanding that data scientists attend to the subjective choices intrinsic to algorithm design. This shifts from a sole focus on technical aspects of design—such as input–output relations and training data—and expands the problem space to include the social responses and outcomes that algorithms generate. While data scientists need not abandon technical considerations, these decisions must follow from an analysis of the normative choices that go into, e.g., generating classifications, formulating problems, labelling data, and selecting proxy variables. That is, designers should take a broader socio-technical perspective, which considers systems as not merely technological but enmeshed with embodied, organizational, social, and political factors (Simon et al., ; van der Bijl-Brouwer & Malcom, ). By considering how these dynamics interact and affect AI design together instead of assessing them in isolation, data scientists can better conceptualize, identify, and respond to biased algorithms. For instance, by reflecting on values woven into targets like ‘fit,’ one can parse out legitimate indicators of employability, such as word choice, from ones steeped in ableist norms and unrelated to performance, such as speech patterns and tone. Widening the circle of factors considered relevant to ethical AI design beyond data and input–output relations not only alters design processes but also renders thinkable a wider range of alternatives. Looking beyond the optimization of technical fairness criteria intervenes in solutionism and reveals a variety of other mechanisms for combatting bias and promoting social transformation. Notably, this gives us the tools to situate algorithms in a socio-technical context and raise the question of whether algorithms are appropriate instruments to measure and shape certain domains in the first place. For instance, given concerns about algorithms’ ability to capture shifting, contextual aspects of disability, what amount of discretion should they have over disabled people’s hiring outcomes, if any? While simply omitting groups from algorithmic models would exacerbate inequality, there may be reason to be much more skeptical of their inferences. Already, protests by Google employees about collaboration with the Departments of Defense and Homeland Security on Projects JEDI and MAVEN (Wakabayashi & Shane, ) and pledges by computer science students to refuse work at Palantir in light of its cooperation with Immigrations and Customs Enforcement (Birnbaum, ) point to the ways in which data scientists can affect change beyond the algorithm itself. As such, disability justice indicates that we ought to expand the scope of AI ethics and situate algorithms in their socio-technical context. Moreover, a disability justice approach suggests that data scientists should critically analyze the ways in which disability is defined, measured, and labelled. Given that most data about disability are medical, inferences from them typically are limited to the medical model, reducing disability to a biological dysfunction. Yet, by drawing on disability scholarship, we can uncover the ways in which these categories shift and trace their roots to ableist assumptions. For instance, consider proposals to increase the amount and quality of disability data by drawing from people’s electronic health records. While well-intentioned, these efforts would further entrench the medical model of disability, leave little room for self-identification, and underrepresent disabled people who lack access to health care. Making explicit the ways in which algorithms model disability creates space to imagine it differently through methods like value-sensitive design (Friedman & Hendry, ; Simon et al., ). Such an approach melds conceptual, empirical, and technical elements, e.g., assessing models of disability in terms of their assumptions, impact on data gathering, relationship to outcomes, and so on. In so doing, data scientists could better attend to the ways that different models influence people’s claims about their disabilities and how those claims are mediated by intersecting hierarchies of class, race, and gender. By investigating how different definitions and measures of disability shape who ‘counts’ as disabled, data scientists can intervene in these ongoing omissions. This requires altering how disabilities are operationalized so they are defined not solely as medical diagnoses but also in terms of how people navigate institutions, norms, and each other. While fair-ML idealizes abstraction so that data can be applied across contexts, (in)capacities emerge from a multitude of contextual factors like competing values and built environments. Instead, a disability justice framework suggests that an algorithm’s accuracy does not always trade off with its ability to accommodate contingency because many disabilities are intrinsically situational and resist quantification. Implementing this approach might involve, e.g., assigning more weight to contextual factors, assessing how aspects of a given disability shift over time, and designing scales that are sensitive to social values and attitudes. For example, some researchers have demonstrated that disability data can be disaggregated to more easily study the variables that cooccur with certain disabilities or contexts (Brown & Broido, ; Peña et al., ). Others have called for analyzing how people’s capacities expand and contract in relation to built accessibility, office culture, and the type of labor at hand (Bennett et al., ). This suggests that data scientists could model disability differently by altering processes of data collection, label generation, and proxy selection so as not to abstract away its context. In addition, disability justice demands more fundamental shifts in access, communication, and ownership across the AI design pipeline. Indeed, the biases that hiring algorithms exhibit are inseparable from developers’ sole focus on employer values and widening asymmetries between firms and applicants. Although applicants’ ability to secure jobs is shaped by these AI, they cannot examine or offer input on them easily, if at all. Expanding opportunities for these engagements and removing barriers to access is essential to designing more equitable algorithms. Already, initiatives like Access Computing offer accessible training in computer science (Trewin et al., ) and groups like (Dis)Ability in AI provide support to disabled people at conferences about AI (Whittaker et al., ). Despite efforts to increase disability representation at design forums and in training datasets, the mere presence of disability is not enough to transform existing institutions and restore trust. That is, vendors must design with disabled people, not merely for them. As the scope of design expands, data scientists ought to draw insights from other disciplines, including disability studies and socio-technical design, and team up with a wider range of stakeholders. This process cannot simply tack on these perspectives but must integrate and consult with them throughout the design pipeline. By collaborating with a diverse array of disciplines and voices, data scientists could better identify stakeholders in AI hiring, explore their values and needs, study how AI might implicate them, and involve them in design processes. This bears similarities to a participatory design perspective, in which workers are empowered by directly influencing the design of technologies they will take up (Korsgaard et al., ). Using a variety of techniques—from prototyping and mock-ups to future-oriented workshops and scenarios—stakeholders reflect on the situation, imagine innovative solutions, and transform those speculative approaches into realistic, concrete alternatives. This might involve, to give an example, asking stakeholders to evaluate whether automated interviews that assess affect and gestures align with their values, what they communicate about disabled applicants, and how they might be navigated. In parallel, insights from disability studies suggest granting disabled people’s perspectives significant weight in debates about, e.g., generating labels and selecting proxies for disability status. For instance, developers might look to data platform co-ops operated by people with rare disabilities, in which they oversee the use of their data and share profits (Scholz, ; Treviranus, ), to inform their own decisions about responsible data use. By working alongside these other disciplines and voices from the start of the design process, data scientists could help bridge gaps in knowledge, foster a common conceptual language, and develop new research methods. Beyond expanding the scope of AI ethics concerns, redefining design values, and changing stakeholder dynamics, a disability justice approach suggests pursuing policy measures that can empower data activism. While legislation often lags behind advances in computing, existing regulatory uncertainty allows AI vendors to selectively endorse transparency and fairness criteria in ways that preserve existing arrangements. Greater transparency could help promote the conditions necessary for activists and data scientists alike to study, uncover, and contest algorithmic bias. Despite the explanatory difficulties associated with the black box, policies ought to be introduced mandating that AI developers regularly audit and disclose the purview, inputs and outputs, and limitations of their algorithms. For example, some have suggested requiring employers to make public the automated inferences they make about applicants (Hoffman, ) and tasking them with retaining detailed records of their bias mitigation procedures (Hoffman, ). Developing legislation to establish minimum standards for data transparency and disclosure of bias mitigation measures—even for opaque machine learning AI—is necessary to identify and make claims against discriminatory algorithms. In addition to setting baselines for transparent design and implementation, recognizing how ableist structures can affect people’s identification with and disclosure of disability status highlights the importance of more robust privacy regulations. Existing protections are ill-equipped to grapple with the ethical risks posed by automated hiring tools or predictions of disability status. The ADA (1990) only shields those who are disabled, treated as disabled, or were disabled, while GINA (2008) and HIPAA (OCR, ) protect health data while it is being stored, transferred, and analyzed by certain classes of covered entities. However, since these laws neither cover AI vendors and data brokers nor explicitly forbid discrimination based on inferred risk factors for disabilities, AI can circumvent them (Givens & Morris, ). Similarly, while the ADA prohibits discrimination in places of public accommodation, it does not explicitly include algorithms, social media platforms, and so on (Rothstein, ). Expanding existing privacy protections would help make disability disclosure less precarious and, in so doing, assist efforts to gather more representative datasets. Conclusion Advances in AI have the potential to transform hiring and labor but are ethically fraught and risk exacerbating ableist bias. While hiring practices that discriminate against disabled people long predate AI, myths of algorithmic objectivity, the difficulty of interpreting their inferences, and widening asymmetries in power and information threaten to further entrench those disparities. Indeed, even efforts to promote fair machine learning can end up abstracting away important context, reinforcing the medical model of disability, and imposing artificial fixity on disabled people. By engaging with the ways in which disability has been classified time and time again, we can begin to address the normative issues and assumptions at the heart of AI design. Centering disability can help expand our approach to algorithmic bias beyond narrow technical tools to frameworks that consider the structures, norms, and institutions in which bias emerges. Fair machine learning methods cannot by themselves address these concerns; instead, a genuinely accessible AI ethics must focus on the inherently social and political nature of AI. Such a framework raises issues of justice that cannot be answered through solely technical means. How can algorithms account for difference in ways that do not reinforce rigid systems of classification? How can marginalized populations forge coalitions and contest algorithmic outputs? What ethical, policy, and legal tools can be brought to bear to empower these efforts? Data scientists cannot merely include but must actively center the embodiments, lived experiences, and values of marginalized populations in these discussions. In this paper, I outlined a disability justice approach, which aims to expand the problem space of AI ethics; attend to the ways in which algorithms model disability; bridge divides between stakeholders; and pursue policies promoting transparency and privacy. Disability justice offers a promising framework for analyzing access and (in)capacity in algorithms and merits further consideration in ongoing debates about bias and fairness in AI.