AI data is being leaked and used to discriminate against people with disability
Marks 19 (Mason Marks, Harvard Law School; Yale Law School; University of New Hampshire Franklin Pierce School of Law; Leiden Law School, Center for Law and Digital Technologies Date Written: February 19, 2019 I. Glenn Cohen et al., eds., Disability, Health, Law and Bioethics (Cambridge University Press, 2020)
Many people with disabilities rely on technology to communicate, socialize, and navigate physical and virtual environments. For example, people with vision impairments use assistive devices to identify objects while walking.6 People with mobility impairments use voice-recognition software to access the Internet and draft emails.7 Tom Gruber, a cocreator of Apple’s AI-powered personal assistant Siri, claims his invention helps disabled people engage with society in ways that nondisabled people take for granted.8 These assistive technologies can increase the abilities of people with physical and mental impairments. However, they also serve as continuous sources of EMD, putting people with disabilities at increased risk for discrimination. Imagine an individual named John who is paraplegic and uses a wheelchair. John may never have disclosed his disability publicly. Yet algorithms can infer his disability through his digital traces. After profiling John and sorting him into the category “wheelchair user,” marketing algorithms may serve him targeted ads designed to appeal to people in that category, such as ads for wheelchairs and related accessories. This example seems benign, yet there is a darker side to this type of categorization. On August 13, 2018, the US Department of Housing and Urban Development filed a formal complaint alleging that Facebook engaged in discrimination against people with disabilities, in violation of the FHA, by allowing advertisers to exclude people with disabilities from receiving housing-related advertisements.9 According to the complaint, Facebook enabled advertisers to prevent users from seeing ads if the users had been sorted into categories such as interested in “assistance dog” and interested in “deaf culture.”10 Similarly, John’s categorization as a wheelchair user could be used to exclude him from receiving ads for employment, housing, and other resources. In addition to violating antidiscrimination laws such as the FHA, excluding people with disabilities from receiving these ads strips them of agency and excludes them from participating fully in society. Targeted advertising can also harm people with disabilities more directly. Consider an individual with an eating disorder such as anorexia. Profiling this consumer and serving him customized ads for products such as weight loss pills, laxatives, and stimulants can exacerbate his condition and increase his degree of disability. Targeting ads to consumers based on their unique susceptibilities is called vulnerability-based marketing.11 It exploits people’s physical or mental sensitivities and reinforces patterns of unhealthy behavior. Other examples of vulnerability-based targeted advertising include identifying people with chronic pain or substance use disorders and targeting them with ads for illegal opioids.12 Disability-related EMD is also used in a growing industry called “people analytics” that scores and ranks individuals. Scores are used in a variety of sectors including employment, lending, insurance, education, and public administration. The Chinese government is implementing a social credit system that incorporates data from all spheres of people’s lives. It is intended to reflect people’s trustworthiness, and it governs access to housing, employment, healthcare, transportation, and other resources including potential romantic partners.13 Though the Chinese system may seem shocking to Western societies, elements of the system have already been implemented by American companies.14 Hewlett-Packard pioneered a score called “flight risk” to predict when new or existing employees will quit.15 The score incorporates data on divorce, serious illness, and the frequency and pattern of sick days used. If an employee becomes disabled and uses an increasing number of sick days for medical treatment, the trend will be incorporated into the employee’s flight risk score and could result in early termination. In a second example from human resources, Fortune 500 companies increasingly rely on AI-driven interview tools to rank job applicants. One company called HireVue provides software that analyzes applicants’ facial expressions, tone of voice, and word choice to compare them against the “ideal candidate” in various fields.16 If people with mobility or communication impairments are interviewed by these automated systems, their impairments may negatively impact their rankings compared to other applicants. Disabled job candidates could be screened out of the application process before their materials reach human reviewers.
The above examples may violate the ADA. Title I of the Act prohibits employers from discriminating based on disabilities. However, it puts the burden on employees to identify instances of disability discrimination and report them to the Equal Employment Opportunity Commission (EEOC). An employee cannot file a lawsuit against an employer unless the EEOC has had a chance to investigate the claim. Yet instances of algorithmic disability discrimination may be difficult for employees and job applicants to detect. They are often unaware that their performance on the job or in an interview is being analyzed by algorithms. Moreover, the algorithms used to mine EMD and rank employees and job applicants are often maintained as trade secrets, the confidential intellectual property of the companies that develop and use them. Even if the code of these algorithms was released to the public, their logic is often opaque and may be inscrutable even to the engineers who design them. Because automated employment and hiring-related decisions are hidden from employees and job applicants, yet Title I requires them to identify instances of discrimination, Title I is inadequate to prevent algorithmic disability discrimination. Title III of the ADA prohibits discrimination against people with disabilities in places of public accommodation, which include private businesses that are generally open to the public. Title III lists twelve categories of businesses that provide public accommodations under the ADA such as hotels, restaurants, and movie theaters. Title III could be used to prevent algorithmic disability discrimination against people using social media platforms and other websites. However, it is unclear whether these companies qualify as places of public accommodation because the categories listed in Title III do not include online companies, and historically, places of public accommodation were brick-and-mortar business. Many online businesses do not fit that mold, and courts have been divided on whether they qualify.17 In National Federation of the Blind v. Scribd Inc., 18 an internet-based document repository was deemed a place of public accommodation. The US District Court for the District of Vermont reasoned it would not make sense for people to be treated differently by the law if they bought an insurance plan from a brick-and-mortar store or bought the same plan from an online vendor.19 However, in Cullen v. Netflix,20 the US District Court for the Northern District of California disagreed, finding that because Netflix’s services are not connected to any “actual physical place,” Netflix is not subject to the ADA.21 In National Association of the Deaf v. Netflix, 22 the US District Court of the District of Massachusetts reached the opposite conclusion.23 The court said excluding companies that deliver services online from the definition of places of public accommodation would frustrate Congress’s intent that people with disabilities “fully enjoy the goods, services, privileges and advantages, available indiscriminately to other members of the general public.”24 Until the debate over whether internet businesses provide public accommodations is resolved, companies that operate within this legal grey area may lack sufficient incentives to abstain from algorithmic disability discrimination. Like the ADA, HIPAA has limitations when it comes to preventing algorithmic discrimination. It applies only to disability-related information held by traditional healthcare entities and their business associates.25 It does not apply to the growing variety of smartphone apps and websites that collect our digital traces or the algorithms that transform them into disability-related information.26 The legislative history of HIPAA suggests its primary purpose is to protect people’s rights by controlling the inappropriate use of their health information.27 It includes quotes by jurists, writers, and philosophers on the importance of maintaining privacy. According to Janna Malamud Smith, “If we continually, gratuitously, reveal other people’s privacies, we harm them and ourselves, we undermine the richness of the personal life, and we fuel a social atmosphere of mutual exploitation.”28 The ability to mine EMD undermines assumptions on which HIPAA was based and expands the types of entities that have access to medical information. Before making recommendations to regulate algorithmic disability discrimination, I will explain why it is important to protect the privacy and autonomy of people with disabilities by restricting the flow of their personal data.
II. WHY WORRY ABOUT THE DISRUPTED FLOW OF DISABILITY-RELATED DATA? In the twentieth century, great strides were made toward securing rights for people with disabilities, and the passage of the ADA is a landmark in this process.29 However, we are far from achieving full equality.30 To ensure that progress continues to be made, we must observe how emerging technologies affect people with disabilities. There is a growing trend to segment, score, and rank the population based on personal characteristics.31 EMD collected from disabled people can serve as a means of sorting them into categories that are assigned positive or negative weights before being used in automated decision-making. Negatively weighted categories can stigmatize disabled people and contribute to the narrative that disabilities are bad. There are ongoing debates about whether the differences between disabled and nondisabled people should be described as “mere differences” or “bad differences.”32 As we progress further into the Information Age, human decisionmakers are being replaced with software that will increasingly weigh in on this debate. By negatively weighting categories into which disabled people are sorted, algorithms may stigmatize disabled people and screen them out of life opportunities without considering their desires or qualifications. Thus, what may have been deemed mere differences by human decision-makers could be flagged as bad differences by algorithms. The disability literature describes two models that attempt to explain the origins of disabilities. The medical model frames mental and physical impairments as health problems requiring treatment. According to the model, disabilities result from the states of people’s bodies and minds, and the primary way to reduce their degree of disability is to correct physical and mental impairments. In contrast, according to the social model of disability, there is nothing inherently negative or disabling about having physical or mental impairments.33 It is within the context of society, where the physical surroundings and social environment shape the experiences of people with impairments, that people’s impairments become disabling and affect their participation in society.34 The social model has changed the way people with mental and physical impairments view themselves. Because it places the blame for disabilities on society, rather than on people with impairments, it serves as a source of self-esteem for many people with disabilities.35 AI increasingly shapes our physical and virtual environments and influences our culture. Under the social model, algorithms that limit the opportunities of people with disabilities contribute to their degree of disability. People with disabilities may wish to define identities for themselves that are independent of their impairments.36 Some people take pride in their disabilities and choose to display them proudly as symbols of their individuality.37 However, others may wish to keep their impairments private and define themselves by other qualities. EMD-based profiling strips them of this option by sorting them into disability categories without their knowledge or consent. Similarly, people with disabilities may wish to make life choices uninfluenced by their disabilities. However, in many cases, communities and societies make decisions for them.38 If a wheelchair user cannot enter a concert hall because the venue lacks a ramp, then the decision to attend the concert has been taken out of the person’s hands. Similarly, if algorithms exclude a person with disabilities from receiving ads for important resources or opportunities, the algorithms have made life decisions for that person. As many industries increasingly shift decision-making responsibilities from humans to AI-based systems, algorithms will increasingly make decisions for disabled people and deprive them of some degree of control over their lives.39 To prevent this from happening, new laws are required to restrict the collection and use of their data.