Free beta access. 5 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

Gender-biased AI has a detrimental effect on the health and safety of minority groups

Smith and Rustagi 21 Genevieve Smith & Ishita Rustagi, 3-31-2021, "When Good Algorithms Go Sexist: Why and How to Advance AI Gender Equity (SSIR)," No Publication, https://ssir.org/articles/entry/when_good_algorithms_go_sexist_why_and_how_to_advance_ai_gender_equity [AJL]

In 2019, Genevieve (co-author of this article) and her husband applied for the same credit card. Despite having a slightly better credit score and the same income, expenses, and debt as her husband, the credit card company set her credit limit at almost half the amount. This experience echoes one that made headlines later that year: A husband and wife compared their Apple Card spending limits and found that the husband’s credit line was 20 times greater. Customer service employees were unable to explain why the algorithm deemed the wife significantly less creditworthy. Many institutions make decisions based on artificial intelligence (AI) systems using machine learning (ML), whereby a series of algorithms takes and learns from massive amounts of data to find patterns and make predictions. These systems inform how much credit financial institutions offer different customers, who the health care system prioritizes for COVID-19 vaccines, and which candidates companies call in for job interviews. Yet gender bias in these systems is pervasive and has profound impacts on women’s short- and long-term psychological, economic, and health security. It can also reinforce and amplify existing harmful gender stereotypes and prejudices. As we conclude Women's History Month, social change leaders—including researchers and professionals with gender expertise—and ML systems developers alike need to ask: How can we build gender-smart AI to advance gender equity, rather than embed and scale gender bias? Where AI Gender Bias Comes From AI systems are biased because they are human creations. Who makes decisions informing AI systems and who is on the team developing AI systems shapes their development. And unsurprisingly, there is a huge gender gap: Only 22 percent of professionals in AI and data science fields are women—and they are more likely to occupy jobs associated with less status. At a more granular level, humans generate, collect, and label the data that goes into datasets. Humans determine what datasets, variables, and rules the algorithms learn from to make predictions. Both of these stages can introduce biases that become embedded in AI systems. In terms of gender bias from data, data points are snapshots of the world we live in, and the large gender data gaps we see are partly due to the gender digital divide. For example, some 300 million fewer women than men access the Internet on a mobile phone, and women in low- and middle-income countries are 20 percent less likely than men to own a smartphone. These technologies generate data about their users, so the fact that women have less access to them inherently skews datasets. Even when data is generated, humans collecting data decide what to collect and how. No industry better illustrates this than health care (another industry with gender imbalance among leadership): Men and male bodies have long been the standard for medical testing. Women are missing from medical trials, with female bodies deemed too complex and variable. Females aren’t even included in animal studies on female-prevalent diseases. This gap is reflected in medical data. Data that isn’t disaggregated by sex and gender (as well as other identities) presents another problem. It paints an inaccurate picture, concealing important differences between people of different gender identities, and hides potential overrepresentation or underrepresentation. For example, few urban datasets track and analyze data on gender, so infrastructure programs don’t often factor in women’s needs. Even when representative data points do exist, they may have prejudice built-in and reflect inequities in society. Returning to the consumer credit industry, early processes used marital status and gender to determine creditworthiness. Eventually, these discriminatory practices were replaced by ones considered more neutral. But by then, women had less formal financial history and suffered from discrimination, impacting their ability to get credit. Data points tracking individuals’ credit limits capture these discriminatory trends. Labeling of data can be subjective and embed harmful biases and perspectives too. For instance, most demographic data end up labeled on the basis of simplistic, binary female-male categories. When gender classification collapses gender in this way, it reduces the potential for AI to reflect gender fluidity and self-held gender identity. In terms of gender bias from algorithms, one of the first steps in developing an algorithm is the selection of training dataset(s). Again, back to the consumer credit industry, when AI systems that determine creditworthiness learn from historical data, they pick up on the patterns of women receiving lower credit limits than men. They reproduce the same inequitable access to credit along gender (and race) lines, as seen in Genevieve’s case and the Apple Card story. Relatedly, the Gender Shades research project found that commercial facial-recognition systems used image data sets that lack diverse and representative samples. These systems misclassified women far more often than men. In particular, darker-skinned women were misclassified at an error rate of 35 percent, compared to an error rate of .8 percent for lighter-skinned men. Developers tell algorithms what variables to consider when making decisions, but those variables and proxies may penalize certain identities or communities. For example, an online tech hiring platform, Gild (since acquired by Citadel), developed an AI system to help employers rank candidates for programming jobs. Gild not only screened information gleaned from traditional sources such as resumes, but also used a proxy called “social data” (data generated by actions in the digital realm) to measure how integral the candidate was to the digital community. In this case, social data was drawn from time spent sharing and developing code on platforms like GitHub. But factors such as the societal expectations around unpaid care, which women tend to bear, translate to women having less time to chat online. Women therefore produce less of this social data. In addition, women may assume male identities on platforms like GitHub to circumvent sexist, gender-specific safety concerns (such as targeted harassment and trolling), and other forms of bias. Instead of removing human biases, Gild created an algorithm predisposed to penalizing women and systematically ranking female candidates lower than male counterparts. Impacts of Gender-Biased AI Gender-biased AI not only has immense impacts on individuals but also can contribute to setbacks in gender equality and women’s empowerment. As part of our work at the Berkeley Haas Center for Equity, Gender and Leadership on mitigating bias in artificial intelligence, we track publicly available instances of bias in AI systems using ML. In our analysis of around 133 biased systems across industries from 1988 to present day, we found that 44.2 percent (59 systems) demonstrate gender bias, with 25.7 percent (34 systems) exhibiting both gender and racial bias. Gender-biased AI systems have six primary impacts: Of the 59 systems exhibiting gender bias, 70 percent resulted in lower quality of service for women and non-binary individuals. Voice-recognition systems, increasingly used in the automotive and health care industries, for example, often perform worse for women. Second, unfair allocation of resources, information, and opportunities for women manifested in 61.5 percent of the systems we identified as gender-biased, including hiring software and ad systems that deprioritized women’s applications. Reinforcement of existing, harmful stereotypes and prejudices (in 28.2 percent of gender-biased systems) is exacerbated by feedback loops between data inputs and outputs. For instance, translation software, which learns from vast amounts of online text, has historically taken gender-neutral terms (such as “the doctor” or “the nurse” in English) and returned gendered translations (such as “el doctor” and “la enfermera,” respectively, in Spanish), reinforcing stereotypes of male doctors and female nurses. Relatedly, we find that AI systems—most commonly in Internet-related services—result in derogatory and offensive treatment or erasure of already marginalized gender identities (6.84 percent). For example, using the gender binary in gender classification builds in an inaccurate, simplistic view of gender in tools such as facial analysis systems. In addition, certain systems affect the physical and mental well-being of women and non-binary individuals. Gender-biased systems used in health care, welfare, and the automotive industry, in particular, pose detriments to physical safety (18.8 percent of gender-biased systems) and health hazards (3.42 percent). AI systems supporting skin cancer detection, for example, struggle to detect melanoma for Black people, putting Black women who are already underserved by the health care industry at risk.

Sign in to recut, update, or save this card.