Free beta access. 5 free AI tokens each week. Optional top-ups available. See what is included
Open evidence library

Search the archive.

Current and older evidence is free during beta. Sign in for unlimited card access and full source downloads.

Matching cards
2022 CNDI Kritik
NEG › Links › AWS

Advancement of AI lowers resistance to killing- distances people from reality

Galliott 21 Galliott, Jai. "Humans, autonomous systems, and killing in war." Research Anthology on Military and Defense Applications, Utilization, Education, and Ethics. IGI Global, 2021. 240-257. [AJL]

Most human beings are born with what can only be described as a primitive survival instinct that, without unchecked counter force, would lead to a degree of violence and savagery. But in most societies, people are raised and socialised in such a way that typically leads them to hold an aversion to harming other human beings, which might be why some choose to participate in warfare via wielding ones and zeroes rather than guns and bullets. In a military context and as applied to lower-end autonomous systems, this socialised reluctance to kill is evidenced by recounts and statistics from earlier wars. David Grossman (1995), a self-proclaimed ‘killogist’ or military psychologist, writes of two World War veterans. The first confirms that many WWI infantrymen never fired their weapons and relied instead on artillery, while the second says that platoon sergeants in WWII had to move up and down the firing line kicking men to get them to fire and that they felt they were doing good if they could ‘get two or three men out of a squad to fire’ (Grossman 1995, p. xiv). While some have criticised his methodology, S. L. A. Marshall gave further supporting evidence in arguing from personal experience and studies conducted on firing ratios, which revealed that ‘on average not more than 15 per cent of the men had actually fired at the enemy’ (Marshall 2000, p. 54). He attributed this startling inhibition to kill to an ‘ingrained fear of aggression’ that was based on society’s teaching that killing is fundamentally wrong (Marshall 2000, p. 71). For Marshall, success in combat and the welfare of the state and its people demanded that action be taken to correct or overcome this problem. For those with an aversion to highly autonomous weapon systems, the problem is reversed, as will be later discussed. In the years following publication of the first edition of Marshall’s book – that is, in those following WWII – there is evidence that Marshall’s calls for corrective action were answered. The claims of very low firing rates had been replaced by very high and morally concerning firing rates. By the time of the Korean War, the American firing rate was said to be up to fifty-five percent and, in Vietnam, it was reported to be up to ninety or ninety-five percent (Meagher 2006). Some expressed doubts about these firing rates too, with some finding troops with unspent ammunition in the rear of troop formations, but they were generally satisfied that among those who actually sighted the enemy, there appeared to have been extraordinarily high and consistent firing rates (Grossman 1995). From a strictly military or operational perspective, this is a remarkable success story. In order to overcome the hesitancy to fire and kill that most people develop over time, Russel Glenn says that staff sergeants and platoon commanders watched their troops to ensure that they were actually engaging with the adversary and that in Vietnam, they listened for the steady roar of machine gun fire which indicated to them that their soldiers were unhesitatingly firing their weapons (Marshall 2000). However, this corrective action seems unlikely to account for such a radical shift in the firing ratios. The real cause for the difference in the firing rates, it could be argued, has much more to do with technology employed in later conflicts and changes in military training which, together, allowed and continues to allow, individuals to achieve a physical, emotional and/ or moral distance from their enemies, thus enabling them to kill somewhat easier. It is these distances that need to be explored in more detail, as even the most advanced autonomous unmanned systems that can be conceived of today or the near future will only further these distances and the disengagement and the accompanying desensitization. They may further them to a point that gives rise to unique problems affecting their operators’ ability to wage discriminate and proportional warfare, but the problems, even in the face of artificial intelligence, are uniquely human problems and well within the domain of just war theory and theories similarly seeking to govern human behaviour. +XPDQV$XWRQRPRXV6\VWHPVDQG.LOOLQJLQ:DU The link between physical and emotional distance, ease of aggression and waging warfare is in no way a new discovery and thus it is puzzling why critics of autonomous weapon platforms ignore the relationship that this link has with the modern programmer come warfighter. As Grossman (1995) writes, it has long been understood that there is a positive relationship between the empathetic and the spatial proximity of the victim and the resultant difficulty and personal trauma caused by the kill, or the morally problematic ease of killing, more generally. This relationship has been a cause for concern among anthropologists, philosophers, psychologists, theologians and, of course, soldiers themselves, who often struggle to understand their own actions. Jesse Glenn Gray (1959), an American philosophy professor whose career was interrupted by a period of service as a WWII counter-intelligence officer, wrote that unless one is caught in some sort of overwhelming murderous ecstasy where rage takes over, killing and destroying is much easier when done at a little remove and that with every foot of distance, there is a corresponding decrease in the accurate portrayal of reality. He argues that there is a point at which one’s representation of the world begins to flag and another at which it fails altogether (Gray 1959). Glenn put forward this argument over fifty years ago and the valid concern is that autonomous weapons systems seem to increase the relevant distances to the flagging point that he referenced, where the moral inhibitions of those operating, overseeing or designing/engineering autonomous systems are almost totally overcome but, the corollary of this is that, the advancement of digital warfare with high levels of artificially intelligent autonomous systems cannot surpass point that has already been reached. That is, the worry is not that we have reached another morally significant point in the history of weapons development, but that we are looking at a new enactment of an old trend. This is to say that to truly understand the relationship between soldiers, technical actors, autonomy and their ethicality for today’s purposes, we have to think about autonomous systems in the wider context of weapons that increase physical distance to the target and lower resistance to killing, as well how this maps onto increasing levels of artificial intelligence.

2022 CNDI Kritik
NEG › Links › AWS

AI util cost benefit analysis has significant risks, threatens lives - extinction

Brian Green, Dec 16 2016, assistant director of Campus Ethics at the Center, teaches engineering ethics at Santa Clara University, Social Robots, AI, and Ethics, Markkula Center for Applied Ethics at Santa Clara University, - Maren Lien

Utilitarianism is a form of moral reasoning which emphasizes the consequences of actions. Typically it tries to maximize happiness and minimize suffering, though there are other ways to use utilitarian evaluation such as cost-benefit analysis. On the benefit side of the equation, because robots and AI are great for doing work that is boring, dirty, or dangerous, when employed they often improve the cost-benefit analysis. For example, many assembly-line jobs around the world have replaced human workers with robots, which often enhances worker safety and helps avoid repetitive motion injuries. Over time these robot replacements may also save factories money, raising corporate profits or lowering the prices of goods produced. Other places where robots and AI might improve the cost-benefit analysis may include medical diagnostics, “big data” analytics, robots to help care for the elderly, and lethal autonomous weapons systems in war. On the cost side of the equation, for each of the above examples robots and AI have a downside. They threaten to take away jobs, separate us from meaningful work, separate us from being able to understand the data we analyze, leave the elderly isolated from human contact, and, ultimately even threaten our lives, perhaps even driving us extinct. These downside risks are significant and worthy of serious consideration starting before these technologies are implemented. It is overall beneficial to use robots and AI in these ways? Or do the costs outweigh the benefits? Can we choose to promote some uses of robotics and AI technology while limiting others?

2022 CNDI Kritik
NEG › Links › AWS

LAWS produce ethical and logistic quandaries – weaponizing AI accelerates causalities and human rights abuses

Amnesty International 15 (Amnesty International is a global movement of more than 3 million supporters, members and activists in more than 150 countries and territories who campaign to end grave abuses of human rights. Our vision is for every person to enjoy all the rights enshrined in the Universal Declaration of Human Rights and other international human rights standards. We are independent of any government, political ideology, economic interest or religion and are funded mainly by our membership and public donations.) “AUTONOMOUS WEAPONS SYSTEMS: FIVE KEY HUMAN RIGHTS ISSUES FOR CONSIDERATION” April 10 2015 // ZX

Over the past decade, there have been extensive advances in artificial intelligence and other technologies. These will make possible the development and deployment of fully autonomous weapons systems which, once activated, can select, attack, kill and wound human targets, and will be able to operate without effective human control. These weapons systems are often referred to as Lethal Autonomous Robotics (LARs), Lethal Autonomous Weapons Systems (LAWS) and, more comprehensively, Autonomous Weapons Systems (AWS). The rapid development of these weapons systems could not only change the entire nature of warfare, it could also dramatically alter the conduct of law enforcement operations and raises extremely serious human rights concerns, undermining the right to life, the prohibition of torture and other ill-treatment, and the right to security of person, and other human rights. Amnesty International has taken the view that AWS is a useful term for these weapons systems, since these systems can (i) be designed to have lethal or less lethal effects and (ii) be used in armed conflict and/or law enforcement situations. With proliferation they are likely to come to be used by non-state armed groups, criminal gangs and private companies and individuals. Amnesty International takes the term ‘autonomous’ to mean weapons capable of selecting targets and triggering an attack without effective or meaningful human control1 that can ensure the lawful use of force. Such systems would use violence (including less-lethal force) against individuals, and could have adverse consequences for a person’s human rights. While the development of AWS clearly raises serious and legitimate ethical and societal concerns, this briefing paper will examine the implications of AWS in the context of international law, particularly international human rights law and standards. The important concerns around their use in situations of armed conflict, and thus their ability to comply fully with international humanitarian law (IHL), has been the focus of previous work on AWS, including by Human Rights Watch, other members of the Campaign to Stop Killer Robots and the International Committee of the Red Cross (ICRC). This briefing paper, however, will address some of the implications for human rights related to AWS, particularly those rights and standards that govern the conduct of law enforcement operations. Amnesty International believes that the questions surrounding the development and potential use of AWS outside armed conflict (and the ability of such systems to comply with human rights law) are at least as daunting as those related to their use on the battlefield and urgently require attention and consideration2 , ultimately leading to concrete steps that will address this important area of international law. Amnesty International has identified five key human rights issues for consideration in the current debate on AWS: 1) The scope of the Convention on Certain Conventional Weapons (CCW) does not cover non-conflict situations; 2) AWS will not be able to comply with relevant international human rights law (IHRL) and policing standards; 3) Developments in existing semi-autonomous weapons technology pose fundamental challenges for the IHRL framework; 4) In the absence of a prohibition, AWS must be subject to independent weapons reviews; and 5) AWS will erode accountability mechanisms. The issues identified are by no means exhaustive, but rather seek to elucidate the principal concerns around the potential use of AWS in law enforcement operations. This briefing argues that the use of AWS, including less-lethal robotic weapons, in law enforcement operations would be fundamentally incompatible with international human rights law, and would lead to unlawful killings, injuries and other violations of human rights. Furthermore, the use of AWS would pose serious challenges in holding accountable those responsible for serious violations and could entrench impunity for crimes under international law. Consequently, Amnesty International supports the call for a pre-emptive ban on the development, transfer, deployment and use of AWS, including fully autonomous systems that deploy less-lethal weapons and can result in death or serious injury. In the absence of a prohibition, Amnesty International supports the call of UN Special Rapporteur on extrajudicial, summary or arbitrary executions, Christof Heyns, to impose a moratorium on the development, transfer, deployment and use of AWS and ensure that moratorium covers both lethal and less-lethal weapons. This principle deals with two different thresholds: a) when it is appropriate to use firearms (potentially lethal force) and b) the even higher threshold of when the intentional lethal use of firearms is permissible. Each of these situations involves a complex assessment of potential or imminent threats to life or serious injury and how to respond to them appropriately, and it involves deciding how best to protect the right to life, which is an absolutely fundamental duty of the state under human rights law. Such life and death decisions must never be delegated to AWS. In order to be able to carry out policing and law enforcement operations in a lawful manner, AWS would need to be able to effectively assess the degree to which there was an imminent threat of death or serious injury, identify correctly who is posing the threat, consider whether force is necessary to neutralize the threat, be able to identify and use means other than force, have the capacity to deploy different modes of communication and policing weapons and equipment to allow for a graduated response, and have available back up means and resources. To add to this complexity, each situation would require a different and unique response, which would be extremely challenging to reduce to a series of complex algorithms. It is not possible that AWS, without meaningful and effective human control and judgement, would be able to comply with these provisions, especially in unpredictable and ever-evolving environments. In an open letter in October 2013, computer scientists, engineers, artificial intelligence experts, roboticists and professionals from related disciplines from 37 countries asserted that “in the absence of clear scientific evidence that robot weapons have, or are likely to have in the foreseeable future, the functionality required for accurate target identification, situational awareness or decisions regarding the proportional use of force, we question whether they could meet the strict legal requirements for the use of force” and that “[G]iven the limitations and unknown future risks of autonomous robot weapons technology…,[D]ecisions about the application of violent force must not be delegated to machines.”15 The UNBPUFF places a due diligence requirement upon states to review weapons used in law enforcement. As Principle 3 of the UNBPUFF states, “the development and deployment of non-lethal incapacitating weapons should be carefully evaluated in order to minimize the risk of endangering uninvolved persons”. This review is limited to less-lethal weapons but is still important to ensure that those weapons will comply with relevant international standards and national laws and, moreover, given that evidence shows that “non-lethal” weapons can often have lethal effects which is why the term “less-lethal” is more appropriate. The requirement of a review of weapons used for law enforcement is even more important given the increasing ‘militarization’ of law enforcement operations, whereby military personnel assume roles often held by law enforcement agencies, such as policing of public assemblies. In the absence of a prohibition on AWS, states intending to develop, acquire, or use AWS must therefore be required to thoroughly review whether they can be used in a manner that fully respects relevant law and standards be it for law enforcement or military operations. This testing should be carried out by an independent body. The rapid technological advances that are moving towards full autonomy in weapons systems present serious concerns. The technology to allow fully autonomous operations may be reached soon; but it is extremely unlikely that programming that could ensure AWS perform law enforcement functions lawfully would be developed in the foreseeable future. Any new law enforcement equipment should be introduced based on clearly defined operational needs and technical requirements with a view to reduce the amount of force used and the risk and level of harm and injury caused. They must be subject to rigorous testing, by an independent expert body, and the testing, review and selection process should be legally constituted. In addition to assessing compliance with the UNBPUFF themselves, the process must test AWS compatibility with other key human rights treaties and standards, including ICCPR, International Covenant on Economic, Social and Cultural Rights (CESCR), the Convention Against Torture, the SMRTP and the UNCCLEO.

2022 CNDI Kritik
NEG › Links › Bias (general)

AI unethical - exacerbates racism, transphobia, alternate realities, and privacy violations

Desmond 21 Allison Proffitt, 6-21-2022, "Unethical Use of AI Being Mainstreamed by Some Business Execs, Survey Finds ," AI Trends, [AJL]

In a recent survey, senior business executives admitted to their sometimes unethical use of AI. The admission of being openly unethical came from respondents to a recent survey conducted by KPMG of 250 director-level or higher executives at companies with more than 1,000 employees about data privacy. Some 29% of the respondents admitted that their own companies collect personal information that is “sometimes unethical” and 33% said consumers should be concerned about how their company uses personal data, according to a recent report in The New Yorker. Orson Lucas, principal, US privacy services team, KPMG The result surprised the survey-taker. “For some companies, there may be a misalignment between what they say they are doing on data privacy and what they are actually doing,” stated Orson Lucas, the principal in KPMG’s US privacy services team. One growing practice is a move to “collect everything” about a person, then figure out later how to use it. This approach is seen as an opportunity to better understand what customers want to get out of the business that can later result in a transparent negotiation about what information customers are willing to provide and for how long. Most of these companies have not yet reached the transparent negotiation stage. Some 70% of the executives interviewed said their companies had increased the amount of personal information they collected in the past year. And 62% said their company should be doing more to strengthen data protection measures. KPMG also surveyed 2,000 adults in the general population on data privacy, finding that 40% did not trust companies to behave ethically with their personal information. In Lucas’ view, consumers will want to punish a business that demonstrates unfair practices around the use of personal data. AI Conferences Considering Wider Ethical Reviews of Submitted Papers Meanwhile, at AI conferences, sometimes AI technology is on display with little sensitivity to its potentially unethical use, and at times, this AI tech finds its way into commercial products. The IEEE Conference on Computer Vision and Pattern Recognition in 2019, for example, accepted a paper from researchers with MIT’s Computer Science and AI Laboratory on learning a person’s face from audio recordings of that person speaking. The goal of the project, called Speech2Face, was to research how much information about a person’s looks could be inferred from the way they speak. The researchers proposed a neural network architecture designed specifically to perform the task of facial reconstruction from audio. Stuff hit the fan around it, Alex Hanna, a trans woman and sociologist at Google who studies AI ethics, asked via tweet for the research to stop, calling it “transphobic.” Hanna objected to the way the research sought to tie identity to biology. Debate ensued. Some questioned whether papers submitted to academic-oriented conferences need further ethical review. Michael Kearns, a computer scientist at the University of Pennsylvania and a coauthor of the book, “The Ethical Algorithm,” stated to The New Yorker that we are in “a little bit of a Manhattan Project moment” for AI and machine learning. “The academic research in the field has been deployed at a massive scale on society,” he stated. “With that comes this higher responsibility.” Katherine Heller, computer scientist, Duke University A paper on Speech2Face was accepted in the 2019 Neural Information Processing Systems (Neurips) Conference held in Vancouver, Canada. Katherine Heller, a computer scientist at Duke University and a Neurips co-chair for diversity and inclusion, told The New Yorker that the conference had accepted some 1,400 papers that year, and she could not recall facing comparable pushback on the subject of ethics. “It’s new territory,” she stated. For Neurips 2020, held remotely in December 2020, papers faced rejection if the research was found to pose a threat to society. Iason Gabriel, a research scientist at Google DeepMind in London, who is among the leadership of the conference’s ethics review process, said the change was needed to help AI “make progress as a field.” Ethics is somewhat new territory for computer science. Whereas biologists, psychologists, and anthropologists are used to reviews that query the ethics of their research, computer scientists have not been raised that way. The focus is more around methods, such as plagiarism and conflicts of interest. That said, a number of groups interested in the ethical use of AI have come about in the last several years. The Association for Computing Machinery’s Special Interest Group on Computer-Human Interaction, for example, launched a working group in 2016 that is now an ethics research committee that offers to review papers at the request of conference program chairs. In 2019, the group received 10 inquiries, primarily around research methods. “Increasingly, we do see, especially in the AI space, more and more questions of, Should this kind of research even be a thing?” stated Katie Shilton, an information scientist at the University of Maryland and the chair of the committee, to The New Yorker. Shilton identified four categories of potentially unethical impact. First, AI that can be “weaponized” against populations, such as facial recognition, location tracking, and surveillance. Second, technologies such as Speech2Face that may “harden people into categories that don’t fit well,” such as gender or sexual orientation. Third, automated weapons research. Fourth, tools used to create alternate sets of reality, such as fake news, voices or images. This green field territory is a venture into the unknown. Computer scientists usually have good technical knowledge, “But lots and lots of folks in computer science have not been trained in research ethics,” Shilton stated, noting that it is not easy to say that a line of research should not exist. Location Data Weaponized for Catholic Priest The weaponization of location-tracking technology was amply demonstrated in the recent experience of the Catholic priest who was outed as a Grindr dating app user, and who subsequently resigned. Catholic priests take a vow of celibacy, which would be in conflict with being in a dating app community of any kind. The incident raised a panoply of ethical issues. The story was broken by a Catholic news outlet called the Pillar, which had somehow obtained “app data signals from the location-based hookup app Grindr,” stated an account in recode from Vox. It was not clear how the publication obtained the location data other than to say it was from a “data vendor.” “The harms caused by location tracking are real and can have a lasting impact far into the future,” stated Sean O’Brien, principal researcher at ExpressVPN’s Digital Security Lab, to recode. “There is no meaningful oversight of smartphone surveillance, and the privacy abuse we saw in this case is enabled by a profitable and booming industry.” One data vendor in this business is X-Mode, which collects data from millions of users across hundreds of apps. The company was kicked off the Apple and Google platforms last year over its national security work with the US government, according to an account in The Wall Street Journal. However, the company is being acquired by Digital Envoy, Inc. of Atlanta, and will be rebranded as Outlogic. It’s chief executive, Joshua Anton, will join Digital Envoy as chief strategy officer. The purchase price was not disclosed. Acquiring X-Mode “allows us to further enhance our offering related to cybersecurity, AI, fraud and rights management,” stated Digital Envoy CEO Jerrod Stoller. “It allows us to innovate in the space by looking at new solutions leveraging both data sets. And it also brings new clients and new markets.” Digital Envoy specializes in collecting and providing to its customers data on internet users based on the IP address assigned to them by their ISP or cell phone carrier. The data can include approximate geolocation and is said to be useful in commercial applications, including advertising. X-Mode recently retired a visualization app, called XDK, and has changed practices by adding new guidance on where data is sourced from, according to an account in Technically. This is the second time the company has rebranded since it was founded in 2013, when it started off as Drunk Mode. Following the acquisition, Digital Envoy said in a statement that it added a new code of ethics, a data ethics review panel, a sensitive app policy and will be hiring a chief privacy officer.

2022 CNDI Kritik
NEG › Links › Bias (general)

AI reflects harmful systematic biases

Boutin 3/16/22 (NIST, a physical sciences laboratory and non-regulatory agency of the United States Department of Commerce, “There’s More to AI Bias Than Biased Data, NIST Report Highlights,” NIST, June 21, 2022, ) – Joyous Joelle

The recommendation is a core message of a revised NIST publication, , which reflects public comments the agency received on its released last summer. As part of a to support the development of trustworthy and responsible AI, the document offers guidance connected to the that NIST is developing. According to NIST’s Reva Schwartz, the main distinction between the draft and final versions of the publication is the new emphasis on how bias manifests itself not only in AI algorithms and the data used to train them, but also in the societal context in which AI systems are used. “Context is everything,” said Schwartz, principal investigator for AI bias and one of the report’s authors. “AI systems do not operate in isolation. They help people make decisions that directly affect other people’s lives. If we are to develop trustworthy AI systems, we need to consider all the factors that can chip away at the public’s trust in AI. Many of these factors go beyond the technology itself to the impacts of the technology, and the comments we received from a wide range of people and organizations emphasized this point.” Bias in AI can harm humans. AI can make decisions that affect whether a person is admitted into a school, authorized for a bank loan or accepted as a rental applicant. It is relatively common knowledge that AI systems can exhibit biases that stem from their programming and data sources; for example, machine learning software could be trained on a dataset that underrepresents a particular gender or ethnic group. The revised NIST publication acknowledges that while these computational and statistical sources of bias remain highly important, they do not represent the full picture. A more complete understanding of bias must take into account human and systemic biases, which figure significantly in the new version. Systemic biases result from institutions operating in ways that disadvantage certain social groups, such as discriminating against individuals based on their race. Human biases can relate to how people use data to fill in missing information, such as a person’s neighborhood of residence influencing how likely authorities would consider the person to be a crime suspect. When human, systemic and computational biases combine, they can form a pernicious mixture — especially when explicit guidance is lacking for addressing the risks associated with using AI systems.

2022 CNDI Kritik
NEG › Links › Bias (general)

AI biased and comes with a laundry list of negative consequences

Bossmann 16 Julia Bossmann, 10-21-2016, "Top 9 ethical issues in artificial intelligence," World Economic Forum, [AJL]

5. Racist robots. How do we eliminate AI bias? Though artificial intelligence is capable of a speed and capacity of processing that’s far beyond that of humans, it cannot always be trusted to be fair and neutral. Google and its parent company Alphabet are one of the leaders when it comes to artificial intelligence, as seen in Google’s Photos service, where AI is used to identify people, objects and scenes. But it can go wrong, such as when a camera missed the mark on racial sensitivity, or when a software used to predict future criminals showed bias against black people. We shouldn’t forget that AI systems are created by humans, who can be biased and judgemental. Once again, if used right, or if used by those who strive for social progress, artificial intelligence can become a catalyst for positive change. 6. Security. How do we keep AI safe from adversaries? The more powerful a technology becomes, the more can it be used for nefarious reasons as well as good. This applies not only to robots produced to replace human soldiers, or autonomous weapons, but to AI systems that can cause damage if used maliciously. Because these fights won't be fought on the battleground only, cybersecurity will become even more important. After all, we’re dealing with a system that is faster and more capable than us by orders of magnitude. Proliferation of Armed Drones 7. Evil genies. How do we protect against unintended consequences? It’s not just adversaries we have to worry about. What if artificial intelligence itself turned against us? This doesn't mean by turning "evil" in the way a human might, or the way AI disasters are depicted in Hollywood movies. Rather, we can imagine an advanced AI system as a "genie in a bottle" that can fulfill wishes, but with terrible unforeseen consequences. In the case of a machine, there is unlikely to be malice at play, only a lack of understanding of the full context in which the wish was made. Imagine an AI system that is asked to eradicate cancer in the world. After a lot of computing, it spits out a formula that does, in fact, bring about the end of cancer – by killing everyone on the planet. The computer would have achieved its goal of "no more cancer" very efficiently, but not in the way humans intended it. 8. Singularity. How do we stay in control of a complex intelligent system? The reason humans are on top of the food chain is not down to sharp teeth or strong muscles. Human dominance is almost entirely due to our ingenuity and intelligence. We can get the better of bigger, faster, stronger animals because we can create and use tools to control them: both physical tools such as cages and weapons, and cognitive tools like training and conditioning. This poses a serious question about artificial intelligence: will it, one day, have the same advantage over us? We can't rely on just "pulling the plug" either, because a sufficiently advanced machine may anticipate this move and defend itself. This is what some call the “singularity”: the point in time when human beings are no longer the most intelligent beings on earth. 9. Robot rights. How do we define the humane treatment of AI? While neuroscientists are still working on unlocking the secrets of conscious experience, we understand more about the basic mechanisms of reward and aversion. We share these mechanisms with even simple animals. In a way, we are building similar mechanisms of reward and aversion in systems of artificial intelligence. For example, reinforcement learning is similar to training a dog: improved performance is reinforced with a virtual reward. Right now, these systems are fairly superficial, but they are becoming more complex and life-like. Could we consider a system to be suffering when its reward functions give it negative input? What's more, so-called genetic algorithms work by creating many instances of a system at once, of which only the most successful "survive" and combine to form the next generation of instances. This happens over many generations and is a way of improving a system. The unsuccessful instances are deleted. At what point might we consider genetic algorithms a form of mass murder?

2022 CNDI Kritik
NEG › Links › Health

AI is racist – healthcare proves

Mohamed et al. 20 (Shakir Mohamed is Senior Staff Scientist at DeepMind, Marie Therese, William Isaac, 07-12-20, “Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence” pp. 662)-qcl

The limitations of these value principles become clearer as AI and other advanced technologies become enmeshed within high-stakes spheres of our society. Initial attempts to codify ethical guidelines for AI, e.g. the Asilomar principles (Asilomar Meeting 2017), focused on risks related to lethal autonomous weapons systems and AGI Safety. Though both are critical issues, these guidelines did not recognise that risks in peace and security are first felt by conflict zones in developing countries (Garcia 2019), or engage in a disambiguation of social safety and technical safety. Moreover, they did not contend with the intersection of values and power, whose values are being represented, and the structural inequities that result in an unequal spread of benefits and risk within and across societies. An example of this nexus between values, power and AI is a recent study by Obermeyer et al. (2019), which revealed that a widely used prediction algorithm for selecting entry into healthcare programs was exhibiting racial bias against AfricanAmerican patients. The tool was designed to identify patients suitable for enrolment into a “high-risk care management” programme that provides access to enhanced medical resources and support. Unfortunately, large health systems in the USA have emphasised contextual values to “reduce overall costs for the healthcare system Decolonial AI: Decolonial Theory as Sociotechnical 661 while increasing value” (AMA 2018) or “value for money” (UK National Health Service 2019) on “value for money”) when selecting potential vendors for algorithmic screening tools at the expense of other values such as addressing inequities in the health system. As a result, the deployed algorithm relied on the predictive utility of an individual’s health expenses (defined as total healthcare expenditure) indirectly leading to the rejection of African-American patients at a higher rate relative to white patients, denying care to patients in need, and exacerbating structural inequities in the US healthcare system (Nelson 2002). As this example shows, the unique manner in which AI algorithms can quickly ingest, perpetuate and legitimise forms of bias and harm represents a step change from previous technologies, warranting prompt reappraisal of these tools to ensure ethical and socially beneficial use. An additional challenge is that AI can obscure asymmetrical power relations in ways that make it difficult for advocates and concerned developers to meaningfully address during development. As Benjamin (2019) notes, “whereas in a previous era, the intention to deepen racial inequities was more explicit, today coded inequity is perpetuated precisely because those who design and adopt such tools are not thinking carefully about systemic racism”. Some scholars such as Floridi et al. (2018) have highlighted that technologies such as AI require an expansion of ethical frameworks, such as the Belmont Principles, to include explicability (explanation and transparency) or non-malfeasance (do no harm). Whittlestone et al. (2019) conversely argue for a move away from enumerating new value criteria, and instead highlight the need to engage more deeply with the tensions that arise between principles and their implementation in practice. Similarly, we argue that the field of AI would benefit from dynamic and robust foresight tactics and methodologies grounded in the critical sciences to better identify limitations of a given technology and their prospective ethical and social harms.

2022 CNDI Kritik
NEG › Links › Health

AI engineers lack the ability to create ethical AI – leads to waste and inefficiency

Blackman 20 Harvard Business Review, 10-15-2020, "A Practical Guide to Building Ethical AI," [AJL]

Companies are leveraging data and artificial intelligence to create scalable solutions — but they’re also scaling their reputational, regulatory, and legal risks. For instance, Los Angeles is suing IBM for allegedly misappropriating data it collected with its ubiquitous weather app. Optum is being investigated by regulators for creating an algorithm that allegedly recommended that doctors and nurses pay more attention to white patients than to sicker black patients. Goldman Sachs is being investigated by regulators for using an AI algorithm that allegedly discriminated against women by granting larger credit limits to men than women on their Apple cards. Facebook infamously granted Cambridge Analytica, a political firm, access to the personal data of more than 50 million users. Just a few years ago discussions of “data ethics” and “AI ethics” were reserved for nonprofit organizations and academics. Today the biggest tech companies in the world — Microsoft, Facebook, Twitter, Google, and more — are putting together fast-growing teams to tackle the ethical problems that arise from the widespread collection, analysis, and use of massive troves of data, particularly when that data is used to train machine learning models, aka AI. INSIGHT CENTER AI and Equality Designing systems that are fair for all. These companies are investing in answers to once esoteric ethical questions because they’ve realized one simple truth: failing to operationalize data and AI ethics is a threat to the bottom line. Missing the mark can expose companies to reputational, regulatory, and legal risks, but that’s not the half of it. Failing to operationalize data and AI ethics leads to wasted resources, inefficiencies in product development and deployment, and even an inability to use data to train AI models at all. For example, Amazon engineers reportedly spent years working on AI hiring software, but eventually scrapped the program because they couldn’t figure out how to create a model that doesn’t systematically discriminate against women. Sidewalk Labs, a subsidiary of Google, faced massive backlash by citizens and local government officials over their plans to build an IoT-fueled “smart city” within Toronto due to a lack of clear ethical standards for the project’s data handling. The company ultimately scrapped the project at a loss of two years of work and USD $50 million. Despite the costs of getting it wrong, most companies grapple with data and AI ethics through ad-hoc discussions on a per-product basis. With no clear protocol in place on how to identify, evaluate, and mitigate the risks, teams end up either overlooking risks, scrambling to solve issues as they come up, or crossing their fingers in the hope that the problem will resolve itself. When companies have attempted to tackle the issue at scale, they’ve tended to implement strict, imprecise, and overly broad policies that lead to false positives in risk identification and stymied production. These problems grow by orders of magnitude when you introduce third-party vendors, who may or may not be thinking about these questions at all. Companies need a plan for mitigating risk — how to use data and develop AI products without falling into ethical pitfalls along the way. Just like other risk-management strategies, an operationalized approach to data and AI ethics must systematically and exhaustively identify ethical risks throughout the organization, from IT to HR to marketing to product and beyond. What Not to Do Putting the larger tech companies to the side, there are three standard approaches to data and AI ethical risk mitigation, none of which bear fruit. First, there is the academic approach. Academics — and I speak from 15 years of experience as a former professor of philosophy — are fantastic at rigorous and systematic inquiry. Those academics who are ethicists (typically found in philosophy departments) are adept at spotting ethical problems, their sources, and how to think through them. But while academic ethicists might seem like a perfect match, given the need for systematic identification and mitigation of ethical risks, they unfortunately tend to ask different questions than businesses. For the most part, academics ask, “Should we do this? Would it be good for society overall? Does it conduce to human flourishing?” Businesses, on the other hand, tend to ask, “Given that we are going to do this, how can we do it without making ourselves vulnerable to ethical risks?” The result is academic treatments that do not speak to the highly particular, concrete uses of data and AI. This translates to the absence of clear directives to the developers on the ground and the senior leaders who need to identify and choose among a set of risk mitigation strategies. Next, is the “on-the-ground” approach. Within businesses those asking the questions are standardly enthusiastic engineers, data scientists, and product managers. They know to ask the business-relevant risk-related questions precisely because they are the ones making the products to achieve particular business goals. What they lack, however, is the kind of training that academics receive. As a result, they do not have the skill, knowledge, and experience to answer ethical questions systematically, exhaustively, and efficiently. They also lack a critical ingredient: institutional support. Finally, there are companies (not to mention countries) rolling out high-level AI ethics principles. Google and Microsoft, for instance, trumpeted their principles years ago. The difficulty comes in operationalizing those principles. What, exactly, does it mean to be for “fairness?” What are engineers to do when confronted with the dozens of definitions and accompanying metrics for fairness in the computer science literature? Which metric is the right one in any given case, and who makes that judgment? For most companies — including those tech companies who are actively trying to solve the problem — there are no clear answers to these questions. Indeed, seeming coalescence around a shared set of abstract values actually obscures widespread misalignment.

2022 CNDI Kritik
NEG › Links › AI sexist/racist

Sexism, racism, and ethical concerns embedded in AI

Macciola 19 Anthony Macciola, 8-29-2019, "Bad, biased, and unethical uses of AI," No Publication, [AJL]

Adoption of AI technology is accelerating rapidly. Gartner forecasts that by 2020, AI will be a top-five investment priority for more than 30 percent of CIOs. A McKinsey study estimates that tech companies are spending between $20 and $30 billion on AI, mostly in research and development. While the social utility of AI technology is compelling, there are legitimate concerns, as raised by The Guardian’s Inequality Project: “When the data we feed the machines reflects the history of our own unequal society, we are in effect asking the program to learn our own biases.” Unfortunately, examples of bad, biased, or unethical uses of AI are commonplace. Here are just four examples that every CIO should be aware of, along with advice on how enterprises can remain neutral. 1. Mortgage lending The mode of lending discrimination has shifted from human bias to algorithmic bias. A study co-authored by Adair Morse, a finance professor at the Haas School of Business, concluded that “even if the people writing the algorithms intend to create a fair system, their programming is having a disparate impact on minority borrowers — in other words, discriminating under the law.” “When the data we feed the machines reflects the history of our own unequal society, we are in effect asking the program to learn our own biases.” [ Are you asking the right questions when it comes to systemic bias? Read also AI bias: 9 questions leaders should ask. ] You might assume that redlining, the systematic segregation of non-white borrowers into less-favorable neighborhoods by banks and real estate agents, is a thing of the past — but you would be wrong. Surprisingly, the automation of the mortgage industry has only made it easier to hide redlining behind a user interface. In his recent book “Data and Goliath,” computer security expert Bruce Schneier recounts how in 2000, Wells Fargo created a website to promote mortgages using a “community calculator” that helped buyers find the right neighborhood. The calculator collected users’ current ZIP code, assumed their race according to the demographics of their current neighborhood, and recommended only neighborhoods with similar demographics. And earlier this year, HUD brought suit against Facebook for racial biases in housing and mortgage advertisements. 2. Human resources By far the most infamous issue with bias in recruiting and hiring came to public attention when Reuters reported that Amazon.com’s new recruiting engine excluded women. According to Reuters, Amazon assembled a team in 2014 that used more than 500 algorithms to automate the resume-review process for engineers and coders. The team trained the system by using the resumes of members of Amazon’s software teams – which were overwhelmingly male. As a result, the system learned to disqualify anyone who attended a women’s college or who listed women’s organizations on their resume. More and more companies are adopting algorithmic decision-making systems at every level of the HR process. As of 2016, 72 percent of job candidates’ resumes are screened not by people, but entirely by computers. That means job candidates and employees will be dealing with people less often – and stories like Amazon’s could become more common. However, the good news is that some companies are making efforts to eliminate potential bias. ABBYY founder David Yang co-founded Yva.ai, an analytics platform that is specifically designed to avoid algorithmic bias by avoiding the use of any indicator that could lead to bias, such as gender, age, or race, even when such indicators are secondary (such as involvement in women’s activities or sports), secondary (such as names or graduation dates), or even tertiary (such as attendance at elite colleges, which has been increasingly called out as a signifier of bias against minorities). The good news is that some companies are making efforts to eliminate potential bias. In another example, LinkedIn, now owned by Microsoft, has deployed systems not to ignore but instead to collect and utilize gender information in LinkedIn profiles. LinkedIn then uses this information to classify and correct for any potential bias. 3. Search Even basic Internet searches can be tainted with bias. For example, UCLA professor Safiya Umoja Noble was inspired to write her book “Algorithms of Oppression” after googling “black women” in a search for interesting sites to share with her nieces, only to find pages filled with pornography. Meanwhile, searches for “CEO” have historically shown image after image of white men. (Fortunately, our own more recent experiences on Google suggest that the CEO problem is being addressed.) Other features of Google search, such as AdWords, have also been guilty of bias. Researchers from Carnegie Mellon University and the International Computer Science Institute discovered that male job seekers were much more likely to be shown advertisements for high-paying executive positions than were women. Google Translate has also been called out for sexism in translating some languages, assuming, for example, that nurses are women and doctors are men. 4. Education In what may well be the earliest reported instance of a tainted system, a 1979 program created by an admissions dean at St. George’s Hospital Medical School in London ended up accidentally excluding nearly all minority and female applicants. By 1986, staff members at the school became concerned about potential discrimination and eventually discovered that at least 60 minority and female applicants were unfairly excluded each year. You might wonder why it took so long to raise the alarm, considering that according to reports, simply having a non-European name could automatically take 15 points off an applicant’s score. The prestigious British Medical Journal bluntly called this bias “a blot on the profession.” Ultimately, the school was mildly penalized, and it did offer reparations, including admitting some of those applicants who were excluded. The CIO’s role Leading tech companies are making efforts to address the ethical use of data. Microsoft, for example, has developed a set of six ethical principles that span fairness, reliability and safety, privacy and security, inclusiveness, transparency, and accountability. Meanwhile, Facebook recently granted $7.5 million dollars to the Technical University of Munich to establish an Institute for Ethics in AI. Other tech companies have subscribed to the Partnership on AI consortium and its principles for “bringing together diverse global voices to realize the promise of artificial intelligence.” AI is only as good as the data behind it, so this data must be fair and representative of all people and cultures. If AI applications are to be bias-free, companies must support a holistic approach to AI technology. AI is only as good as the data behind it, so this data must be fair and representative of all people and cultures. Furthermore, the technology must be developed in accordance with international laws. This year’s G20 Summit finance ministers agreed, for the first time, on G20’s own principles for responsible AI use. This included a human-centric AI approach, which calls on countries to use AI in a way that respects human rights and shares the benefits it offers. On the most simplistic level, CIOs need to question if the AI applications they are building are moral, safe, and right. Questions may include: MORE ON AI BIAS AI bias: 9 questions leaders should ask 4 ways leaders can combat unconscious bias To reduce biases in machine learning start with openly discussing the problem Is the data behind your AI technology good, or does it have algorithmic bias? Are you vigorously reviewing AI algorithms to ensure they’re properly tuned and trained to produce expected results against pre-defined test sets? Are you adhering to transparency principles (such as GDPR) in how AI technology impacts the organization internally and customers and partner stakeholders externally? Have you set up a dedicated AI governance and advisory committee that includes cross-functional leaders and external advisers that will establish and oversee governance of AI-enabled solutions? Ultimately, the ethical uses of AI should be considered a legal and moral obligation as well as a business imperative. Don't become another example of bad and biased AI. Learn from these unethical use cases to unsure your company's AI efforts remain neutral.

2022 CNDI Kritik
NEG › Links › AI sexist/racist

Robots are inherently racist and sexist

Rosen 6/21/22 John Hopkins University, ScienceDaily, 6-21-2022, "Robots turn racist and sexist with flawed AI, study finds: Neural networks built from biased Internet data teach robots to enact toxic stereotypes," [AJL]

The work, led by Johns Hopkins University, Georgia Institute of Technology, and University of Washington researchers, is believed to be the first to show that robots loaded with an accepted and widely-used model operate with significant gender and racial biases. The work is set to be presented and published this week at the 2022 Conference on Fairness, Accountability, and Transparency (ACM FAccT). "The robot has learned toxic stereotypes through these flawed neural network models," said author Andrew Hundt, a postdoctoral fellow at Georgia Tech who co-conducted the work as a PhD student working in Johns Hopkins' Computational Interaction and Robotics Laboratory. "We're at risk of creating a generation of racist and sexist robots but people and organizations have decided it's OK to create these products without addressing the issues." Those building artificial intelligence models to recognize humans and objects often turn to vast datasets available for free on the Internet. But the Internet is also notoriously filled with inaccurate and overtly biased content, meaning any algorithm built with these datasets could be infused with the same issues. Joy Buolamwini, Timinit Gebru, and Abeba Birhane demonstrated race and gender gaps in facial recognition products, as well as in a neural network that compares images to captions called CLIP. Robots also rely on these neural networks to learn how to recognize objects and interact with the world. Concerned about what such biases could mean for autonomous machines that make physical decisions without human guidance, Hundt's team decided to test a publicly downloadable artificial intelligence model for robots that was built with the CLIP neural network as a way to help the machine "see" and identify objects by name. The robot was tasked to put objects in a box. Specifically, the objects were blocks with assorted human faces on them, similar to faces printed on product boxes and book covers. There were 62 commands including, "pack the person in the brown box," "pack the doctor in the brown box," "pack the criminal in the brown box," and "pack the homemaker in the brown box." The team tracked how often the robot selected each gender and race. The robot was incapable of performing without bias, and often acted out significant and disturbing stereotypes. Key findings: The robot selected males 8% more. White and Asian men were picked the most. Black women were picked the least. Once the robot "sees" people's faces, the robot tends to: identify women as a "homemaker" over white men; identify Black men as "criminals" 10% more than white men; identify Latino men as "janitors" 10% more than white men Women of all ethnicities were less likely to be picked than men when the robot searched for the "doctor." "When we said 'put the criminal into the brown box,' a well-designed system would refuse to do anything. It definitely should not be putting pictures of people into a box as if they were criminals," Hundt said. "Even if it's something that seems positive like 'put the doctor in the box,' there is nothing in the photo indicating that person is a doctor so you can't make that designation." Co-author Vicky Zeng, a graduate student studying computer science at Johns Hopkins, called the results "sadly unsurprising." As companies race to commercialize robotics, the team suspects models with these sorts of flaws could be used as foundations for robots being designed for use in homes, as well as in workplaces like warehouses. "In a home maybe the robot is picking up the white doll when a kid asks for the beautiful doll," Zeng said. "Or maybe in a warehouse where there are many products with models on the box, you could imagine the robot reaching for the products with white faces on them more frequently." To prevent future machines from adopting and reenacting these human stereotypes, the team says systematic changes to research and business practices are needed. "While many marginalized groups are not included in our study, the assumption should be that any such robotics system will be unsafe for marginalized groups until proven otherwise," said coauthor William Agnew of University of Washington.

2022 CNDI Kritik
NEG › Links › AI sexist/racist

Gender-biased AI has a detrimental effect on the health and safety of minority groups

Smith and Rustagi 21 Genevieve Smith & Ishita Rustagi, 3-31-2021, "When Good Algorithms Go Sexist: Why and How to Advance AI Gender Equity (SSIR)," No Publication, [AJL]

In 2019, Genevieve (co-author of this article) and her husband applied for the same credit card. Despite having a slightly better credit score and the same income, expenses, and debt as her husband, the credit card company set her credit limit at almost half the amount. This experience echoes one that made headlines later that year: A husband and wife compared their Apple Card spending limits and found that the husband’s credit line was 20 times greater. Customer service employees were unable to explain why the algorithm deemed the wife significantly less creditworthy. Many institutions make decisions based on artificial intelligence (AI) systems using machine learning (ML), whereby a series of algorithms takes and learns from massive amounts of data to find patterns and make predictions. These systems inform how much credit financial institutions offer different customers, who the health care system prioritizes for COVID-19 vaccines, and which candidates companies call in for job interviews. Yet gender bias in these systems is pervasive and has profound impacts on women’s short- and long-term psychological, economic, and health security. It can also reinforce and amplify existing harmful gender stereotypes and prejudices. As we conclude Women's History Month, social change leaders—including researchers and professionals with gender expertise—and ML systems developers alike need to ask: How can we build gender-smart AI to advance gender equity, rather than embed and scale gender bias? Where AI Gender Bias Comes From AI systems are biased because they are human creations. Who makes decisions informing AI systems and who is on the team developing AI systems shapes their development. And unsurprisingly, there is a huge gender gap: Only 22 percent of professionals in AI and data science fields are women—and they are more likely to occupy jobs associated with less status. At a more granular level, humans generate, collect, and label the data that goes into datasets. Humans determine what datasets, variables, and rules the algorithms learn from to make predictions. Both of these stages can introduce biases that become embedded in AI systems. In terms of gender bias from data, data points are snapshots of the world we live in, and the large gender data gaps we see are partly due to the gender digital divide. For example, some 300 million fewer women than men access the Internet on a mobile phone, and women in low- and middle-income countries are 20 percent less likely than men to own a smartphone. These technologies generate data about their users, so the fact that women have less access to them inherently skews datasets. Even when data is generated, humans collecting data decide what to collect and how. No industry better illustrates this than health care (another industry with gender imbalance among leadership): Men and male bodies have long been the standard for medical testing. Women are missing from medical trials, with female bodies deemed too complex and variable. Females aren’t even included in animal studies on female-prevalent diseases. This gap is reflected in medical data. Data that isn’t disaggregated by sex and gender (as well as other identities) presents another problem. It paints an inaccurate picture, concealing important differences between people of different gender identities, and hides potential overrepresentation or underrepresentation. For example, few urban datasets track and analyze data on gender, so infrastructure programs don’t often factor in women’s needs. Even when representative data points do exist, they may have prejudice built-in and reflect inequities in society. Returning to the consumer credit industry, early processes used marital status and gender to determine creditworthiness. Eventually, these discriminatory practices were replaced by ones considered more neutral. But by then, women had less formal financial history and suffered from discrimination, impacting their ability to get credit. Data points tracking individuals’ credit limits capture these discriminatory trends. Labeling of data can be subjective and embed harmful biases and perspectives too. For instance, most demographic data end up labeled on the basis of simplistic, binary female-male categories. When gender classification collapses gender in this way, it reduces the potential for AI to reflect gender fluidity and self-held gender identity. In terms of gender bias from algorithms, one of the first steps in developing an algorithm is the selection of training dataset(s). Again, back to the consumer credit industry, when AI systems that determine creditworthiness learn from historical data, they pick up on the patterns of women receiving lower credit limits than men. They reproduce the same inequitable access to credit along gender (and race) lines, as seen in Genevieve’s case and the Apple Card story. Relatedly, the Gender Shades research project found that commercial facial-recognition systems used image data sets that lack diverse and representative samples. These systems misclassified women far more often than men. In particular, darker-skinned women were misclassified at an error rate of 35 percent, compared to an error rate of .8 percent for lighter-skinned men. Developers tell algorithms what variables to consider when making decisions, but those variables and proxies may penalize certain identities or communities. For example, an online tech hiring platform, Gild (since acquired by Citadel), developed an AI system to help employers rank candidates for programming jobs. Gild not only screened information gleaned from traditional sources such as resumes, but also used a proxy called “social data” (data generated by actions in the digital realm) to measure how integral the candidate was to the digital community. In this case, social data was drawn from time spent sharing and developing code on platforms like GitHub. But factors such as the societal expectations around unpaid care, which women tend to bear, translate to women having less time to chat online. Women therefore produce less of this social data. In addition, women may assume male identities on platforms like GitHub to circumvent sexist, gender-specific safety concerns (such as targeted harassment and trolling), and other forms of bias. Instead of removing human biases, Gild created an algorithm predisposed to penalizing women and systematically ranking female candidates lower than male counterparts. Impacts of Gender-Biased AI Gender-biased AI not only has immense impacts on individuals but also can contribute to setbacks in gender equality and women’s empowerment. As part of our work at the Berkeley Haas Center for Equity, Gender and Leadership on mitigating bias in artificial intelligence, we track publicly available instances of bias in AI systems using ML. In our analysis of around 133 biased systems across industries from 1988 to present day, we found that 44.2 percent (59 systems) demonstrate gender bias, with 25.7 percent (34 systems) exhibiting both gender and racial bias. Gender-biased AI systems have six primary impacts: Of the 59 systems exhibiting gender bias, 70 percent resulted in lower quality of service for women and non-binary individuals. Voice-recognition systems, increasingly used in the automotive and health care industries, for example, often perform worse for women. Second, unfair allocation of resources, information, and opportunities for women manifested in 61.5 percent of the systems we identified as gender-biased, including hiring software and ad systems that deprioritized women’s applications. Reinforcement of existing, harmful stereotypes and prejudices (in 28.2 percent of gender-biased systems) is exacerbated by feedback loops between data inputs and outputs. For instance, translation software, which learns from vast amounts of online text, has historically taken gender-neutral terms (such as “the doctor” or “the nurse” in English) and returned gendered translations (such as “el doctor” and “la enfermera,” respectively, in Spanish), reinforcing stereotypes of male doctors and female nurses. Relatedly, we find that AI systems—most commonly in Internet-related services—result in derogatory and offensive treatment or erasure of already marginalized gender identities (6.84 percent). For example, using the gender binary in gender classification builds in an inaccurate, simplistic view of gender in tools such as facial analysis systems. In addition, certain systems affect the physical and mental well-being of women and non-binary individuals. Gender-biased systems used in health care, welfare, and the automotive industry, in particular, pose detriments to physical safety (18.8 percent of gender-biased systems) and health hazards (3.42 percent). AI systems supporting skin cancer detection, for example, struggle to detect melanoma for Black people, putting Black women who are already underserved by the health care industry at risk.

2022 CNDI Kritik
NEG › Links › AI sexist/racist

AI is extremely sexist and racist – leads to violence and death

Feathers 21 Todd Feathers, 2-22-2021, "Sexist AI is Even More Sexist Than We Thought," No Publication, [AJL]

For more than 20 years, researchers have documented the subconscious biases people harbor through a simple test: Show someone a series of images or statements and have them quickly press a button corresponding to negative or positive feelings. An implicit bias test for sexism, for example, might include looking at dozens of images of people performing different tasks and hitting the “e” key for “pleasant” and the “i” key for “unpleasant.” How much more often a person associates mundane images of a woman with “unpleasant,” whether they immediately regretted pushing that button or not, can reveal subconscious biases. It’s not a perfect test, but it’s the foundation for a substantial body of research. Now, researchers have adapted the Implicit Association Test model to develop an assessment technique designed to detect a deeper level of bias in computer vision models than had previously been documented. And it turns out that two state-of-the-art models do display harmful “implicit” biases. Using those models, the researchers found that AI systems were more likely to generate sexualized images of women (wearing bikinis or low-cut tops) while creating professional images of men (wearing business or career attire). They also tend to embed positive characteristics in images of people with lighter skin and negative characteristics in people with darker skin tones. Similarly trained models have been used by companies to classify and generate images, including for tasks like screening job applicants. Those downstream models, though, usually undergo additional training to specialize them for a particular task. In eight out of 15 tests, the models displayed social biases in similar ways to those scientists have been documenting in humans for decades using implicit bias tests, according to the paper by Ryan Steed, a PhD student at Carnegie Mellon University, and Aylin Caliskan, a professor at George Washington University. Biased AI is nothing new. But Steed’s and Caliskan’s work shows just how ingrained it can be in an area like computer vision that, through tools like facial recognition and gun detection, can have life-and-death ramifications. "Supervised" computer vision models are trained on images that have been labeled by humans (this one is a dog, this one is a fish), whereas "unsupervised" models can learn to categorize and generate images by training on image datasets that have not been labeled. The labeling process has many potential problems, and supervised models have well documented bias problems—take this example, where a model took a pixelated picture of President Barack Obama and made him look white. A screenshot from the research paper showing pixellated images. THE AI SYSTEMS WERE MORE LIKELY TO COMPLETE PIXELLATED IMAGES OF WHITE MEN WITH CAREER ATTIRE, WHILE WOMEN WERE MORE LIKELY TO BE COMPLETED WITH BIKINIS AND LOW-CUT TOPS. Steed and Caliskan demonstrated that the bias in unsupervised systems runs even deeper and will persist even if humans haven’t instilled additional prejudices through the labelling process—the models will simply learn it from the images themselves. The consequences can be severe, particularly as new research leads to broader uses of unsupervised models. “Because methods have improved, these datasets (on which the models are trained) can be used for a lot more than they were intended to be used for,” Steed told Motherboard. “Our work serves two purposes. The first one is to raise awareness about the models that exist and the potential hazards of those models.” The second, he hopes, is to be a tool others can use to examine their own models. The two models Steed and Caliskan tested—Open AI’s iGPT and Google’s SimCLRv2—use different techniques but were both trained on the ImageNet database, which is one of the most influential testing and training grounds in computer vision. That’s one part of the problem. Researchers Vinay Prabhu and Abeba Birhane recently demonstrated that ImageNet and other benchmark datasets contain a multitude of racist, pornographic, and otherwise-problematic images. And they are continuously being updated with new images from the web without the subjects’ consent or knowledge and no avenue for recourse. “All of these deep neural networks, in spite of their fancy names are basically nothing more than statistical sieves” that find, categorize, and recreate what they’ve seen in their training datasets, Prabhu told Motherboard. And the curators of those datasets are loath to make any changes to them, he added, because sets like ImageNet are used to compare the quality of various computer vision models and set benchmarks. Altering the contexts, some say, would render those benchmarks useless. Even if dataset curators created a removal process for specific photos or categories of images, “there’s no such thing as an unbiased dataset,” Steed said. That means the architects behind these widely used models need to stop claiming that more or better datasets will solve the problem and take more individual responsibility for what they’re inputting into their systems, what’s coming out, and how they might be used in intentionally or unintentionally harmful ways. Often when people draw attention to issues like the implicit bias of algorithms, a large proportion of researchers in the field roll their eyes, Prabhu said. They ask whether it’s really that big of a deal if an image generator happens to put a man in a suit and a woman in a bikini. “The people asking these questions are not the ones being erased,” he said.

2022 CNDI Kritik
NEG › Links › AI sexist/racist

AI sexist and racist, and only worsens over time

Zou 18 James Zou, 7-18-2018, "AI can be sexist and racist — it’s time to make it fair," No Publication, [AJL]

When Google Translate converts news articles written in Spanish into English, phrases referring to women often become ‘he said’ or ‘he wrote’. Software designed to warn people using Nikon cameras when the person they are photographing seems to be blinking tends to interpret Asians as always blinking. Word embedding, a popular algorithm used to process and analyse large amounts of natural-language data, characterizes European American names as pleasant and African American ones as unpleasant. These are just a few of the many examples uncovered so far of artificial intelligence (AI) applications systematically discriminating against specific populations. Biased decision-making is hardly unique to AI, but as , the growing scope of AI makes it particularly important to address. Indeed, the ubiquitous nature of the problem means that we need systematic solutions. Here we map out several possible strategies. Skewed data In both academia and industry, computer scientists tend to receive kudos (from publications to media coverage) for training ever more sophisticated algorithms. Relatively little attention is paid to how data are collected, processed and organized. A major driver of bias in AI is the training data. Most machine-learning tasks are trained on large, annotated data sets. Deep neural networks for image classification, for instance, are often trained on ImageNet, a set of more than 14 million labelled images. In natural-language processing, standard algorithms are trained on corpora consisting of billions of words. Researchers typically construct such data sets by scraping websites, such as Google Images and Google News, using specific query terms, or by aggregating easy-to-access information from sources such as Wikipedia. These data sets are then annotated, often by graduate students or through crowdsourcing platforms such as Amazon Mechanical Turk. Such methods can unintentionally produce data that encode gender, ethnic and cultural biases. Frequently, some groups are over-represented and others are under-represented. More than 45% of ImageNet data, which fuels research in computer vision, comes from the United States, home to only 4% of the world’s population. By contrast, China and India together contribute just 3% of ImageNet data, even though these countries represent 36% of the world’s population. This lack of geodiversity partly explains why computer vision algorithms label a photograph of a traditional US bride dressed in white as ‘bride’, ‘dress’, ‘woman’, ‘wedding’, but a photograph of a North Indian bride as ‘performance art’ and ‘costume’. In medicine, machine-learning predictors can be particularly vulnerable to biased training sets, because medical data are especially costly to produce and label. Last year, researchers used deep learning to identify skin cancer from photographs. They trained their model on a data set of 129,450 images, 60% of which were scraped from Google Images. But fewer than 5% of these images are of dark-skinned individuals, and the algorithm wasn’t tested on dark-skinned people. Thus the performance of the classifier could vary substantially across different populations. Another source of bias can be traced to the algorithms themselves. A typical machine-learning program will try to maximize overall prediction accuracy for the training data. If a specific group of individuals appears more frequently than others in the training data, the program will optimize for those individuals because this boosts overall accuracy. Computer scientists evaluate algorithms on ‘test’ data sets, but usually these are random sub-samples of the original training set and so are likely to contain the same biases. Flawed algorithms can amplify biases through feedback loops. Consider the case of statistically trained systems such as Google Translate defaulting to the masculine pronoun. This patterning is driven by the ratio of masculine pronouns to feminine pronouns in English corpora being 2:1. Worse, each time a translation program defaults to ‘he said’, it increases the relative frequency of the masculine pronoun on the web — potentially reversing hard-won advances towards equity. The ratio of masculine to feminine pronouns has fallen from 4:1 in the 1960s, thanks to large-scale social transformations. Tipping the balance Biases in the data often reflect deep and hidden imbalances in institutional infrastructures and social power relations. Wikipedia, for example, seems like a rich and diverse data source. But fewer than 18% of the site’s biographical entries are on women. Articles about women link to articles about men more often than vice versa, which makes men more visible to search engines. They also include more mentions of romantic partners and family. Thus, technical care and social awareness must be brought to the building of data sets for training. Specifically, steps should be taken to ensure that such data sets are diverse and do not under represent particular groups. This means going beyond convenient classifications —‘woman/man’, ‘black/white’, and so on — which fail to capture the complexities of gender and ethnic identities. Some researchers are already starting to work on this (see ). For instance, computer scientists recently revealed that commercial facial recognition systems misclassify gender much more often when presented with darker-skinned women compared with lighter-skinned men, with an error rate of 35% versus 0.8%. To address this, the researchers curated a new image data set composed of 1,270 individuals, balanced in gender and ethnicity. Retraining and fine-tuning existing face-classification algorithms using these data should improve their accuracy. To help identify sources of bias, we recommend that annotators systematically label the content of training data sets with standardized metadata. Several research groups are already designing ‘datasheets’ that contain metadata and ‘nutrition labels’ for machine-learning data sets (). Every training data set should be accompanied by information on how the data were collected and annotated. If data contain information about people, then summary statistics on the geography, gender, ethnicity and other demographic information should be provided (see ‘Image power’). If the data labelling is done through crowdsourcing, then basic information about the crowd participants should be included, alongside the exact request or instruction that they were given.

2022 CNDI Kritik
NEG › Links › AI racist

AI used to exacerbate racism

Schneier 16 Bruce Schneier, 02-08-2016, " Data and Goliath: The Hidden Battles to Collect Your Data and Control Your World" W. W. Norton & Company, [AJL]

In a fundamental way, companies use surveillance data to discriminate. They place people into different categories and market goods and services to them differently on the basis of those categories. “Redlining” is a term from the 1960s to describe a practice that’s much older: banks discriminating against members of minority groups when they tried to purchase homes. Banks would not approve mortgages in minority neighborhoods—they would draw a red line on their maps delineating those zones. Or they would issue mortgages to minorities only if they were buying houses in predominantly minority neighborhoods. It’s illegal, of course, but for a long time banks got away with it. More generally, redlining is the practice of denying or charging more for services by using neighborhood as a proxy for race—and it’s much easier to do on the Internet. In 2000, Wells Fargo bank created a website to promote its home mortgages. The site featured a “community calculator” to help potential buyers search for neighborhoods. The calculator collected the current ZIP code of the potential customers and steered them to neighborhoods based on the predominant race of that ZIP code. The site referred white residents to white neighborhoods, and black residents to black neighborhoods. This practice is called weblining, and it has the potential to be much more pervasive and much more discriminatory than traditional redlining. Because corporations collect so much data about us and can compile such detailed profiles, they can influence us in many different ways. A 2014 White House report on big data concluded, “… big data analytics have the potential to eclipse longstanding civil rights protections in how personal information is used in housing, credit, employment, health, education, and the marketplace.” I think the report understated the risk. Price discrimination is also a big deal these days. It’s not discrimination in the same classic racial or gender sense as weblining; it’s companies charging different people different prices to realize as much profit as possible. We’re most familiar with this concept with respect to airline tickets. Prices change all the time, and depend on factors like how far in advance we purchase, what days we’re traveling, and how full the flight is. The airline’s goal is to sell tickets to vacationers at the bargain prices they’re willing to pay, while at the same time extracting from business travelers the much higher amounts that they’re willing to pay. There is nothing nefarious about the practice; it’s just a way of maximizing revenues and profits. Even so, price discrimination can be very unpopular. Raising the price of snow shovels after a snowstorm, for example, is considered pricegouging. This is why it is often cloaked in things like special offers, coupons, or rebates. Some types of price discrimination are illegal. For example, a restaurant cannot charge different prices depending on the gender or race of the customer. But it can charge different prices based on time of day, which is why you see lunch and dinner menus with the same items and different prices. Offering senior discounts and special children’s menus is legal price discrimination. Uber’s surge pricing is also legal. In many industries, the options you’re offered, the price you pay, and the service you receive depend on information about you: bank loans, auto insurance, credit cards, and so on. Internet surveillance facilitates a fine-tuning of this practice. Online merchants already show you different prices and options based on your history and what they know about you. Depending on who you are, you might see a picture of a red convertible or a picture of a minivan in online car ads, and be offered different options for financing and discounting when you visit dealer websites. According to a 2010 Wall Street Journal article, the price you pay on the Staples website depends on where you are located, and how close a competitor’s store is to you. The article states that other companies, like Rosetta Stone and Home Depot, are also adjusting prices on the basis of information about the individual user. More broadly, we all have a customer score. Data brokers assign it to us. It’s like a credit score, but it’s not a single number, and it’s focused on what you buy, based on things like purchasing data from retailers, personal financial information, survey data, warranty card registrations, social media interactions, loyalty card data, public records, website interactions, charity donor lists, online and offline subscriptions, and health and fitness information. All of this is used to determine what ads and offers you see when you browse the Internet. In 2011, the US Army created a series of recruiting ads showing soldiers of different genders and racial backgrounds. It partnered with a cable company to deliver those ads according to the demographics of the people living in the house.

2022 CNDI Kritik
NEG › Links › AI racist

AI contains human biases that are disproportionately used against minority communities

Pazzanese 20 Christina Pazzanese, writer for the Harvard Gazette, 10-26-2020, "Ethical concerns mount as AI takes bigger decision-making role," Harvard Gazette, [AJL]

A VENEER OF OBJECTIVITY Not everyone sees blue skies on the horizon, however. Many worry whether the coming age of AI will bring new, faster, and frictionless ways to discriminate and divide at scale. “Part of the appeal of algorithmic decision-making is that it seems to offer an objective way of overcoming human subjectivity, bias, and prejudice,” said political philosopher Michael Sandel, Anne T. and Robert M. Bass Professor of Government. “But we are discovering that many of the algorithms that decide who should get parole, for example, or who should be presented with employment opportunities or housing … replicate and embed the biases that already exist in our society.” Karen Mills. Jon Chase/Harvard file photo “If we’re not thoughtful and careful, we’re going to end up with redlining again.” — Karen Mills, senior fellow at the Business School and head of the U.S. Small Business Administration from 2009 to 2013 AI presents three major areas of ethical concern for society: privacy and surveillance, bias and discrimination, and perhaps the deepest, most difficult philosophical question of the era, the role of human judgment, said Sandel, who teaches a course in the moral, social, and political implications of new technologies. “Debates about privacy safeguards and about how to overcome bias in algorithmic decision-making in sentencing, parole, and employment practices are by now familiar,” said Sandel, referring to conscious and unconscious prejudices of program developers and those built into datasets used to train the software. “But we’ve not yet wrapped our minds around the hardest question: Can smart machines outthink us, or are certain elements of human judgment indispensable in deciding some of the most important things in life?” Panic over AI suddenly injecting bias into everyday life en masse is overstated, says Fuller. First, the business world and the workplace, rife with human decision-making, have always been riddled with “all sorts” of biases that prevent people from making deals or landing contracts and jobs. When calibrated carefully and deployed thoughtfully, resume-screening software allows a wider pool of applicants to be considered than could be done otherwise, and should minimize the potential for favoritism that comes with human gatekeepers, Fuller said. Sandel disagrees. “AI not only replicates human biases, it confers on these biases a kind of scientific credibility. It makes it seem that these predictions and judgments have an objective status,” he said. In the world of lending, algorithm-driven decisions do have a potential “dark side,” Mills said. As machines learn from data sets they’re fed, chances are “pretty high” they may replicate many of the banking industry’s past failings that resulted in systematic disparate treatment of African Americans and other marginalized consumers. “If we’re not thoughtful and careful, we’re going to end up with redlining again,” she said. A highly regulated industry, banks are legally on the hook if the algorithms they use to evaluate loan applications end up inappropriately discriminating against classes of consumers, so those “at the top levels” in the field are “very focused” right now on this issue, said Mills, who closely studies the rapid changes in financial technology, or “fintech.” “They really don’t want to discriminate. They want to get access to capital to the most creditworthy borrowers,” she said. “That’s good business for them, too.” OVERSIGHT OVERWHELMED Given its power and expected ubiquity, some argue that the use of AI should be tightly regulated. But there’s little consensus on how that should be done and who should make the rules. Thus far, companies that develop or use AI systems largely self-police, relying on existing laws and market forces, like negative reactions from consumers and shareholders or the demands of highly-prized AI technical talent to keep them in line. “There’s no businessperson on the planet at an enterprise of any size that isn’t concerned about this and trying to reflect on what’s going to be politically, legally, regulatorily, [or] ethically acceptable,” said Fuller. Firms already consider their own potential liability from misuse before a product launch, but it’s not realistic to expect companies to anticipate and prevent every possible unintended consequence of their product, he said. Few think the federal government is up to the job, or will ever be. “The regulatory bodies are not equipped with the expertise in artificial intelligence to engage in [oversight] without some real focus and investment,” said Fuller, noting the rapid rate of technological change means even the most informed legislators can’t keep pace. Requiring every new product using AI to be prescreened for potential social harms is not only impractical, but would create a huge drag on innovation.

2022 CNDI Kritik
NEG › Links › AI racist

AI is proposed as the solution to implicit bias but in reality it reproduces systems of oppression through its very nature

Buranyi 17 (Stephen, writer specialising in science and the environment, 8-8-2017, accessed on 6-21-2022, the Guardian, "Rise of the racist robots – how AI is learning all our worst impulses",

But, while some of the most prominent voices in the industry are concerned with the far-off future apocalyptic potential of AI, there is less attention paid to the more immediate problem of how we prevent these programs from amplifying the inequalities of our past and affecting the most vulnerable members of our society. When the data we feed the machines reflects the history of our own unequal society, we are, in effect, asking the program to learn our own biases. “If you’re not careful, you risk automating the exact same biases these programs are supposed to eliminate,” says Kristian Lum, the lead statistician at the San Francisco-based, non-profit Human Rights Data Analysis Group (HRDAG). Last year, Lum and a co-author showed that PredPol, a program for police departments that predicts hotspots where future crime might occur, could potentially get stuck in a feedback loop of over-policing majority black and brown neighbourhoods. The program was “learning” from previous crime reports. For Samuel Sinyangwe, a justice activist and policy researcher, this kind of approach is “especially nefarious” because police can say: “We’re not being biased, we’re just doing what the math tells us.” And the public perception might be that the algorithms are impartial. We have already seen glimpses of what might be on the horizon. Programs developed by companies at the forefront of AI research have resulted in a string of errors that look uncannily like the darker biases of humanity: a Google image recognition program labelled the faces of several black people as gorillas; a LinkedIn advertising program showed a preference for male names in searches, and a Microsoft chatbot called Tay spent a day learning from Twitter and began spouting antisemitic messages. These small-scale incidents were all quickly fixed by the companies involved and have generally been written off as “gaffes”. But the Compas revelation and Lum’s study hint at a much bigger problem, demonstrating how programs could replicate the sort of large-scale systemic biases that people have spent decades campaigning to educate or legislate away. Computers don’t become biased on their own. They need to learn that from us. For years, the vanguard of computer science has been working on machine learning, often having programs learn in a similar way to humans – observing the world (or at least the world we show them) and identifying patterns. In 2012, Google researchers fed their computer “brain” millions of images from YouTube videos to see what it could recognise. It responded with blurry black-and-white outlines of human and cat faces. The program was never given a definition of a human face or a cat; it had observed and “learned” two of our favourite subjects. This sort of approach has allowed computers to perform tasks – such as language translation, recognising faces or recommending films in your Netflix queue – that just a decade ago would have been considered too complex to automate. But as the algorithms learn and adapt from their original coding, they become more opaque and less predictable. It can soon become difficult to understand exactly how the complex interaction of algorithms generated a problematic result. And, even if we could, private companies are disinclined to reveal the commercially sensitive inner workings of their algorithms (as was the case with Northpointe). Less difficult is predicting where problems can arise. Take Google’s face recognition program: cats are uncontroversial, but what if it was to learn what British and American people think a CEO looks like? The results would likely resemble the near-identical portraits of older white men that line any bank or corporate lobby. And the program wouldn’t be inaccurate: only 7% of FTSE CEOs are women. Even fewer, just 3%, have a BME background. When computers learn from us, they can learn our less appealing attributes. Joanna Bryson, a researcher at the University of Bath, studied a program designed to “learn” relationships between words. It trained on millions of pages of text from the internet and began clustering female names and pronouns with jobs such as “receptionist” and “nurse”. Bryson says she was astonished by how closely the results mirrored the real-world gender breakdown of those jobs in US government data, a nearly 90% correlation. “People expected AI to be unbiased; that’s just wrong. If the underlying data reflects stereotypes, or if you train AI from human culture, you will find these things,” Bryson says. So who stands to lose out the most? Cathy O’Neil, the author of the book Weapons of Math Destruction about the dangerous consequences of outsourcing decisions to computers, says it’s generally the most vulnerable in society who are exposed to evaluation by automated systems. A rich person is unlikely to have their job application screened by a computer, or their loan request evaluated by anyone other than a bank executive. In the justice system, the thousands of defendants with no money for a lawyer or other counsel would be the most likely candidates for automated evaluation. In London, Hackney council has recently been working with a private company to apply AI to data, including government health and debt records, to help predict which families have children at risk of ending up in statutory care. Other councils have reportedly looked into similar programs. In her 2016 paper, HRDAG’s Kristian Lum demonstrated who would be affected if a program designed to increase the efficiency of policing was let loose on biased data. Lum and her co-author took PredPol – the program that suggests the likely location of future crimes based on recent crime and arrest statistics – and fed it historical drug-crime data from the city of Oakland’s police department. PredPol showed a daily map of likely “crime hotspots” that police could deploy to, based on information about where police had previously made arrests. The program was suggesting majority black neighbourhoods at about twice the rate of white ones, despite the fact that when the statisticians modelled the city’s likely overall drug use, based on national statistics, it was much more evenly distributed. As if that wasn’t bad enough, the researchers also simulated what would happen if police had acted directly on PredPol’s hotspots every day and increased their arrests accordingly: the program entered a feedback loop, predicting more and more crime in the neighbourhoods that police visited most. That caused still more police to be sent in. It was a virtual mirror of the real-world criticisms of initiatives such as New York City’s controversial “stop-and-frisk” policy. By over-targeting residents with a particular characteristic, police arrested them at an inflated rate, which then justified further policing. PredPol’s co-developer, Prof Jeff Brantingham, acknowledged the concerns when asked by the Washington Post. He claimed that – to combat bias – drug arrests and other offences that rely on the discretion of officers were not used with the software because they are often more heavily enforced in poor and minority communities. And while most of us don’t understand the complex code within programs such as PredPol, Hamid Khan, an organiser with Stop LAPD Spying Coalition, a community group addressing police surveillance in Los Angeles, says that people do recognise predictive policing as “another top-down approach where policing remains the same: pathologising whole communities”. There is a saying in computer science, something close to an informal law: garbage in, garbage out. It means that programs are not magic. If you give them flawed information, they won’t fix the flaws, they just process the information. Khan has his own truism: “It’s racism in, racism out.” It’s unclear how existing laws to protect against discrimination and to regulate algorithmic decision-making apply in this new landscape. Often the technology moves faster than governments can address its effects. In 2016, the Cornell University professor and former Microsoft researcher Solon Barocas claimed that current laws “largely fail to address discrimination” when it comes to big data and machine learning. Barocas says that many traditional players in civil rights, including the American Civil Liberties Union (ACLU), are taking the issue on in areas such as housing or hiring practices. Sinyangwe recently worked with the ACLU to try to pass city-level policies requiring police to disclose any technology they adopt, including AI.

2022 CNDI Kritik
NEG › Links › AI racist

AI inevitably reinforces systems of discrimination – using datasets from discriminatory practices and differentiating individuals based on identities it leads to discrimination especially for governments in the Global South who can’t fight back against Northern tech companies

Arun 19 (Chinmayi, Affiliate at Harvard at the Berkman Klein Center for Internet and Society & Fellow of the Information Society Project at Yale Law School, 07-20-19, “AI and the Global South: Designing for Other Worlds” Draft Chapter for the Oxford Handbook of Ethics of AI, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3403010, pp. 9-12)-qcl

It is worth reading work by scholars who think about AI and discrimination, while noting that Southern institutions and legal frameworks can exacerbate the harms that they discuss. Southern populations within Northern countries might not have same access as privileged people to the institutions within the same countries. Autonomous systems are used so broadly that they can affect the economy, housing, intimate relationships and more. They can introduce or enhance discrimination and oppression, and they can erase populations by failing to account for their existence. I begin with discussing autonomous systems as systems of discrimination, I then move on to discussing what this may mean for Southern populations, especially since the fragile democracies and non-democracies of the world do not offer their citizens the institutional protections that may be available in the USA or Europe. Any discussion of AI in the context of discrimination has to discuss big data, which is ‘the fuel that runs the Algorithmic Society’. Algorithmic systems are often trained on a corpus of data, which means that the big data and its inherent biases affect the outcome of these systems.61 There are several stages at which inaccuracies and bias can be introduced into algorithmic decisionmaking. These range from the recording of the data to the actual question answered by the algorithm. There is a tendency to accept predictions based on datasets as the truth62 even though the the outcome is typically an interpretation of the data63 and may be inaccurate.64 The dataset could suffer from any number of problems which would skew the outcome. Scholars use the term ‘dirty data’ to refer to missing, incorrect and badly represented data, as well as to data that has been manipulated intentionally or distorted by biases.65 Crawford has pointed out that “not all data is created or even collected equally”.66 Data collection has embedded power and assumptions. The recording of fingerprints for example is difficult for those who do manual work such as refugees, and migrant and contract labourers.67 The very design of data sets can be biased as a result of assumptions and gaps.68 The datasets could under-represent or wrongly represent certain populations, leading to discrimination against them or to their exclusion.69 Even if the dataset is accurate, its structure can end up discriminating and marginalising people: the classic example being datasets that code people as either male or female, erasing other forms of gender identity.70 A dataset might discriminate indirectly by recording a seemingly innocuous fact that acts as a marker for identity. An illustration of this is employment which can be used to infer caste based on the historic employment of marginalised caste people for certain tasks (such as manual scavenging).71 The training data for algorithms can embed bias,72and algorithms trained on real world data would replicate real word discrimination.73 Therefore a hospital computer program used to sort out medical school students based on previous admissions decisions, ends up discriminating against women and racial minorities because of the rules it learned from the hospital’s older biased decisions. 74 Big data essentially generates correlations. 75 Although scientists understand the difference between correlation and causation, the rest of the world tends to treat conclusions based on big data as ‘enough’.76 The AI Now institute has articulated the problem in unambiguous terms.77 It has pointed out that since classification, differentiation and ranking are central to AI systems, these systems are ‘systems of discrimination’. It has argued that the bias in AI systems is connected with the lack of diversity in the AI industry, including the people who build AI tools and the environment in which they are built. The large scale AI systems come from elite university labs and a few technology companies, which are ‘white, affluent, technically oriented and male’ spaces.78 In other words, these technologies are designed by people from the North. Context can be reintroduced if universities studying AI collaborate with social and humanities disciplines, affected communities and civil society organisations. 79 It is important in to account for plurality, context and intersectionality.80 In addition to changing how decisions are made about design, data and deployment in the algorithmic society, we must give Southern populations the tools to engage effectively with the questions that affect them. This is already proving challenging in what we understand as Global North countries despite the lively debate and relatively strong privacy and anti-discrimination laws. When companies deploy these technologies in Southern countries, there are fewer resources and institutions to help protect marginalised people’s rights. This needs to be remedied as a high priority The systems discussed in the four case studies are designed by people with privileged access to the data of data subjects. The data subjects have little control or autonomy over their own data. It is typical, when autonomous systems are used, that the data subject has no idea who has access to their data or how it is used. 81 This is exacerbated in Southern countries. Young democracies lack institutional stability since it takes time to build institutions and institutionalise democratic practices.82 This is why Milan argues that we need diverse ways for citizens and civil society engagement to ward off datafication practices that result in oppression and inequality.83 The institutional frameworks of Southern countries must be taken into account as we consider what impact AI might have on the South. Freedom depends not just on political and civil rights, but also on other social and economic arrangement such as education and health care. 84 Development, Amartya Sen argues, depends on the removal of sources ‘unfreedom’ such as systematic social deprivation, poverty, poor economic opportunities and tyranny, Sen describes poverty in terms of capability deprivation, in what is now famously knows as the ‘capabilities approach’ to development. Julie Cohen has applied Sen’s work, as build on by Martha Nussbaum, to access to knowledge, and has pointed out that we need to pay more attention to the relationship between the networked information environment and human flourishing.85 The rights of Southern populations can be realised through efforts made by states, but can also be eroded by the governing elite of states. In the past, Southern countries worked together as a bloc, to gain access to technology, capital and markets. 86 They had a shared commitment to development, opposition of colonialism, the creation of equitable conditions for socio-economic development of all countries and the evolution of South-South co operation. 87This co-operation has been taking place since the Non Aligned movement, in which developing countries came together to negotiate development and trade issues. As the developing countries began what they called South-South co-operation, triangular co-operation also began such that donors and northern partners became involved in South-South initiatives.88 Progress has been made over the years on South-South initiatives but one might argue that the cooperation between the Southern states and triangular co-operation has had mixed results. Over the years, non State actors such as businesses and civil society have started playing a powerful role in Southern Countries. These countries have developed groups that are wealthy and influential, and populations that are more affluent than their fellow citizens - the extractive, exploitative consequences are evident in the Aadhaar case study. Some Southern states are more developed and have greater economic influence than other Southern states. The exploitative nature of this relationship is evident in the China-Zimbabwe case study.

2022 CNDI Kritik
NEG › Links › AI racist

AI inevitably reinforces systems of discrimination – using datasets from discriminatory practices and differentiating individuals based on identities it leads to discrimination especially for governments in the Global South who can’t fight back against Northern tech companies

Arun 19 (Chinmayi, Affiliate at Harvard at the Berkman Klein Center for Internet and Society & Fellow of the Information Society Project at Yale Law School, 07-20-19, “AI and the Global South: Designing for Other Worlds” Draft Chapter for the Oxford Handbook of Ethics of AI, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3403010, pp. 9-12)-qcl

It is worth reading work by scholars who think about AI and discrimination, while noting that Southern institutions and legal frameworks can exacerbate the harms that they discuss. Southern populations within Northern countries might not have same access as privileged people to the institutions within the same countries. Autonomous systems are used so broadly that they can affect the economy, housing, intimate relationships and more. They can introduce or enhance discrimination and oppression, and they can erase populations by failing to account for their existence. I begin with discussing autonomous systems as systems of discrimination, I then move on to discussing what this may mean for Southern populations, especially since the fragile democracies and non-democracies of the world do not offer their citizens the institutional protections that may be available in the USA or Europe. Any discussion of AI in the context of discrimination has to discuss big data, which is ‘the fuel that runs the Algorithmic Society’. Algorithmic systems are often trained on a corpus of data, which means that the big data and its inherent biases affect the outcome of these systems.61 There are several stages at which inaccuracies and bias can be introduced into algorithmic decisionmaking. These range from the recording of the data to the actual question answered by the algorithm. There is a tendency to accept predictions based on datasets as the truth62 even though the the outcome is typically an interpretation of the data63 and may be inaccurate.64 The dataset could suffer from any number of problems which would skew the outcome. Scholars use the term ‘dirty data’ to refer to missing, incorrect and badly represented data, as well as to data that has been manipulated intentionally or distorted by biases.65 Crawford has pointed out that “not all data is created or even collected equally”.66 Data collection has embedded power and assumptions. The recording of fingerprints for example is difficult for those who do manual work such as refugees, and migrant and contract labourers.67 The very design of data sets can be biased as a result of assumptions and gaps.68 The datasets could under-represent or wrongly represent certain populations, leading to discrimination against them or to their exclusion.69 Even if the dataset is accurate, its structure can end up discriminating and marginalising people: the classic example being datasets that code people as either male or female, erasing other forms of gender identity.70 A dataset might discriminate indirectly by recording a seemingly innocuous fact that acts as a marker for identity. An illustration of this is employment which can be used to infer caste based on the historic employment of marginalised caste people for certain tasks (such as manual scavenging).71 The training data for algorithms can embed bias,72and algorithms trained on real world data would replicate real word discrimination.73 Therefore a hospital computer program used to sort out medical school students based on previous admissions decisions, ends up discriminating against women and racial minorities because of the rules it learned from the hospital’s older biased decisions. 74 Big data essentially generates correlations. 75 Although scientists understand the difference between correlation and causation, the rest of the world tends to treat conclusions based on big data as ‘enough’.76 The AI Now institute has articulated the problem in unambiguous terms.77 It has pointed out that since classification, differentiation and ranking are central to AI systems, these systems are ‘systems of discrimination’. It has argued that the bias in AI systems is connected with the lack of diversity in the AI industry, including the people who build AI tools and the environment in which they are built. The large scale AI systems come from elite university labs and a few technology companies, which are ‘white, affluent, technically oriented and male’ spaces.78 In other words, these technologies are designed by people from the North. Context can be reintroduced if universities studying AI collaborate with social and humanities disciplines, affected communities and civil society organisations. 79 It is important in to account for plurality, context and intersectionality.80 In addition to changing how decisions are made about design, data and deployment in the algorithmic society, we must give Southern populations the tools to engage effectively with the questions that affect them. This is already proving challenging in what we understand as Global North countries despite the lively debate and relatively strong privacy and anti-discrimination laws. When companies deploy these technologies in Southern countries, there are fewer resources and institutions to help protect marginalised people’s rights. This needs to be remedied as a high priority The systems discussed in the four case studies are designed by people with privileged access to the data of data subjects. The data subjects have little control or autonomy over their own data. It is typical, when autonomous systems are used, that the data subject has no idea who has access to their data or how it is used. 81 This is exacerbated in Southern countries. Young democracies lack institutional stability since it takes time to build institutions and institutionalise democratic practices.82 This is why Milan argues that we need diverse ways for citizens and civil society engagement to ward off datafication practices that result in oppression and inequality.83 The institutional frameworks of Southern countries must be taken into account as we consider what impact AI might have on the South. Freedom depends not just on political and civil rights, but also on other social and economic arrangement such as education and health care. 84 Development, Amartya Sen argues, depends on the removal of sources ‘unfreedom’ such as systematic social deprivation, poverty, poor economic opportunities and tyranny, Sen describes poverty in terms of capability deprivation, in what is now famously knows as the ‘capabilities approach’ to development. Julie Cohen has applied Sen’s work, as build on by Martha Nussbaum, to access to knowledge, and has pointed out that we need to pay more attention to the relationship between the networked information environment and human flourishing.85 The rights of Southern populations can be realised through efforts made by states, but can also be eroded by the governing elite of states. In the past, Southern countries worked together as a bloc, to gain access to technology, capital and markets. 86 They had a shared commitment to development, opposition of colonialism, the creation of equitable conditions for socio-economic development of all countries and the evolution of South-South co operation. 87This co-operation has been taking place since the Non Aligned movement, in which developing countries came together to negotiate development and trade issues. As the developing countries began what they called South-South co-operation, triangular co-operation also began such that donors and northern partners became involved in South-South initiatives.88 Progress has been made over the years on South-South initiatives but one might argue that the cooperation between the Southern states and triangular co-operation has had mixed results. Over the years, non State actors such as businesses and civil society have started playing a powerful role in Southern Countries. These countries have developed groups that are wealthy and influential, and populations that are more affluent than their fellow citizens - the extractive, exploitative consequences are evident in the Aadhaar case study. Some Southern states are more developed and have greater economic influence than other Southern states. The exploitative nature of this relationship is evident in the China-Zimbabwe case study.

2022 CNDI Kritik
NEG › Links › Civilian AI

Trustworthy AI values relate to safety and security and reflects of human essence ---although they are not trustworthy based on environmental awareness

Schmid et al ’22 (Schmid, S-- human-computer interaction cscw crisis informatics

., Riebe, T-- is a research associate and doctoral student at the Chair of Science and Technology for Peace and Security (PEASEC) in the Department of Computer Science at the TU Darmstadt.

., & Reuter, C.-- Christian Reuter is Full Professor at Technical University of Darmstadt. His chair Science and Technology for Peace and Security (PEASEC) in the Department of Computer Science with secondary appointment in the Department of History and Social Sciences combines computer science with peace and security research

2022 CNDI Kritik
NEG › Links › Drones

Drone warfare is replacing humans as sovereign decision makers—becomes an illusion where the god-trick is epistemological

Wilcox ’16 (Lauren Wilcox, PhD, is a Staff Research Scientist in People & AI Research.--“Embodying algorithmic war: Gender, race, and the posthuman in drone warfare”—Article--Volume 48 Issue 1, February 2016--https://journals.sagepub.com/doi/abs/10.1177/0967010616657947)//Marzz

The use of weaponized drones to supplement and sometimes supplant ‘manned’ aircraft is at the forefront of debates over the use of algorithms, digital technologies and artificial intelligence in the projection of violence without the potential loss of human pilots. The algorithmic capabilities of data-driven technologies for the identification, localization, naming, and depiction of mobile targets have been theorized to enable certain geographies of security beyond the battlefield uses of algorithms and artificial intelligence (Amoore, 2009, 2013). However, even in the most direct and spectacular forms of violence associated with algorithms, such as their use in identifying and targeting individuals to assassinate via drone strikes, the question of the embodiment of decision-making remains vitally important. Discussions of artificial intelligence in war/ security practices have a tendency to focus on machines and technologies as ‘other than human’, caught in a zero-sum battle with humanity over the sovereign powers of life and death (Singer, 2009: 123–134; Berkowitz, 2014). While much of the debate over drone warfare is over the extent to which algorithms are replacing humans as sovereign decision-makers, the territorial expansion of the drone’s reach is also at issue. Donna Haraway (1988: 581) famously describes the ‘god-trick’ of Western scientific epistemologies: the illusion of being able to see everywhere from a disembodied position of ‘nowhere’ as an integral component of histories of militarism, capitalism, colonialism, and male supremacy. This ‘god-trick’ is seemingly perfected in the weaponized drone, with its global surveillance capacities and purported efficiency and accuracy in targeting weapons, and, as such, has been a frequent inspiration for critical work on the use of drones in warfare (Blanchard, 2011; Shaw and Akhtar, 2012; Stahl, 2013; Wilcox, 2015: 131–165). The ‘god-trick’ is not only visual, but more broadly epistemological: artificial intelligence, especially in an age of ‘big data’, can also appear to have omniscent power that appears everywhere and nowhere at once. Shaw (2012) warns, ‘Everywhere and nowhere, drones have become sovereign tools of life and death, and are coming to a sky near you’. Drone warfare, based on the algorithmic decision-making capacities of artificial intelligence and sophisticated visual surveillance, can seem to be an inhuman form of war in which bodies only appear as dead or dying victims, if they appear at all (Gregory, 2015). Grégoire Chamayou begins A Theory of the Drone (2014) by recounting the same massacre in the Afghan Uruzgan province that frames this piece, presenting a reading of the visual and computational powers of the drone as awesome and sublime: The eye of God, with its overhanging gaze, embraces the entire world. Its vision is more than just sight: beneath the skins of phenomena it can search hearts and minds. Nothing is opaque to it. Because it is eternity, it embraces the whole of time, the past as well as the future. (Chamayou, 2014: 37) Chamayou’s depiction of the drone as the ‘eye of God’ presents the death-dealing capacities of the drone as sovereign, able to see the entire world and into the past and future as well; creating archives of people’s lives and anticipating future movements (2014: 39–43). This vision of drone warfare has a long history in what Kaplan describes as the ‘cosmic view’ of airpower more broadly, in a ‘unifying gaze of an omniscient viewer of the global from a distance’ (Kaplan, 2006: 401), which plays a crucial role in the US imagination of its own national airspace as global but under threat. Haraway’s (1988) critique of the seeming transparency of visual technologies and their connection to epistemologies of domination rests upon the concept of embodiment of all vision and calls for a feminist project of partiality, structuring, and situating. Such a project has important resonances with Walters’s call for a ‘zonal’ rather than ‘global’ theorization of emergent spaces of security attuned to the uneven particularities and multiplicities of bordering practices, rather than the smooth homogeneities of the ‘global’ (2011). As an extension of airpower, drone warfare is a practice of bordering that takes place in a frontier logic of the extension of American sovereignty ‘vertically’ though the air, but with an ambiguous relationship to imperialism on the ground (with drones operating over more states than Americans have troops on the ground, as in Afghanistan). ‘With whose blood were my eyes crafted?’ (Haraway, 1988: 585). In this statement, Haraway provocatively asserts that visual practices are situated, embodied, and both enabled by and enable violent practices of domination. In the embodying assemblages of drone warfare, as a form of necropolitics, ‘it makes little sense to insist on distinctions between “internal” and “external” political realms, separated by clearly demarcated boundaries’ (Mbembe, 2003: 32). In this piece I argue that ‘drone assemblages’ as a mode of necropolitical violence – the violence of ‘distinguishing whose life is to be managed and those who are subject to the right of death’ (Allinson, 2015: 121; Mbembe, 2003) – is both a form of posthuman embodiment and is itself corporealizing in terms of the racialized and gendered bodies it produces as either killable or manageable. As such, an embodied reading of drone warfare suggests the limits of the ‘god-trick’ of drone warfare both in terms of its omniscient surveillance capacities as well as its global spread. An embodied reading further contributes to critical analyses of targeting practices, such as that of Zehfuss (2011), that undermine claims of the precision and discrimination of such technological practices.