2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Cyber Attacks Advantage › Cyber - Solvency
Haugh, 2018 - project leader for the Institute for Defense Analysis [Brian A., September “The Status of Test, Evaluation, Verification, and Validation (TEV&V) of Autonomous Systems,” , 6/23/22 MD]
7. Exploitable vulnerabilities When systems operate themselves, they can be vulnerable to modes of attack— cyber, electronic, or physical—that would not be as much of a concern for a human-operated system. For example, a cyberattack that compromised the ability of an autonomous unmanned aircraft system (UAS) to recognize other aircraft or physical proximity illusions that repeatedly triggered the UAS’s collision avoidance subroutine might be much more effective than they would be against a human-piloted aircraft. AI based on machine learning has its own set of potential vulnerabilities, both during training of the AI and in operation. TEV&V of autonomous systems will need to be aware of this expanded attack surface. 8. Emergent behavior US DoD Directive 3000.09 specifically warns against the possibility of “unanticipated emergent behavior resulting from the effects of complex operational environments on autonomous or semi-autonomous systems.” Developing T&E methods to analyze the potential for emergent behavior in order to avoid it will be central to providing assured dependability for autonomous systems.
2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Cyber Attacks Advantage › Cyber - Solvency
National Security Commission on Artificial Intelligence 2022 [March 19, “NCSAI Final Report 2021 – Chapter 7: Establishing Justified Confidence in AI Systems”, ]
1. Robust and Reliable AI. Current AI systems, such as those used for perception and classification, have different kinds of failure––characterized as rates of false positives and false negatives. They are often brittle when operating at the edges of their performance competence, and it is difficult to anticipate their competence boundaries.3 They are also vulnerable to attack, and they can exhibit unwanted bias in operation. For national security missions, these can be serious problems. U.S. government agencies should: Focus more federal R&D investments on advancing AI security and robustness. These investments should also advance the interpretability and explainability of AI systems, so users can better understand whether the systems are operating as intended. Consult interdisciplinary groups of experts to conduct risk assessments, improve documentation practices, and build overall system architectures to limit the consequences of system failure.4 Such architectures should securely monitor component performance and handle errors when anomalies are detected5 ; contain AI components that are self-protecting (validating input data) and self-checking (validating data passed to the rest of the system); and include aggressive stress testing.
2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Cyber Attacks Advantage › Cyber - Solvency
Taddeo and Floridi, 2018 - Senior research fellow at the Oxford Internet Institute and Professor of Philosophy and Ethics of Information at the University of Oxford [Mariarosaria, Luciano, Nature.com, “Regulate artificial intelligence to avert cyber arms race,” 6/18/22 MD]
Cyber-attacks are becoming more frequent, sophisticated, and destructive. Each day the US suffers more than 4,000 ransomware attacks that encrypt computer files until the owner pays to release them [1]. This is triple the number in 2015. In May 2017, when WannaCry virus crippled hundreds of IT systems across the UK’s National Health Service, 20,000 appointments were cancelled. A month later, the NotPetya ransomware cost pharmaceutical giant Merck, shipping firm Maersk, and logistics company FedEx around $300 million each in immediate damages. Global damages from cyber-attacks totalled $5 billion in 2017 and may reach $6 trillion a year by 2021. States are partly behind this rise. They use cyber-attacks both offensively and defensively. For example, North Korea has been linked to WannaCry, Russia to NotPetya. As the threats escalate, so do defence strategies. Since 2012, the US has employed ‘active’ cyber defence strategies that enable computer experts to neutralise or distract viruses with decoy targets, and to break back into a hacker’s system to delete data or to destroy it completely. In 2016, the UK announced a £1.9bn investment and a 5-year plan to combat cyber threats. NATO also began drafting principles for such operations, to be agreed by 2019. The US and the UK are leading this initiative, with Denmark, Germany, the Netherlands, Norway, and Spain also involved [2]. Artificial intelligence (AI) is poised to revolutionize this activity. Attacks and responses will become faster, more precise, and more disruptive. Threats will be dealt with in hours, not days or weeks. AI solutions are being rolled out that can verify code and identify bugs and vulnerabilities, making the digital systems they protect more robust. For example, in April 2017 software firm DarkTrace launched Antigena, which uses machine learning to spot abnormal behaviour on an IT network, shut off communications to that part of the system, and issue an alert. Implementations of AI in cyber security 2 will become increasingly pervasive. According to some estimates, the value of AI in cyber security will reach $18.2 billion by 2023 (Figure 1). Experts welcome the potential of AI to improve system verification and robustness [6]. However, this potential is crippled by two risks: the escalation of responses and the lack of control over snowballing effects that AI-led counter-attacks may have. Thus, it is crucial that experts ensure transparency about problems, limitations, and possible shortcomings of unregulated uses of AI for defence, while also working with policy-makers and end users to design adequate testing and oversight mechanisms for this technology. By the end of this decade, many states plan to deploy AI for national cyber defence [4,5]. AI makes deterrence possible [3] because algorithms can identify the source and neutralise it without having to identify the actor behind it, so that attacks can be punished. Currently, countries hesitate to push back because they are unsure who is responsible, given that campaigns may be waged through third-party computers and often use common software. States need regulations that establish legitimate uses of AI for national defence, shape the process leading to the deployment of AI, such as testing and auditing mechanisms, and define the chain of responsibilities [7]. Cyberspace is a domain of warfare and AI is a new defence capability, so regulations are necessary for state use of AI, much like there exist regulations for the other domains of warfare – air, see, land, and space – and the corresponding military capabilities. What is long overdue are criteria for determining proportional responses; clear thresholds, so called ‘red lines’, for distinguishing legal and illegal cyber-attacks; and appropriate sanctionsfor illegal acts[8]. In each case, unilateral approaches will be ineffective. Rather, it is crucial to define an international doctrine for state action in cyberspace. Alarmingly, international efforts to regulate cyber conflicts have stalled. We call on regional forums, like NATO and EU, to revive efforts and prepare the ground for an UN-led initiative.
2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Cyber Attacks Advantage › Cyber - Solvency
Scharre, 2019 - Vice President and Director of Studies at CNAS [Paul, May-June, “Killer Apps: The Real Dangers of an AI Arms Race,” https://omnilogos.com/killer-apps-real-dangers-of-ai-arms-race/6/18/22 MD]
Testing AI systems often takes even more time and money than testing traditional military hardware. Their complexity, which makes them more capable, also creates more opportunities for unexpected glitches. Imagine that a government develops an AI system that can hack into its adversaries' computer networks while avoiding detection. The first government to deploy such a system would gain a huge advantage over its competitors. Worried that an adversary was developing a similar tool, the government might feel compelled to cut testing short and deploy the system early. This dynamic has already played out in other industries, such as self-driving cars. The consequences of accidents caused by AI tools could be far worse. AI wouldn't be the first case of governments relying on a powerful but unsafe technology. That's exactly what happened with computers, which play critical roles in everything from trading stocks to guiding missiles even though they suffer from enormous vulnerabilities. In 2018, investigators at the U.S. Government Accountability Office found that U.S. weapons systems were riddled with cybersecurity loopholes that could be exploited with "relatively simple tools and techniques." Even worse, Defense Department program managers didn't know about the problems and dismissed the GAO'S findings, claiming the tests were not realistic. Computer security vulnerabilities aren't limited to government-run systems. Company after company has suffered major data breaches. Digital security is already too often an afterthought. A world of widespread, unprotected AI systems isn't just a possibility; it's the default setting. SAFETY FIRST Urgent threats require urgent responses. One of the most important ways policymakers can deal with the dangers of AI is to boost funding for AI safety research. Companies are spending billions of dollars finding commercial applications for AI, but the U.S. government can play a valuable role in funding basic AI research, as it has since the field's early days. The AI Next initiative, a program run by the Defense Advanced Research Projects Agency that is set to spend $2 billion over the next five years, is aimed at tackling many of the limitations of narrow AI systems. Expanding on this effort, the White House should increase the funding going to AI safety research as part of its new American AI Initiative, and it should ask Congress for additional money for R & D and safety research. When it comes to applying AI to national security, government agencies will have to reconsider their traditional approaches to testing new systems. Verifying that a system meets its design specifications isn't enough. Testers also need to ensure that it will continue to function properly in the real world when an adversary is trying to defeat it. In some cases, they can use computer simulations to tease out bugs, as manufacturers now do for autonomous cars. On top of that, the Departments of Defense and Homeland Security and the intelligence community should create red teams--groups that act as attackers to test a system's defenses--to ferret out vulnerabilities in AI systems so that developers can fix them before the systems go live.
2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Cyber Attacks Advantage › Cyber – Data Attacks
Goldfarb and Lindsay, 2022 – Chair in AI and health care at the Univ of Toronto, Prof of Cybersecurity at the Georgia Tech [Avi, Jon, 2/25/22, , “Prediction and Judgement: Why Artificial Intelligence Increases the Importance of Humans in War” 6/18/22, LND]
Second, and for the same reasons, adversaries have incentives to complicate both data and judgment. In a highly competitive environment, organizational strengths become attractive targets and potential vulnerabilities. Since predictable adversaries will play to AI strengths, intelligent adversaries will behave unpredictably. If AI creates military power in one area, adversaries will create military challenges in another. Facing an AI-empowered force, the enemy will attempt to change the game by either undermining the quality of predictions or making them irrelevant. Therefore, strategies to contest, manipulate, or disrupt data and judgment become more relevant as military competitors adopt AL The informational and organizational dimensions of war will continue to increase in salience and complexity. Again, this leads us to the conclusion that more military AI will make the human aspects of conflict more important. This increased importance of human personnel challenges the emerging wisdom about AI and war. Many analyses either assume that AI will replace warriors for key military tasks or speculate that war will occur at machine speed, which in turn creates first-mover advantages that incentivize aggression and undermine deterrence. The states that are first to substitute machines for warriors, moreover, are assumed to gain significant military advantages that will shift the balance of power toward early adopters. These outcomes are plausible, but they are based on problematic assumptions about AI substitutability. Conflicts based on AI complementarity may exhibit very different dynamics. We argue that it is more useful to consider the militarized contestation of AI complements (i.e., data and judgment) than to conceive of wars between automated military forces. Conflicts in which data and judgment are perennially at stake may be full of friction, controversy, and unintended consequences, and they may drag on in frustrating ways. In short, we expect the growing salience of data and judgment in war to subtly alter strategic incentives. As a result, AI-enabled conflicts are more likely to be decided by the slow erosion of resolve and institutional capacity than set-piece battles between robotic forces.
2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Cyber Attacks Advantage › Cyber – Data Attacks
Goldfarb and Lindsay, 2022 – Chair in AI and health care at the Univ of Toronto, Prof of Cybersecurity at the Georgia Tech [Avi, Jon, 2/25/22, , “Prediction and Judgement: Why Artificial Intelligence Increases the Importance of Humans in War” 6/18/22, LND]
We adopt standard international relations distinctions between the international system and domestic institutions. The “strategic environment” in refers to the external problems confronting a military organization. To alter or preserve facts on the ground, through conquest or denial, a military needs information about many things, such as the international balance of power, diplomatic alignments and coalitions, geographical terrain and weather, the enemy's operational capabilities and disposition, and interactions with civil society. These external matters constitute threats, targets, opportunities, resources, and constraints for military operations. A military also needs information about internal matters, such as the capabilities and activities of friendly forces, but these are a means to an end. The strategic environment is ultimately what military data are about, and the structure and dynamics of the environment affect the quality of the data. “Institutions and preferences” in refer to the ways in which a military organization solves its strategic problems. This general category encompasses bureaucratic structures and processes, interservice and coalition politics, civil-military relations, interactions with the defense industry, and other domestic politics. Any of these factors might influence the goals and values of a military organization or the way it interprets a given situation. Organizational institutions embody preferences, whatever their source, which in turn affect the quality of judgment. Institutional structures and processes may produce coordination problems, political controversies, or interpretive difficulties that make it hard for a military organization to figure out what matters and why. Furthermore, as discussed below, we expect that the adoption of AI for some military decision tasks will (endogenously) affect the strategic environment and military institutions over time. As data and judgment become more valuable, strategic competitors will have incentives to improve and contest them. We thus expect conflicts over information to become more salient while organizational coordination will become more complex.
2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Cyber Attacks Advantage › Cyber – AI Vulnerable to Attacks
Goldfarb and Lindsay, 2022 – Chair in AI and health care at the Univ of Toronto, Prof of Cybersecurity at the Georgia Tech [Avi, Jon, 2/25/22, , “Prediction and Judgement: Why Artificial Intelligence Increases the Importance of Humans in War” 6/18/22, LND]
If a military organization can figure out how to recruit, train, and retain highly talented personnel, and to thoroughly reorganize and decentralize its C2 institutions, such reforms may help to inculcate and coordinate judgment. Doing so would enable the military to make the most of human-machine teaming in war. If judgment is a source of military strength, however, then it may also be a political vulnerability. As organizational and political judgment becomes the preeminent source of strength for AI-enabled military forces, we expect that judgment will also become the most attractive target for adversaries. If AI relies on federated data and command structures, then adversaries will pursue wedge strategies to break up military coalitions. If the consensus about war aims depends on robust political support, adversaries will conduct disinformation and influence campaigns to generate controversy and undermine popular support. If automated systems operate under tightly controlled rules of engagement, adversaries will attempt to manipulate normative frameworks that legitimize the use of force. If AI enables more efficient targeting, the enemy will present more controversial and morally fraught targets to test political resolve. As prediction machines make some aspects of military operations more certain, we argue that the entire military enterprise will become less certain.
2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Cyber Attacks Advantage › Cyber – AI Vulnerable to Attacks
Artificial Intelligence/Machine Learning Risk & Security Working Group, 2020 [Wharton, ‘Artificial Intelligence Risk & Governance’, ]
Key potential risks of AI relate to potential harms that may affect organizations, consumers, or create broader detrimental effects on society. Such potential risks may arise in whole or in part from sources including the data used to train the AI system; potential risks arising from the AI system itself; potential risks arising from the usage of the AI system; and potential risks arising from poor overall governance of the AI system. 2.1 Risk Categorization Various research papers, articles, and discussions have covered the topics of risks associated with AI. It is up to individual financial services firms to categorize potential risks of using AI: however, we have included a suggested approach to AI risk categorization in the financial services industry below. The areas of data related risks, AI/ML attacks, testing and trust, as well as people risk constitute potential areas of risk, which could be subcategorized as illustrated in Figure 1. We address these sub-categories in further detail below. It’s important to note that the applicability and relevance of risks illustrated in Figure 1 are dependent on an individual organization’s risk profile, appetite, and existing controls, and it is up to each firm to determine whether their existing controls are sufficient. 2.1.1 Inadequate Governance Learning Limitations Unlike humans, AI systems lack the judgment and context for many of the environments in which they are deployed. An AI/ML system is generally as effective as the data used to train it and the various scenarios considered while training the system. In most cases, it is not possible to train the AI system on all possible scenarios and data. Lack of context, judgment, and overall learning limitations may play a key role in informing risk-based reviews, and strategic deployment discussions. Data Quality The risk of poor data quality is not unique to AI, but for AI/ML systems, poor data quality could not only limit the learning capability of the system, but could also potentially negatively impact how it makes inferences and decisions in the future. Poor data quality could include incomplete data, erroneous or unsuitable data, stale data, or data used in the wrong context. Such deficiencies may give rise to potentially erroneous or poor predictions, or potentially result in a failure to achieve the intended objectives. 2.1.2 Potential AI/ML Attacks The proliferation of research papers on AI/ML has increased significantly over the last decade, with many of these devoted to potential security weaknesses in AI/ML systems. Most of the known potential attacks against AI/ML systems could be grouped into one of the following categories: data privacy, data poisoning, and model extraction. The likelihood and impact of various potential attacks are specific to each organization’s risk posture and controls. It is possible that some potential attacks illustrated below may not be relevant to a particular organization and may be mitigated by customary security controls. Data Privacy Attacks In data privacy attacks, an attacker is potentially able to infer the data set used to train the model, thereby potentially compromising the privacy of the data. An adversary could potentially infer sensitive information from the training data set by analyzing the parameters or querying the model. Two major attack types in data privacy include membership inference and model inversion attacks. In a membership inference attack, an attacker could potentially determine if a particular record (or set of records) exists in a training data set. Generally, if the attack is successful, an attacker could determine, to a certain degree of probability, whether a particular record was part of the training data set used to train the AI system. In model inversion attacks, an attacker could potentially extract the training data used to train the model directly. Training Data Poisoning Data poisoning is the contamination of data used to train the AI/ML system, negatively affecting its learning process or output. Data poisoning could potentially be used to increase the error rate of the AI/ML system or to potentially influence the retraining process. Some of the attacks in this category are known as “label-flipping” and “frog-boil” attacks. Adversarial Inputs AI systems that use input from external system(s)/ user(s), interpret the input and perform an action, like classifying the input data. An adversary could potentially use a malicious input or a payload explicitly designed to bypass AI systems classifier. Such malicious inputs are known as adversarial inputs. Model Extraction In this attack, an adversary tries to steal the model itself. Model extraction attacks are potentially the most impactful types of AI/ML attacks, as the stolen model could be used as a ‘tool’ to create additional risks. Research into such attacks indicates that, given unlimited ability to query the model, extraction could occur without requiring high levels of technical sophistication and could be accomplished at high speeds.
2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Cyber Attacks Advantage › Cyber – AI Vulnerable to Attacks
Nurkin and Konaev, 2022 - senior fellows at the Center for Strategy and Security at the Atlantic Council [Tate and Margarita, 5-25-2022, “Eye to eye in AI: Developing artificial intelligence for national security and defense” Atlantic Council ARD]
There are, however, important limitations and drawbacks to AI systems—particularly in operational environments—in large part, because of their brittleness. These systems perform well in stable simulation and training settings, but they can struggle to function reliably or correctly if the data inputs change, or if they encounter uncertain or novel situations. ML systems are also particularly vulnerable to adversarial attacks aimed at the algorithms or data upon which the system relies. Even small changes to data sets or algorithms can cause the system to malfunction, reach wrong conclusions, or fail in other unpredictable ways.20 Another challenge is that AI/ML systems do not typically have the capacity to explain their own reasoning, or the processes by which they reach certain conclusions, provide recommendations, and take action, in a way that is evident or understandable to humans. Explainability—or what some have referred to as interpretability—is critical for building trust in human-AI teams, and is especially important as advances in AI enable greater autonomy in weapons, which raises serious ethical and legal concerns about human control, responsibility, and accountability for decisions related to the use of lethal force. A related set of challenges includes transparency, traceability, and integrity of the data sources, as well as the prevention or detection of adversary attacks on the algorithms of AI-based systems. Having visibility into who trains these systems, what data are used in training, and what goes into an algorithm’s recommendations can mitigate unwanted bias and ensure these systems are used appropriately, responsibly, and ethically. All these challenges are inherently linked to the question of trust explored later in the report.
2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Cyber Attacks Advantage › Cyber – Arms Race
Taddeo and Floridi, 2018 - Senior research fellow at the Oxford Internet Institute and Professor of Philosophy and Ethics of Information at the University of Oxford [Mariarosaria, Luciano, Nature.com, “Regulate artificial intelligence to avert cyber arms race,” 6/18/22 MD]
Meanwhile, research on AI for cyber defence is progressing fast. The US is in the lead, technologically. It aims to incorporate AI into its cyber-defence systems by 2019 [4]. The Department of Defense (DoD) has earmarked $150 million for research. The US Defense Advanced Research Projects Agency (DARPA) is developing techniques to do this. Steps have already been taken. In its 2016 Cyber Grand Challenge competition, seven AI systems, developed by academics and private companies from Greece, Russia, Switzerland, and the US played against each other, identifying and targeting opponents’ weaknesses, while finding and patching their own. The DoD will issue the first US report on AI strategies for national defence in May. There is no indication of what its approach will be. Previous documents, like the 2015 US Cyber Strategy or the 2016 National Cyber Incident Response Plan, did not cover autonomous systems, machine learning, or AI. The 2012 DoD Directive on ‘Autonomy in Weapon Systems’ [9] focused on internal procedures for deploying AI, but was silent on when the US would do so in the international arena. AI is a priority for China, which aims to become a world leader in machine learning technologies. In July 2017, the Chinese government issued its AI Development Plan [5]. Military implementation of AI, on the battlefield as well as in cyberspace, is a crucial part of the strategy. But it is unclear to what degree China plans to deploy AI actively in cyber defence. Russia has not released any public documents about its strategies for AI in defence. However, in a video message released in 2017, President Putin expressly referred to AI and stated “whoever becomes the leader in this sphere will become the ruler of the world”. Experts agree that Russia is focusing on developing AI-enhanced tools for its conventional forces. However, since 2014, the Russian National Defense Control Center (NDCC) has also used machine learning algorithms to detect online threats. Allegedly, Russia has pioneered the use of AI to spread disinformation and intervene in the public debate of other nations, like in the case of the US Presidential election and the UK European Union membership referendum, in 2016. Although these operations are not part of national defence strategies, they speak to the level of AI capabilities developed by Russia over the past two years. North Korea has a history of cyberspace aggression. WannaCry in 2016 and the cyber-attack against Sony Picture in 2014 are two good example of North Korea aggressive attitude in cyberspace. The country lacks technical expertise in AI but is likely to want to catch up with its adversaries.
2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Nuclear AI Advantage › Nuclear – Early Warning
Fitzpatrick, 2019 - former the IISS Non-Proliferation and Nuclear Policy Programme [Mark, 21 May, “Artificial Intelligence and Nuclear Command and Control,” MD]
Skewed early-warning assessments Deep fakes by third parties can be magnified by AI capabilities that fall into the wrong hands and are used to generate false positives. In our scenario, three months before the crisis and start of play, US Cyber Command’s ‘Unified Platform’ for managing and coordinating integrated cyber-, electronic- and information-warfare operations was providing statistically anomalous outputs regarding situational-awareness assessments related to the early warning of selected advanced persistent threat datasets linked to Russian cyber actors. The questionable reports appeared to be skewed by mathematical coefficients derived from large-scale metadata analysis during the beta-testing training phase of the Unified Platform’s software. A workshop background paper noted that AI programmes are driven by the data that they receive – the digital equivalent of the adages ‘you are what you eat’ and ‘garbage in, garbage out’. This is also true of the process by which such programmes initially formulate their pattern-recognition models and evaluative procedures for use in future contexts. The malign manipulation of input data can not only pervert the output of AI functions in specific instances, but also undermine the reliability of an entire algorithm if accomplished during the ‘training’ phase for such programmes. Our scenario showed vividly how this could play out. Non-malign, human-design factors can also corrupt AI assessments. Because human beings define an AI system’s algorithms and curate its training data, they can unintentionally insert their own biases into the system. This can cause the system to behave in unintended ways that may be undetectable to its operators due to the feedback loop created by those very biases. In addition, nuclear command- and-control personnel also faced a black-box problem in determining how or why a system came to a certain conclusion. While system inputs and outputs can be observed, the speed and scale of system processes make it difficult for personnel to isolate the logic behind any particular prediction. Once the operation of AI systems is triggered, humans are unable to monitor the systems’ decision calculus in real time. In the exercise, the US team was aware of this range of potential problems with respect to AI-driven assessments provided by Cyber Command’s Unified Platform. They worried that Russian strategic-warning systems might be similarly skewed. In particular, the Americans realised that the false positives produced by Russia’s strategic-warning system need not have been caused by a malign actor, as they could also have resulted from a problem intrinsic to the algorithm. To preclude Russia from perceiving any US effort to investigate as an attempt to manipulate Russia’s early-warning and command-and-control systems, the US team decided not to conduct any reconnaissance. In turn, the scenario in round two had Kremlin officials learning of a prior intelligence operation by the Main Directorate of the General Staff of the Russian Armed Forces (GRU), Russia’s military-intelligence agency, injecting skewed training data into AI-driven situational-awareness algorithms of the US military during their developmental phase. The objective was to prevent American AI systems from being able to recognise Russia as an aggressor. The background papers for the exercise had noted how data-poisoning attacks introducing skewed inputs into the training dataset for an AI machine-learning process can distort an AI system’s output by degrading its ability to distinguish between good data and bad data. These types of operations are usually carried out through techniques known as content generation, feedback weaponisation, perturbation injection and man-in-the-middle attacks. However, an attacker usually will not have direct access to the actual training data. To overcome this obstacle, the attacker will target the actors or methods used to collect and store machine-learning training data, such as cloud graphics-processing units, web-based repositories, and contractors or third-party service providers. Understanding that early-warning assessments from its own AI-driven software could be skewed, the Russian team chose largely to ignore the AI algorithms that were producing results so disparate from the reality they otherwise perceived, while also deciding not to reveal any analytic problems they were encountering to their adversaries. Unaware that the Russians had, in effect, pulled the plug on the AI-driven early-warning system, the US team spent considerable time trying to persuade their counterparts to investigate its flaws. The US team also considered offering to work with Russia to neutralise any malicious third-party-introduced software problems. They recognised, however, that any such effort might itself be seen as an attempt to compromise the Russian command-and-control system. Indeed, the Russians reacted defensively. With team members playing to stereotype, trilateral meetings and a session of the United Nations Security Council ended inconclusively amid mutual recriminations.
2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Brittle AI Advantage › Brittle - Black Box Scenario
Goldfarb and Lindsay, 2022 – Chair in AI and health care at the Univ of Toronto, Prof of Cybersecurity at the Georgia Tech [Avi, Jon, 2/25/22, , “Prediction and Judgement: Why Artificial Intelligence Increases the Importance of Humans in War” 6/18/22, LND]
The second characteristic of a well-specified AI utility function is that all stakeholders should agree on what goals to pursue. When it is difficult for people to agree on what to optimize, transparent institutional processes for evaluating or aggregating different preferences may help to validate or legitimate decisions that guide AI systems. Unfortunately, consensus becomes elusive as “genius” becomes more geographically distributed, socially collaborative, and technically exacting. In an ethnography of divisional command in Afghanistan, Anthony King writes that “a general must define a mission, manage the tasks of which it is comprised and motivate the troops.” The first of these three factors—specifying positive objectives and negative limitations— is the consummate function of judgment; AI offers little help here. AI might provide some support for the second factor, oversight and administration, which involves a mixture of judgment and prediction. The third factor is leadership, which is fundamentally a judgment problem insofar as leaders attempt to socialize common purposes, values, and interpretations throughout an organization. Again, AI is of little use for issues of leadership, which become more important as organizations become geographically and functionally distributed: “The bureaucratic expertise of the staff has been improved and their cohesiveness has been condensed so that they are now bound in dense solidarity, even when they are not co-present.” Indeed, “decision-making has proliferated” in all three areas—strategy, management, leadership— because a “commander can no longer direct operations alone.” According to King, the commanding general is now less of a central controller and more of a social focal point for coordinating the complex interactions of “the command collective.” Collective command, however, is a collective action problem. In some cases, standard operating procedures and socialization rituals can simplify judgment tasks. King finds that “command teams, command boards, principal planning groups and deputies have appeared to assist and to support the commander and to manage discrete decision cycles to which the commander cannot attend.” Yet, in other cases, personnel from different services, branches, or units may disagree over how to interpret even basic tactics, techniques, and procedures. Disagreement may turn into controversy when mission assignments fall outside the scope of what professionals deem right or appropriate, as when armies are tasked with counterinsurgency, air forces are tasked with close air support, or cultural preferences clash. More serious disagreements about war aims and military methods can emerge within the civil-military chain of command or among coalition partners. Just as data availability and bias vary for any given decision task, we also expect variability in the clarity and consensus of judgment. Any factors that exacerbate confusion or disagreement in military institutions should be expected to make judgment more difficult for AI automation.
2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Brittle AI Advantage › Brittle - Black Box Scenario
Goldfarb and Lindsay, 2022 – Chair in AI and health care at the Univ of Toronto, Prof of Cybersecurity at the Georgia Tech [Avi, Jon, 2/25/22, , “Prediction and Judgement: Why Artificial Intelligence Increases the Importance of Humans in War” 6/18/22, LND]
Numerous empirical studies of civilian AI systems have validated the basic finding that AI depends on quality data and clear judgment. To the extent that decision-making is indeed a universal activity, it is reasonable to expect models of it to apply to militaries. But we caution against generalizing from the business world to military affairs (and vice versa). Commercial and military organizations perform dissimilar tasks in different contexts. Militaries are only infrequently “in business” because wars are rare events. Objectives such as “victory” or “security” are harder to define than “shareholder value” or “profit.” combatants attempt to physically destroy their competitors, and the consequences of failure are potentially existential. One underappreciated reason why AI has been applied successfully in many commercial situations is because the enabling conditions of quality data and clear judgment are often present. Peaceful commerce generally takes place in institutionalized circumstances. Laws, property rights, contract enforcement mechanisms, diversified markets, common expectations, and shared behavioral norms all benefit buyers and sellers. These institutional features make transactions more consistent and efficient. Consistency, in turn, provides the essential scaffolding for full automation. We expect AI to be more successful in more institutionalized circumstances and for more structured tasks. War, by contrast, occurs in a more anarchic environment. In the international system, according to the intellectual tradition of realism, there are no legitimate overarching institutions to adjudicate disputes, enforce international agreements, or constrain behavior. Actors must be prepared to defend themselves or ally with others for protection. Allies and adversaries alike have incentives to misrepresent their capabilities and interests, and for the same reasons to suspect deception by others. Militarized crises and conflicts abound in secrecy and uncertainty. War aims are controversial, almost by definition, and they mobilize the passions of the nation, for better or worse. We expect the absence of constraining institutions in war to undermine the AI-enabling conditions of quality data and clear judgment. One exception that proves the rule is that a military bureaucracy may be able to provide scaffolding for some military tasks. Robust organizational institutions, in other words, might substitute for weak international institutions. Yet, there are limits to what organizations can accomplish in the inherently uncertain and contested environment of war. The specific context of data and judgment will determine the viability of automation for any given task.
2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Brittle AI Advantage › Brittle - Black Box Scenario
Fields et al, 2016 – Chair of the Defense Science Board [Craig, with Dr. Ruth David Dr. Paul Nielsen – co chairs, June “Defense Science Board Summer Study on Autonomy” https://www.hsdl.org/?view&did=794641, LMSi]
Autonomy in support of command and control One of the most contentious applications of autonomy is for command and control in military operations or warfighting, but the potential benefits are real. The time for concepts of operations (CONOPs) development, target selection, and mission assignments can be significantly reduced, and, during combat operations, commanders could be better equipped to respond to changing situations and redirect forces. While commanders understand they could benefit from better, organized, more current, and more accurate information enabled by application of autonomy to warfighting, they also voice significant concerns. Implementation of autonomous capabilities will require significant changes in command and control concepts. A previous DSB study acknowledged the importance of addressing command and control of autonomous systems, but found that it was an unsolved problem and did not address it further.21 Whether mediated by man or machine, all acts, but especially acts related to warfighting, must be executed in accordance with policy and so, in some sense, there is no completely autonomous behavior. Any use of autonomy must conform to a substantive command and control regime laying out objectives, methods and express limitations to ensure that autonomous behavior meets mission objectives while conforming to policy.22 In fact, most autonomous combat systems will and should act under the guidance and instructions of a field commander who will exercise direct oversight. Initial use of autonomous systems for combat will likely assist commanders and their staffs with developing situational awareness and planning missions. The volume and velocity of the data used by the underlying system will change the pace of operations. For example, current air operations planning often involves a several-day process, beginning with identification of objectives; moving to general target selection; to intelligence support identifying particular targets; to determination of final plans by the Commander balancing risks and potential value in achieving objectives; coordination of air, land, and sea assets; and, finally, mission execution and battle damage assessment. Given human limitations, each stage results in static point in time, and planning is generally a linear and timeconsuming process. Because planning often needs to respond to new information, autonomous systems will greatly accelerate the pace of information update and can suggest significant plan changes far more quickly. Commanders will not only need to develop models to calibrate understanding of machine generated information, but will need better automated tools to adjust to the pace of update. This requires careful design of autonomous systems so they can explain and justify recommendations in principled terms that build trust between commanders, staff, and machine-generated output as well as develop a concrete understanding of mission outcomes that depend critically on the data and methodology employed by the autonomous systems. 21 Commanders and their staff need to both exercise oversight of fielded autonomous systems, and have access to efficient mechanisms to develop control datasets that teach autonomous systems so they can react in well-understood ways to unexpected and subtle changes. This requires new approaches to human factors that are informed by warfighting practice and tradition. It also requires development of commander and staff training.
2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Brittle AI Advantage › Brittle – Escalation Impacts
Sharre, 2018 - director of the technology and national security program at the Center for a New American Security [Paul, 12 September, “Ultrafast computing is critical to modern warfare. But it also ensures a lot could go very wrong, very quickly.” ST]
Real-world accidents with existing highly automated weapons point to these dangers. During the initial invasion of Iraq in 2003, the U.S. Army’s Patriot air defense system accidentally shot down two friendly aircraft, killing three allied service members. During the initial invasion of Iraq in 2003, the U.S. Army’s Patriot air defense system accidentally shot down two friendly aircraft, killing three allied service members. The first fratricide was due to a confluence of factors: a known flaw that caused the radar to mischaracterize a descending plane as a missile, outdated equipment, and human error. The second blue-on-blue incident was due to a situation that had never arisen before. In the hectic march to Baghdad, Patriot operators deployed their radars in a nonstandard configuration likely resulting in electromagnetic interference between the radars that caused a “ghost track”—a signal on the radars of a missile that wasn’t there. The missile battery was in automatic mode and fired on the ghost track, and no one overruled it. A U.S. Navy F-18 fighter jet just happened to be in the wrong place at the wrong time. Both incidents were flukes caused by unique circumstances—but also statistically inevitable ones. Coalition aircraft flew 41,000 sorties in the initial phases of the Iraq War, and with more than 60 allied Patriot batteries in the area, there were millions of possible interactions, seriously raising the risk for even low-probability accidents. Richard Danzig, a former U.S. secretary of the Navy, has argued that bureaucracies actually systematically underestimate the risk of accidents posed by their own weapons. It’s also a problem that it’s nearly impossible to fully test a system’s actual performance outside of war. In the Iraq invasion, these accidents had tragic consequences but did not alter the course of the war. Accidents with fully autonomous weapons where humans cannot intervene could have much worse results, causing large-scale fratricide, civilian casualties, or even unintended attacks on adversaries. Attempts at arms control go back to antiquity, from the Bible’s prohibition on wanton environmental destruction in Deuteronomy to the Indian Laws of Manu that forbade barbed, poisoned, or concealed weapons. In the intervening centuries, some efforts to ban or regulate certain weapons have succeeded, such as chemical or biological weapons, blinding lasers, land mines, cluster munitions, using the environment as a weapon, placing weapons in space, or certain delivery mechanisms or deployment postures of nuclear weapons. Many other attempts at arms control have failed, from the papal decrees denouncing the use of the crossbow in the Middle Ages to 20th-century attempts to ban aerial attacks on cities, regulate submarine warfare, or eliminate nuclear weapons. The United Nations began a series of meetings in 2014 to discuss the perils of autonomous weapons. But so far the progress has been far slower than the pace of technological advances. Despite that lack of success, a growing number of voices have begun calling for a ban on autonomous weapons. Since 2013, 76 nongovernmental organizations across 32 countries have joined a global Campaign to Stop Killer Robots. To date, nearly 4,000 artificial intelligence and robotics researchers have signed an open letter calling for a ban. More than 25 national governments have said they endorse a ban, although none of them are major military powers or robotics developers. But such measures only tend to succeed when the weapons in question are of marginal value, are widely seen as especially horrific or destabilizing, are possessed by only a few actors, are clearly distinguished from other weapons, and can be easily inspected to verify disarmament. None of these conditions applies to autonomous weapons. Even if all countries agreed on the need to restrain this class of arms, the fear of what others might be doing and the inability to verify disarmament could still spark an arms race. Less ambitious regulations could fare better, such as a narrow ban on anti-personnel autonomous weapons, a set of rules for interactions between autonomous weapons, or a broad principle of human involvement in lethal force. While such modest efforts might mitigate some risks, however, they would leave countries free to develop many types of autonomous weapons that could still lead to widespread harm. Humanity stands at the threshold of a new era in war, in which machines will make life-or-death decisions at speeds too fast for human comprehension. The risks of such a world are real and profound. Autonomous weapons could lead to accidental death and destruction at catastrophic scales in an instant. The unrestrained pursuit of fully autonomous weapons could lead to a future where humans cede control over what happens on the battlefield, but the critical decisions about how this technology is used still rest in human hands.
2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Brittle AI Advantage › Brittle – Escalation Impacts
Horowitz and Scharre, 2021 - Director of the Emerging Capabilities Policy Office in the Office of the Under Secretary of Defense for Policy and Vice President and Director of Studies at CNAS [Michael, Paul, January 12 2021, “AI and International Stability: Risks and Confidence- Building Measures”, , Acc 6/18/22. M.A.]
Militaries have an inherent interest in staying ahead of their competitors, or at least not falling behind. National militaries want to avoid fielding inferior military capabilities and so will generally pursue emerging technologies that could improve their ability to fight. While the pursuit of new technologies is normal, some technologies raise concerns because of their impact on stability or their potential to shift warfare in a direction that causes net increased harm for all (combatants and/or civilians). For example, around the turn of the 20th century, great powers debated, with mixed results, arms control against a host of industrial era technologies that they feared could alter warfare in profound ways. These included submarines, air-delivered weapons, exploding bullets, and poison gas. After the invention of nuclear weapons, concerns surrounding their potential use dominated the attention of policymakers given the weapons’ sheer destructive potential. Especially after the Cuban Missile Crisis illustrated the very real risk of escalation, the United States and the Soviet Union engaged in arms control on a range of weapons technologies, including strategic missile defense, intermediate-range missiles, space-based weapons of mass destruction (WMDs), biological weapons, and apparent tacit restraint in neutron bombs and anti-satellite weapons. The United States and the Soviet Union also, at times, cooperated to avoid miscalculation and improve stability through measures such as the Open Skies Treaty and the 1972 Incidents at Sea Agreement. It is reasonable and, in fact, vital to examine whether the integration of AI into warfare might also pose risks that policymakers should attend. Some AI researchers themselves have raised alarm at militaries’ adoption of AI and the way it could increase the risk of war and international instability. Understanding risks stemming from military use of AI is complicated, however, by the fact that AI is not a discrete technology like missiles or submarines. As a general-purpose technology, AI has many applications, any of which could, individually, improve or undermine stability in various ways. Militaries are only beginning the process of adopting AI, and in the near term, military AI use is likely to be limited and incremental. Over time, the cognization of warfare through the introduction of artificial intelligence could change warfare in profound ways, just as industrial revolutions in the past shaped warfare. Even if militaries successfully manage safety and security concerns and field AI systems that are robust and secure, properly functioning AI systems could create challenges for international stability. For example, both Chinese and American scholars have hypothesized that the introduction of AI and autonomous systems in combat operations could accelerate the tempo of warfare beyond the pace of human control. Chinese scholars have referred to this concept as a battlefield “singularity,” while some Americans have coined the term “hyperwar” to refer to a similar idea. If warfare evolves to a point where the pace of combat outpaces humans’ ability to keep up, and therefore control over military operations must be handed to machines, it would pose significant risks for international stability, even if the delegation decision seems necessary due to competitive pressure. Humans might lose control over managing escalation, and war termination could be significantly complicated if machines fight at a pace that is faster than humans can respond. In addition, delegation of escalation control to machines could mean that minor tactical missteps or accidents that are part and parcel of military operations in the chaos and fog of war, including fratricide, civilian casualties, and poor military judgment, could spiral out of control and reach catastrophic proportions before humans have time to intervene.
2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Brittle AI Advantage › Brittle – Escalation Impacts
Flournoy, Haines et al 2020 - former Under Secretary of Defense for Policy and Director of National Intelligence [Michèle and Avril, Gabrielle Cheftiz Special Assistant to the Under Secretary of Defense for Policy October, “Building Trust through Testing Adapting DOD’s Test & Evaluation, Validation & Verification (TEVV) Enterprise for Machine Learning Systems, including Deep Learning Systems” https://cset. georgetown.edu/wp-content/uploads/Building-Trust-Through-Testing.pdf Acc 6/23/22 JZ]
ML/DL systems are not robust and it is difficult to characterize system performance. A fundamental challenge of ML/DL is its brittleness; it has trouble functioning correctly if the inputs or environmental conditions change. However, testing these systems in all possible scenarios and with all ranges of inputs is simply not feasible. It is nearly impossible to predict all the ways a system could break, or an adversary could manipulate or spoof it, which is partly why these systems may be especially vulnerable to adversarial attacks. As these systems become more sophisticated, as is the case with deep learning, their output becomes even less transparent, making it harder to determine the conditions under which they might fail and what steps could correct system behavior. Even when operating under the best conditions (within the same distribution of inputs or environmental conditions present during training), DL models generally don’t work to the reliability standards needed for safety-critical systems,10 and therefore may not be appropriate for these applications, at least given the current state of the technology. ML/DL systems are also particularly vulnerable to operational edge cases (both unintended and intended), which are cases that occur beyond the bounds of a system’s operational envelope or normal operating parameters. Because it is very difficult to characterize the actual performance envelop of these systems, it will be important to prioritize stress testing system performance with boundary conditions. Further, interactions within and between systems (including foreign autonomous systems) can induce unintended consequences and are even more complex to predict or understand. The potential for unintended engagement or escalation is even greater when U.S. and/or adversary systems have the sorts of advanced autonomy features that deep learning can enable, and their interaction cannot be studied or fully tested in advance of deployment.
2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Brittle AI Advantage › Brittle – Escalation Impacts
Sharre, 2018 - director of the technology and national security program at the Center for a New American Security [Paul, 12 September, “Ultrafast computing is critical to modern warfare. But it also ensures a lot could go very wrong, very quickly.” ST]
Militaries around the globe are racing to build ever more autonomous drones, missiles, and cyberweapons. Greater autonomy allows for faster reactions on the battlefield, an advantage that is as powerful today as it was 2,500 years ago when Sun Tzu wrote, “Speed is the essence of war.” Today’s intelligent machines can react at superhuman speeds. Modern Chinese military academics have speculated about a coming “battlefield singularity,” in which the pace of combat eclipses human decision-making. The consequences of humans ceding effective control over what happens in war would be profound and the effects potentially catastrophic. While the competitive advantages to be gained from letting machines run the battlefield are clear, the risks would be grave: Accidents could cause conflicts to spiral out of control. Consider what has already happened with stock markets, where computers use algorithms to make decisions so quickly that microseconds make a difference of millions of dollars. Such trading has made brokers huge amounts of money—but has also produced extreme flash crashes that can send markets tumbling in minutes. Regulators have managed these risks by installing circuit breakers that can take a stock offline if the price moves too quickly, but battlefields lack these fail-safes. Flash crashes are bad enough; a flash war would be downright disastrous. Humans have already ceded control to machines in certain military domains. At least 30 countries—with Israel, Russia, and the United States leading the pack—employ human-supervised autonomous weapons to defend bases, vehicles, and ships. These weapons systems, such as the ship-based Aegis combat system, can detect incoming rockets and missiles and, if human supervisors do nothing, respond on their own by firing to eliminate the threat. Such automated responses allow the systems to defend against what are known as saturation attacks, in which salvos of missiles or rockets are launched at a target with such little notice that they could overwhelm human operators. For the time being, autonomous weapons such as these are used purely to protect human-occupied installations or vehicles. Humans supervise the weapons’ operation in real time and can intervene if necessary. Future autonomous weapons could lack these safeguards, however. A number of advanced militaries—including those of China, France, Israel, Russia, the United Kingdom, and the United States—are currently developing stealth combat drones intended to penetrate an adversary’s airspace. Once deep behind enemy lines, these drones might find their communications jammed, so they’re being designed to ensure they can continue to operate on their own. Most countries have not explained how their drones will operate under such circumstances and what rules of engagement they will follow. Countries could require their drones to get human authorization before launching any attacks. Doing so would allow the drones to bomb preapproved fixed targets but would require them to report back and get permission before attacking any newly discovered quarries. Such an approach sounds good in theory, but the problem is that these days many high-priority targets, such as air defense systems and ballistic missile launchers, are highly mobile. This mobility will increasingly tempt military planners to delegate lethal decision-making authority to machines, since doing so could give them an edge in reaction time. No battlefield is static, and the ability to rapidly react to a dynamic environment is critical to mission success—whether in the air, on the ground, or in cyberspace. Air combat strategists call this the OODA (Observe, Orient, Decide, and Act) loop in dogfighting. In the OODA loop paradigm of combat, pilots win dogfights not simply because they enjoy the best hardware but because they assess and react to their situations faster than their opponents, although better sensors and maneuverability might help shorten reaction times. Since machines can react faster than humans, automation will offer tremendous advantages in this competition. That means that the same competitive pressures that led to the creation of systems such as the Aegis could soon be introduced on a wider scale. A military that fully integrated its autonomous systems could always stay one step ahead of its enemy in combat and present a constantly shifting threat. As Gen. Paul Selva, the vice chairman of the U.S. Joint Chiefs of Staff, told the Senate Armed Services Committee in July 2017, “It is very compelling when one looks at the capabilities that artificial intelligence can bring to the speed and accuracy of command and control and the capabilities that advanced robotics might bring to a complex battlespace, particularly machine-to-machine interaction in space and cyberspace, where speed is of the essence.” Yet speed is not an unadulterated good. Forces that react to the enemy so quickly that their own commander does not understand what is happening could risk a breakdown in command and control, a problem that military leaders have struggled with for millennia. Today, email and chat messaging have replaced horses and flags, but the fundamental problem persists. Militaries counter this inherent friction between orders from above and the reality on the ground with a concept known as commander’s intent: succinct goal-oriented statements issued to subordinates that explain the desired goal of a particular mission, and thus ensure that they stick to the general plan, but also allow them the flexibility to adapt to events on the ground. Such statements prevent forces from becoming too predictable and give subordinates the freedom to overcome obstacles in novel ways.
2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Brittle AI Advantage › Brittle - Automation Bias
Horowitz, 2019 - Professor of Political Science, University of Pennsylvania [Michael C. May 2“When Speed Kills: Autonomous Weapon Systems, Deterrence, and Stability” https://ssrn.com/abstract=3348356]
Autonomous Weapon System Deployment The history of cruise and ballistic missile development – and specifically the desire for faster and more accurate missiles – illustrates why countries might seek to accelerate the development and deployment of LAWS. In the Cold War, for example, increased accuracy was mostly viewed as a good in itself, or a natural development.27 The promise of being able to operate faster than adversaries on the battlefield, or get inside their decision loops, are also constant goals of planners, as is the ability to use force more precisely. In combination, at the outset, these autonomous characteristics of weapon systems should make their development and deployment more likely. A key factor that will influence the deployment of LAWS is trust in the system. As referenced above, the automatic mode on the Phalanx and related CIWS systems provides an approximation of a lethal autonomous weapon system. Moreover, Iron Dome (Israel) and the PAC- 3 Patriot (US and others) integrate autonomy into targeting and firing decisions. A track record of effectiveness can lead operators to trust them too much through a process called automation bias, as they outsource judgment to algorithms.28 For example, in 2003, the Patriot missile system led to two fratricides in the US-led invasion of Iraq. In both cases, the autonomous characteristics of the system proved too brittle for the real ambiguities of combat, and operators trusted the sensors too much.29
2022
MNDI
Affirmative
TEVV AFF – MNDI HPA › Brittle AI Advantage › Brittle - AI Accidents Inevitable
Scharre, 2016 -- Vice President and Director of Studies at CNAS [Paul, Feb 2016, Center for New American Security, “Autonomous weapons and operational risk”, https://s3.us-east-1.amazonaws.com/files.cnas.org/documents/CNAS_Autonomous-weapons-operational-risk.pdf?mtime=20160906080515&focal=none, Acc 6/21/22, M. A.]
Neural networks and the black box The challenge of complexity, while problematic for complex rule-based systems, is even more difficult for cutting-edge artificial intelligence (AI) systems that employ neural networks. Neural networks do not perform rule-based calculations like most computers. Instead, they learn by exposure to large data sets. As a result, the internal structure of the network that generates output can be opaque to the designers—a “black box.” Even more unsettling, for reasons that may not be entirely clear to AI researchers, the neural network sometimes can yield odd, counterintuitive results. A study of visual classification AIs using neural networks found that while the AIs were able to generally identify objects as well as humans, in some cases the AIs made confident identifications of objects that were not only incorrect, but that looked vastly different from the purported object to human eyes. The AIs interpreted images that to the human eye looked like static or abstract wavy lines as animals or other objects, and asserted greater than 99.6% confidence in their estimation.