Free beta access. 5 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

Premature deployment of AI weapons risks miscalculation and war - they will fail when they operate outside of their prescribed environments.

Goldfarb and Lindsay, 2022 – Chair in AI and health care at the Univ of Toronto, Prof of Cybersecurity at the Georgia Tech [Avi, Jon, 2/25/22, https://doi.org/10.1162/isec_a_00425, “Prediction and Judgement: Why Artificial Intelligence Increases the Importance of Humans in War” 6/18/22, LND]

In contrast with chess, the risks of premature automation are more extreme in the military realm (e.g., fratricide and civilian casualties), but the logic is the same. Militaries abound with standard operating procedures and tactical doctrines that guide the use of lethal capabilities (e.g., instructions for the safe operation of weapon platforms, playbooks for tactical maneuvers, and policies for employing weapons). To the degree that goals and mission parameters can be clearly specified, tactical operations will appear to be attractive candidates for automation. To the degree that combat timelines are expected to be extremely compressed, moreover, automation may appear to be even more urgent. Rapid decision-making would necessitate the pre-specification of goals and payoffs and the coupling of AI prediction to robotic action. Lethal autonomous weapon systems use prediction to navigate complex environments in order to arrive at destinations or follow targets, within constraints that are supplied by human operators. Their targeting systems base their predictions on training data that identify valid targets. Using algorithms, machines may rapidly and accurately identify targets at far greater distances than human visual recognition, and algorithmic target recognition may be collocated with sensors to reduce response times. Many AI-enabled weapons already or imminently exist. The Israeli Harpy loitering munition can search for and automatically engage targets, and China has plans for similar “intelligentized” cruise missiles. Russia is developing a variety of armed, unmanned vehicles capable of autonomous fire or unarmed mine clearance. The United States has been exploring combat applications for AI in all warfighting domains. In the air, the “Loyal Wingman” program pairs an unmanned F-16 with a manned F-35 or F-22 to explore the feasibility of using humans to direct autonomous aircraft, such as the XQ-58A Valkyrie. Air combat algorithms that can process sensor data and plan effective combat maneuvers in the span of milliseconds have already defeated human pilots in some simulators. At sea, the U.S. Navy's LOCUST project explores the feasibility of launching swarms of expendable surface-to-air drones. The Defense Advanced Research Projects Agency's (DARPA) Continuous Trail Unmanned Vessel program is designed to search for enemy missile submarines and automatically trail them for months at a time, reporting regularly on their locations. On land, U.S. Marine “warbot companies” equipped with networks of small robots might provide distributed sensing and precision fire. Automated counter-battery responses, which accurately retaliate against the origin of an attack, could give human commanders leeway to focus on second- and third-order decisions in the wake of an attack. In the cyber domain, AI systems might autonomously learn from and counter cyberattacks as they evolve in real time, as suggested by the performance of the Mayhem system in DARPA's 2016 Cyber Grand Challenge. AI could be especially useful for detecting new signals in the electromagnetic spectrum and reconfiguring electronic warfare systems to exploit or counter them. Space satellites, meanwhile, have been automated from their inception, and space operations might further leverage AI to enhance surveillance and control. Much of the AI security literature is preoccupied with the risks posed by automated weapons to strategic stability and human security. Risks of miscalculation will increase as the operational context deviates from the training data set in important or subtle ways. The risk of deviation increases with the complexity and competitiveness of the strategic environment, while the costs of miscalculation increase with the lethality of automated action. The machine tries to optimize a specific goal, but in the wrong context, doing so can lead to false positives. AI weapons may inadvertently either target innocent civilians or friendly forces or trigger hostile retaliation. In these cases, the AI would have the authority to kill but would not understand the ramifications. The risks are particularly stark in the nuclear arena. Nuclear war is the rarest of rare events—keeping it that way is the whole point of nuclear deterrence—so training data for AI systems is either nonexistent or synthetic (i.e., based on simulation). Any tendency for AI systems to misperceive or miscalculate when confronted with uncertain or novel situations could have catastrophic consequences. In short, autonomous weapon systems that combine prediction with action can quickly make tragic mistakes.

Premature deployment of AI increases the risk of crisis escalation due to inflexibility and automation bias

Horowitz and Kahn, 2021 – Senior and Research Fellows at the Council on Foreign Relations [Michael and Lauren, The Washington Quarterly 3-19-2021 “Leading in Artificial Intelligence through Confidence Building Measures” TM]

The Risk of Inadvertent Escalation Third, either insufficient training and education or automation bias could make inappropriate uses of AI-enabled systems by leaders or operators that escalate tense situations more likely.13 The risk of deploying complex algorithms could be magnified when algorithmic systems from different sides interact in a crisis. When tensions are on a hair-trigger, human decision-makers generally look for off-roads from war, which is why accidental war and inadvertent escalation are quite rare despite fears of spiraling conflicts.14 Algorithmic proxies will almost inherently have less flexibility than human commanders and could interpret behavior as more threatening and/or not have the capacity to avoid war in the same way as humans, making inadvertent escalation more likely. Even in the case of human engagement with algorithms, avoiding the potential costs of constrained parameters could be tricky. If not trained properly, operators using AI-based aids can experience cognitive offloading and become over-reliant on the AI and less willing to intervene or deviate when presented with a suggestion or action from the AI—a phenomenon known as automation bias—especially in uncertain, unfamiliar, or time-critical situations, making accidents that could increase the risk of inadvertent escalation.15 Combined, certain characteristics of AI, such as the opacity of algorithms and potential speed of operations, along with procedural difficulties like TEVV for algorithms and training operators to use systems and software correctly, especially in the context of distrust due to great power competition, can increase the likelihood of dangerous unforeseen and undesirable outcomes such as accidents, unintended conflict, and inadvertent escalation.

Sign in to recut, update, or save this card.