Free beta access. 50 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

A lack of human control increases crisis instability and escalation risks – recent wargaming simulations prove that the threat is already here.

Sauer, 2021 - Senior Research Fellow at Bundeswehr University [Frank serves on the International Panel on the Regulation of Autonomous Weapons IRRC No. 913 March “Stepping back from the brink: Why multilateral regulation of autonomy in weapons systems is difficult, yet imperative and feasible” https://international-review.icrc.org/articles/stepping-back-from-brink-regulation-of-autonomous-weapons-systems-913 Acc 4/5/22 TA]

Escalation risks and crisis instability Weapons systems operating without human control generate not only new vulnerabilities but also unpredictability due to unforeseeable interactions with their environment, in turn creating new risks of unintended, unwanted escalation.66 In that regard, the interaction between two or more autonomous systems is to be considered in particular. High-frequency trading67 provides a useful analogue, because unforeseen and unwanted interaction processes between two or more autonomously operating trading algorithms occur on a regular basis, sometimes causing so-called “flash crashes” and resulting in financial losses. This can be remedied with regulation of the financial market to an extent, but without internationally binding regulation of autonomy on the battlefield, unforeseeable interactions of LAWS might end in unintended use of force at machine speed, even accidental war before humans can intervene.68 This risk is not in some distant future. At the Dubai Airshow in 2019, the chief of staff of the US Air Force, General David Goldfein, presented the simulated engagement of an enemy navy vessel with a next-to-fully automated kill chain. The vessel was first picked up by a satellite, then target data was relayed to airborne surveillance as well as command and control assets. A US Navy destroyer was then tasked with firing a missile, the only remaining point at which this targeting cycle involved a human decision, with the rest of the “kill chain … completed machine to machine, at the speed of light”.69 Any machine error in such a system would, if left uncorrected by a human due to automation bias, propagate quickly. It stands to reason that the error would propagate “at the speed of light” as well, were the human to be removed. A recent wargaming exercise conducted by the RAND Corporation underlines the risks of crisis instability and unintended escalation; in this exercise, simulated forces were set “on ‘full auto’ to signal resolve …[,] in one case lead[ing] to inadvertent escalation. Systems set to autonomous mode reacted with force to an unanticipated situation in which the humans did not intend to use force.”70

A lack of human control threatens to escalate crises into wars due to the speed of conflicts, the gap between human and AI solutions, and the pressure for pre-delegation

Vestner, 2021 - Head of Security and Law Programme at Geneva Centre for Security Policy [Tobias, July 8 “Warfare and Artificial Intelligence” in Robin Geiß and Henning Lahmann (eds), Research Handbook on Warfare and Artificial Intelligence -forthcoming https://www.gcsp.ch/publications/military-operations-and-artificial-intelligenceGCSP Acc 5/27/22 TA]

Most importantly for military strategy, AI applications may assist decision-makers to monitor the battlefield and develop scenarios. Indeed, AI could be developed to predict the behaviour and reactions of foreign countries or generate simulations of the progression of ongoing conflicts.32 AI may also be useful to assess threats, provide risk analyses, and suggest courses of action, ultimately guiding decision-makers on the best response to take.33 In addition, AI may support the alignment of the armed forces’ ways and means with the given political and strategic objectives - a major function of military strategy. A consequence of such developments would be an increased speed and quality of military processes. While this would provide significant advantages to those states with the most performant AI,34 this may also pressure armed forces to increasingly delegate the orchestration of military operations to AI systems.35 Indeed, the use of AI for military strategy may also lead to challenges. Reliable AI systems would need to be trained with vast data sets.36 Furthermore, it has been warned that AI may exacerbate threats, transform their nature and characteristics, and introduce new security threats.37 A tabletop exercise on the integration of AI into nuclear C2 systems showed that such systems were ‘vulnerable to malicious manipulation that can severely degrade strategic stability’,38 for instance. Such vulnerabilities would derive mostly from the risk posed by third actors using techniques to deceive, disrupt or impair C2 systems,39 which indicates the importance of system safety for AI to be used for military strategy. Another significant challenge is that AI may accelerate the speed of warfare to the extent that humans cease to be able to follow the developments, ultimately leading humans to lose control. 40 This phenomenon has been termed battlefield ‘singularity’ or ‘hyperwar’.41 This may lead to strategic errors and accidents, including involuntary conflict escalation. Even if such risks can be alleviated, the increased reliance on AI would reduce the human element of military strategy, in particular psychology and human judgment. It has been argued that this could lead to a ‘gap between how the AI solves a problem framed by humans, and how those humans would solve it if they possessed the AI’s speed, precision, and brainpower’.42 Yet it has also been argued that strategy development would require the understanding of values, the balance of costs, and the understanding of the complex social system in which war operates, thereby significantly limiting AI’s use for military strategy.43 Yet it is also possible that when enemies possess high levels of rational prediction power provided by AI systems, the decisive factor in warfare will not be the AI systems’ capabilities but the human judgment, in particular concerning critical and difficult choices.44 This, however, presumes a certain level of meaningful human involvement. In sum, AI may enhance military strategy development and strategic decision-making, notably if able to process more data and make sense of complexity with more precision and at a higher speed than humans and simple computing. A likely result is an acceleration of military operations, which may increase pressure on armed forces to integrate AI and may marginalize human judgment. States’ recent adoption of defence strategies on and related to AI indicate that states increasingly intend to develop, acquire, and operationalize AI for military purposes. As such, the possession and use of AI is a strategic objective itself. In light of secrecy around the development of new technologies, states’ investment in military AI can become a strategic liability as it may increase the risk of destabilizing arms races, misperceptions, and miscalculations.

Crisis instability risks nuclear escalation – the lack of human control threatens the firebreak between conventional and nuclear conflict.

Sauer, 2021 - Senior Research Fellow at Bundeswehr University [Frank serves on the International Panel on the Regulation of Autonomous Weapons IRRC No. 913 March “Stepping back from the brink: Why multilateral regulation of autonomy in weapons systems is difficult, yet imperative and feasible” https://international-review.icrc.org/articles/stepping-back-from-brink-regulation-of-autonomous-weapons-systems-913 Acc 4/5/22 TA]

Strategic instability It has recently been suggested that AI as a decision-making aid to humans might help improve the performance of nuclear early-warning and command and control systems, thus reducing the risk of false alarms and inadvertent nuclear use.72 That said, calls for complete automation in the nuclear realm – that is, for handing over the decision to use nuclear weapons from humans to machines – are practically non-existent.73 But even with the proverbial push of the button not yet delegated to algorithms, the rush to increase autonomy in military applications and to automatize military processes increases the risk of nuclear stability.74 For instance, the increasing capacities of conventional weapons systems – including weapon autonomy – are beginning to affect the strategic level. This development has been described as the increasing “entanglement” of the nuclear and the conventional realm resulting, for example, from “non-nuclear threats to nuclear weapons and their associated command, control, communication, and information (C3I) systems”.75 Simply put, advanced conventional capabilities increasingly allow for nuclear assets to be put at risk. Autonomy in conventional weapons systems is one such advanced capability, thus feeding into this increasing entanglement and, in turn, deteriorating strategic stability. One specific illustration of this dynamic is the deployment of stealthy unmanned aerial vehicles and the use of “swarming”. Perdix is a swarming test program pursued by the US Air Force. In the future, drone swarms of this type might facilitate the search for dispersed mobile missile launchers. Another example is the use of maritime autonomous systems for hunting nuclear-powered ballistic missile submarines, known as SSBNs. The DARPA-funded Anti-Submarine Warfare Continuous Trail Unmanned Vessel is a program that resulted in the development of an autonomous trimaran called Sea Hunter, which is currently being tested by the US Navy. Its ability to detect and pursue SSBNs could potentially limit the second-strike capabilities of other nuclear powers. These capabilities are just emerging, and neither Perdix nor Sea Hunter, nor their successors, will single-handedly destabilize the global nuclear order. Also, the hypothesis that systems such as Sea Hunter would render the oceans “transparent”,76 virtually nullifying the utility of sea-launched nuclear weapons as a reliable second-strike asset, is hotly debated. Nevertheless, the mere perception of nuclear capabilities becoming susceptible to new risks from the conventional realm is bound to sow distrust between nuclear-armed adversaries. Furthermore, a system like Sea Hunter demonstrates how autonomous weapon technologies are expediting the completion of the targeting cycle, thus putting the adversary under additional pressure and potentially creating “use-them-or-lose-them” scenarios with regard to executing a nuclear second strike. The entanglement problem, which weapon autonomy is feeding into, is further aggravated by an increasing political willingness to use nuclear means to retaliate against non-nuclear attacks on early-warning and control systems or the weapons themselves. The Trump administration's nuclear posture review77 signals that the United States may, from now on, respond with nuclear means to significant, non-nuclear strategic attacks (moving away from a “single-purpose” nuclear deterrence framing for nuclear weapons). Russia has already held this position for some time due to the United States’ advantage in conventional weapons technology. This does not bode well for stability between the two largest nuclear powers. To sum up this section, weapon autonomy not only promises military benefits but also creates new vulnerabilities and, more importantly, contributes to an overall accumulation of strategic risk and instability. Increasing operational speed beyond the capability of human cognition removes humans as a valuable fail-safe against unwanted escalation.

Autonomous AI undermines NATO’s collective defense by eroding nuclear deterrence

Gilli, 2020 - Senior Researcher at the NATO Defense College [Andrea, NDC Research Paper No.15 – December ““NATO-Mation”: Strategies for Leading in the Age of Artificial Intelligence” https://www.ndc.nato.int/news/news.php?icode=1514 Acc 4/21/22 TA]

Arms control, technology regimes and confidence building NATO’s core task of collective defence has been at the heart of the Atlantic Alliance since its inception. In strategic terms, collective defence translates into deterrence and is in fact fundamental to ensure the security, wellbeing and freedom of Allies and their citizens. Collective defence is, however, just one side of the coin. Historically, NATO Allies have also pursued arms control and disarmament. Recent statements on the role of nuclear weapons bear testament to this: while NATO wishes for a world free of nuclear weapons, nuclear deterrence remains a pillar of the Alliance with a view to preventing intimidation, coercion or attack from outside powers.263 The emergence of AI, ML and BD has generated an important debate on nuclear stability.264 Many worry that, when integrated into strategic forces, AI may bring about a set of nefarious consequences: the brittleness characterizing algorithms may lead to inaccuracy, and thus to crises;265 machine speed may lead to the removal of humans from the loop, thus tipping crises into escalations;266 last but not least, some still emerging technologies such as AI may undermine the survivability of second-strike nuclear capabilities, thus eroding the foundations of nuclear deterrence.267

Sign in to recut, update, or save this card.