Free beta access. 5 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

Failures in AI inevitable – inflexibility in unpredictable situations causes mistakes

Scharre, 2016 -- Vice President and Director of Studies at CNAS [Paul, Feb 2016, Center for New American Security, “Autonomous weapons and operational risk”, https://s3.us-east-1.amazonaws.com/files.cnas.org/documents/CNAS_Autonomous-weapons-operational-risk.pdf?mtime=20160906080515&focal=none, Acc 6/21/22, M. A.]

Autonomous systems lack the flexibility humans have to adapt to novel circumstances. This may mean that in unexpected situations, autonomous systems make mistakes that humans would not have. As militaries think through how they incorporate autonomy into future weapons, a clear-eyed assessment of the risks associated with autonomous weapons is required. Accidents with autonomous weapons could lead to civilian casualties, fratricide, or unintended escalation in a crisis. Understanding these risks is important for policymakers and acquisition professionals weighing whether to build autonomous weapons, as well as for military commanders who are responsible for the weapons they deploy on the battlefield. Of course, humans are far from perfect in war. They make mistakes that lead to fratricide and civilian casualties as well, and humans also commit deliberate acts of atrocity.5 There are important qualitative differences between the risks associated with semi-autonomous (human in the loop) weapons and autonomous weapons, however: • Without a human in the loop to act as a fail-safe, the consequences of failure with an autonomous weapon could be far more severe than an equivalent semi-autonomous weapon. • For extremely simple autonomous weapons, these risks may be manageable if human operators employ them only in limited, controlled contexts. As autonomous weapons increase in complexity, however, it may be more difficult for human operators to fully understand the boundaries of their behavior and accurately predict under what conditions failures might occur, even if they are unlikely. • While improved test and evaluation can mitigate these risks somewhat, they cannot be eliminated entirely. Failures in complex systems may not be likely, but over a long enough time horizon they are inevitable.

Military AI systems are prone to technical failures which cause unpredictable escalation risks.

International Panel on the Regulation of Autonomous Weapons, 2021 [(iPRAW) coordinated by: German Institute for International and Security Affairs, July “Building Blocks for a Regulation on LAWS and Human Control Updated Recommendations to the GGE on LAWS” https://www.readkong.com/page/building-blocks-for-a-regulation-on-laws-and-human-control-8617434 Acc 2/27/22 TA]

Autonomous functions in weapon systems are based on various enabling technologies including sensors, processors, and software. Most prominent are data-driven techniques, like AI and machine learning. Even though a regulation of LAWS should focus on the human role, understanding the options and limitations of enabling technologies is crucial. By integrating, processing, and analyzing large amounts of data quickly, AI-enabled technologies can offer useful decision support and might furthermore allow for new operational options. At the same time, however, due to their brittleness and opaque nature, they could increase unpredictability and escalation risks and perpetuate biases. In some instances, the application of computational methods in the targeting process can lead to better outcomes than human performance alone would. Nevertheless, our discussions and scenario-based workshops showed the limitations of computational methods as enabling technologies in the military domain and highlighted that they most likely cannot replace the unique judgment of human decision-makers, understood as “the ability to evaluate and combine numerous contextual sources of information” . Furthermore, any complex computational system consists of modular subsystems, each of which has inherent limitations and points of failure. Applying multiple computational systems across each step of the targeting cycle may result in cumulative failures that can be hard to anticipate and lead to hazardous and undesired outcomes. Any system that executes sequential processes, such as selecting and engaging targets, can be subject to path dependencies where errors or decisions, in any step, can propagate and reverberate throughout the rest of the targeting process.

Autonomous swarm technology is excessively prone to malfunction due to complexity and hacking

Sauer, 2021 - Senior Research Fellow at Bundeswehr University [Frank serves on the International Panel on the Regulation of Autonomous Weapons IRRC No. 913 March “Stepping back from the brink: Why multilateral regulation of autonomy in weapons systems is difficult, yet imperative and feasible” https://international-review.icrc.org/articles/stepping-back-from-brink-regulation-of-autonomous-weapons-systems-913 Acc 4/5/22 TA]

New operational vulnerabilities The flip side of the force multiplication effect that militaries hope for with this diffusion-prone technology is scalability, creating the potential for weaker parties to change the power dynamics between themselves and their adversaries. The weaponization of simple, commercially available drones by the so-called Islamic State group and the attack against the Saudi Aramco oil facility give non-autonomous foretastes of what is to come and demonstrate that, advanced aerial defence capabilities notwithstanding, new vulnerabilities are on the rise. Particularly from the point of view of US ground forces, having to face serious threats from above after decades of air dominance represents a paradigm shift.47 The United States is thus already being forced to rethink its air defence capabilities by intensifying the development of lasers and microwaves. Conventional solutions, such as Stinger missiles, are not only unsuitable for defence against the swarms of small, cheap, disposable drones that autonomy now renders possible, they are also not cost-effective. Whether the new defensive systems can remedy this situation is still open for debate.48 Suffice it to say that the combination of cheap unmanned systems, autonomy and swarm behaviour creates new risks in general, for troops on the battlefield, for command and control infrastructure and for senior leaders in so-called decapitation scenarios.49 As argued above, the possible elimination of the remote control link is a key incentive for having more autonomy in a weapons system – but handing control over to the machine opens up new attack vectors as well. Feeding the system spoofed GPS data is one example; in 2011, Iran was seemingly able to hijack an autonomously navigating US drone in this manner.50 What is more, systems relying on machine learning that makes use of deep neural networks,51 which currently represent the state of the art in the field of computer vision, are also particularly susceptible to manipulation. Some reflective tape on a stop sign, for example, can fool a self-driving car's image recognition system. This susceptibility to error is a tricky but eventually solvable problem in a civilian application such as self-driving cars. Training data is plentiful and easily available, and self-driving cars are designed to operate cooperatively in a tightly regulated environment. The battlefield presents itself very differently – it is characterized by a paucity of data and much greater degrees of unpredictability and vulnerability.52 After all, an adversary will of course always try to deceive and tamper with your systems. Research on adversarial examples53 suggests that computer vision will leave autonomous weapons systems open to manipulation by tampering with the environment that the machines perceive54 or even by retraining them if they continue learning during their deployment.55 Facial recognition for targeting purposes would be quite easy to fool and defeat, too, as the rapid development of countermeasures against domestic surveillance demonstrates.56 As the complexity of the software driving a weapons system increases, so does the number of bugs it contains. Such programming errors can have critical effects, including friendly fire.57 Normal accidents theory58 suggests that mistakes are basically inevitable. They occur even in domains with extremely high safety and security standards, such as nuclear power plants or manned space travel.59 The software industry can currently reduce the number of bugs to 0.1–0.5 errors per 1,000 lines of code, which means that complex military systems with several million lines of code, such as the software for the F-35 fighter jet, come with thousands of software errors.60 The unavoidable reality of regularly having to update the systems complicates this issue further;61 it is a potential source of new bugs and new errors arising from interactions between newer and older software. Machine learning systems generate specific difficulties because they present themselves as “black boxes” which cannot be debugged the way conventional software can, meaning that they cannot be selectively cleared of specific errors.62 Finally, weapon autonomy evokes a new proneness to errors in regard to any remaining interactions with human operators. Here, automation bias comes into play – that is, the uncritical, unfounded trust in the functioning of a system.63 Put simply, autonomous systems might operate incorrectly over periods of time without anyone noticing.64 A human making a mistake can understand the situation and correct for it, but unmonitored LAWS will not be able to understand and critically reflect in real time the way humans do.65 This gives rise to risks of unintended military escalation.

Sign in to recut, update, or save this card.