Free beta access. 5 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

Autonomous Weapons systems inevitably cause catastrophic accidents and escalation due to the speed of AI decision making

Sharre, 2018 - director of the technology and national security program at the Center for a New American Security [Paul, 12 September, “Ultrafast computing is critical to modern warfare. But it also ensures a lot could go very wrong, very quickly.” https://foreignpolicy.com/2018/09/12/a-million-mistakes-a-second-future-of-war/ ST]

Real-world accidents with existing highly automated weapons point to these dangers. During the initial invasion of Iraq in 2003, the U.S. Army’s Patriot air defense system accidentally shot down two friendly aircraft, killing three allied service members. During the initial invasion of Iraq in 2003, the U.S. Army’s Patriot air defense system accidentally shot down two friendly aircraft, killing three allied service members. The first fratricide was due to a confluence of factors: a known flaw that caused the radar to mischaracterize a descending plane as a missile, outdated equipment, and human error. The second blue-on-blue incident was due to a situation that had never arisen before. In the hectic march to Baghdad, Patriot operators deployed their radars in a nonstandard configuration likely resulting in electromagnetic interference between the radars that caused a “ghost track”—a signal on the radars of a missile that wasn’t there. The missile battery was in automatic mode and fired on the ghost track, and no one overruled it. A U.S. Navy F-18 fighter jet just happened to be in the wrong place at the wrong time. Both incidents were flukes caused by unique circumstances—but also statistically inevitable ones. Coalition aircraft flew 41,000 sorties in the initial phases of the Iraq War, and with more than 60 allied Patriot batteries in the area, there were millions of possible interactions, seriously raising the risk for even low-probability accidents. Richard Danzig, a former U.S. secretary of the Navy, has argued that bureaucracies actually systematically underestimate the risk of accidents posed by their own weapons. It’s also a problem that it’s nearly impossible to fully test a system’s actual performance outside of war. In the Iraq invasion, these accidents had tragic consequences but did not alter the course of the war. Accidents with fully autonomous weapons where humans cannot intervene could have much worse results, causing large-scale fratricide, civilian casualties, or even unintended attacks on adversaries. Attempts at arms control go back to antiquity, from the Bible’s prohibition on wanton environmental destruction in Deuteronomy to the Indian Laws of Manu that forbade barbed, poisoned, or concealed weapons. In the intervening centuries, some efforts to ban or regulate certain weapons have succeeded, such as chemical or biological weapons, blinding lasers, land mines, cluster munitions, using the environment as a weapon, placing weapons in space, or certain delivery mechanisms or deployment postures of nuclear weapons. Many other attempts at arms control have failed, from the papal decrees denouncing the use of the crossbow in the Middle Ages to 20th-century attempts to ban aerial attacks on cities, regulate submarine warfare, or eliminate nuclear weapons. The United Nations began a series of meetings in 2014 to discuss the perils of autonomous weapons. But so far the progress has been far slower than the pace of technological advances. Despite that lack of success, a growing number of voices have begun calling for a ban on autonomous weapons. Since 2013, 76 nongovernmental organizations across 32 countries have joined a global Campaign to Stop Killer Robots. To date, nearly 4,000 artificial intelligence and robotics researchers have signed an open letter calling for a ban. More than 25 national governments have said they endorse a ban, although none of them are major military powers or robotics developers. But such measures only tend to succeed when the weapons in question are of marginal value, are widely seen as especially horrific or destabilizing, are possessed by only a few actors, are clearly distinguished from other weapons, and can be easily inspected to verify disarmament. None of these conditions applies to autonomous weapons. Even if all countries agreed on the need to restrain this class of arms, the fear of what others might be doing and the inability to verify disarmament could still spark an arms race. Less ambitious regulations could fare better, such as a narrow ban on anti-personnel autonomous weapons, a set of rules for interactions between autonomous weapons, or a broad principle of human involvement in lethal force. While such modest efforts might mitigate some risks, however, they would leave countries free to develop many types of autonomous weapons that could still lead to widespread harm. Humanity stands at the threshold of a new era in war, in which machines will make life-or-death decisions at speeds too fast for human comprehension. The risks of such a world are real and profound. Autonomous weapons could lead to accidental death and destruction at catastrophic scales in an instant. The unrestrained pursuit of fully autonomous weapons could lead to a future where humans cede control over what happens on the battlefield, but the critical decisions about how this technology is used still rest in human hands.

Sign in to recut, update, or save this card.