Free beta access. 50 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

AI control of nuclear weapons makes escalation more likely by removing humanity from decisions

Rodriguez, PhD, 19

(Jonathan founded Vergence Labs in 2011, which developed computerized smartglasses. Vergence was acquired by Snap Inc., where he is now manager of new device prototyping. https://www.armscontrol.org/act/2019-04/features/keep-human-control-over-new-weapons)

Michael T. Klare’s article “Autonomous Weapons Systems and the Laws of War” (ACT, March 2019) highlights important issues, but omits enormous strategic risks inhearent to the use of artificial intelligence (AI) in a role of command. In a war between major powers, both sides’ AI systems would be able to outperform humans in the application of game theory, similar to state-of-the-art poker-playing AI, which can now readily defeat the top human players. Nation-states would be tempted to endow these genius-level autonomous killing systems with the authority to escalate, bringing the risk of bad luck causing misunderstandings that lead to uncontrolled escalation. Exacerbating the risk of near-instant autonomous escalation is the risk of an AI system deciding that it, unlike a human, may have nothing to lose. A human commander might be a father or mother who might choose to put the preservation of the human race above nationalistic warmongering. Recall how Soviet air defense officer Stanislav Petrov (a father of two) saved the world in 1983 by choosing to disbelieve a computerized system that claimed a U.S. nuclear first strike was in progress. His decision may have included a desire to put mercy over patriotism and not to participate in the annihilation of the human race. Unfortunately, an AI system would have no such qualms. An AI system might or might not be trained with a self-preservation instinct. If not, it would not care if it is destroyed, and it could take needless risks, carelessly bluffing and escalating, seeking to make the adversary back down through suicidal brinksmanship. One might think, therefore, that it is important to endow battlefield AI systems with a self-preservation instinct, but the consequences of doing so could be even worse. Once a weaponized AI system is given a self-preservation instinct, it may become impossible to shut down, as it would kill anyone who tried to shut it off, including its own country’s military or government leaders. These killings might play out in several ways. On a small scale, the system could limit its “self-defense” killings by targeting only the officials trying to disable the AI system. Worse, the AI system could try to preserve itself by threatening mutually assured destruction: if it ever suspected, even incorrectly, that its shutdown was imminent, the AI system could retaliate with the launch of nuclear weapons. A dying AI system, still energized for a few milliseconds by the last electrical charge remaining in its internal capacitors, could even nuke not just its own nation but the entire surface of Earth.

Sign in to recut, update, or save this card.