Autonomous nuclear command-and-control makes miscalc inevitable---false alarms and automation bias
Horowitz 19 [Michael C. Horowitz (United States) is professor of political science at the University of Pennsylvania and the associate director of its Perry World House. May 2019 “Artificial intelligence and nuclear stability” THE IMPACT OF ARTIFICIAL INTELLIGENCE ON STRATEGIC STABILITY AND NUCLEAR RISK Volume I Euro-Atlantic Perspectives pp. 80-81 https://www.sipri.org/sites/default/files/2019-05/sipri1905-ai-strategic-stability-nuclear-risk.pdf] -os-
I. AI and nuclear command and control Excluding a first strike, the first step in the process leading up to the possible use of nuclear weapons is how a nuclear-armed state attempts to detect whether another country is launching nuclear weapons and how it responds. Many countries already automate parts of their nuclear weapon infrastructure, especially advanced nuclear powers such as the United States.3 This includes early warning, command and control, and missile targeting. Advances in AI could lead to the expansion of the use of autonomous systems in command and control. For example, states could decide to automate additional components of early warning because autonomous systems can detect patterns and changes in patterns faster than humans. This could have potential benefits for nuclear security and stability, because well-functioning algorithms could give decision makers more time in a complex environment. Moreover, autonomous systems could represent another form of redundancy that helps to ensure the dissemination of launch orders in the worst case. However, the 1983 Petrov incident illustrates a clear downside to fully automating command and control. In this case, the Soviet Oko satellite-based early-warning system reported a false alarm—the launch of five US intercontinental ballistic missiles (ICBMs). No missiles had been launched. Lieutenant Colonel Stanislav Petrov was the watch officer on duty. It was his job to alert Soviet leadership of a US attack. While the automated systems reported the ‘highest’ confidence that a missile strike was occurring, Petrov stated that he ‘had a funny feeling in [his] gut’. He instead reported a system malfunction, rather than a nuclear strike.4 The risk is that a future incident could lead to escalation, instead of a malfunction report, for two reasons. First, a decision to fully automate early warning would mean that there was no human operator—no Petrov—to prevent a false alarm from escalating. To be fair, however, it seems unlikely that a country would cut humans entirely out of the early-warning process. Second, automation bias could mean that a future Petrov trusts the algorithm and instead reports that an attack is under way.5 While also unlikely, academic research on automation bias suggests that this is a real risk.6