Adversaries will spoof military A.I., undermining effectiveness and triggering premature escalation.
Johnson & Krabill 20, *Postdoctoral Research Fellow at the James Martin Center for Nonproliferation Studies at the Middlebury Institute of International Studies (MIIS), Monterey. **Master of Arts in Nonproliferation and Terrorism candidate at Middlebury Institute of International Studies (MIIS), Monterey. (James, Eleanor, 1-31-2020, “AI, Cyberspace, and Nuclear Weapons”, War on the Rocks, https://warontherocks.com/2020/01/ai-cyberspace-and-nuclear-weapons/)
In this example, the spoofing attack on the weapon systems’ algorithm is executed in such a way that the imagery appears to the recognition system as indistinguishable from a valid military target, escalating the situation based on a false premise that would be unlikely to fool the human eye. AI experts have proven that even when data appears accurate to AI image recognition software, these systems often hallucinate objects that do not exist.
The explainability (or “black box”) problem associated with AI applications may further compound these dynamics. Insufficient understanding of how and why AI algorithms reach a particular judgment or decision would complicate the task of determining if datasets had been deliberately compromised to manufacture false outcomes — such as attacking incorrect targets or misdirecting allies during combat.
Furthermore, as humans and AI team up to accomplish particular missions, the opacity associated with how AI systems reach a decision may cause an operator to have either too much or too little confidence in a system’s performance.
Consequently, unless the system’s machine learning algorithm is terminated, once deployed at the end of the training phase it could potentially learn something it was not intended to, or even perform a task or mission that its human designers do not expect it to do. This issue is one of the main reasons why the use of AI machine learning in the context of weapon systems is, for now, confined to mostly experimental research.
Even if nuclear early-warning systems might eventually detect the subversion, heightened levels of uncertainty and tension caused by an alert may impel the respective militaries to put their nuclear weapons on high alert status. This skewed assessment in the context of nuclear weapons, ready to launch at a moment’s notice, would likely precipitate worst-case scenario thinking that may spark inadvertent escalation.
Therefore, AI-augmented cyber intelligence gathering tools (or espionage) used during a crisis could easily be misinterpreted by an adversary as a prelude for a preemptive attack on its nuclear force.