Independently, fragmented allied AI systems risk accidents and miscalculated escalation.
Stanley-Lockman & Trabucco 22 --- *Defense and Strategic Studies, Nanyang Technological University. **Political Science, University of Copenhagen.
Zoe & Lena, “NATO’s Role in Responsible AI Governance in Military Affairs,” Oxford Handbook, https://www.oxfordhandbooks.com/view/10.1093/oxfordhb/9780197579329.001.0001/oxfordhb-9780197579329-e-69
Even without being attacked, governability of AI in a NATO context also means understanding how AI-enabled and autonomous systems developed by the 30 Allies – and other partners – will interact with one another. NATO has expressed interest in governability as a principle of AI “to disengage or deactivate in case of unintended behaviour,”86 which echoes the U.S. Department of Defense definition of governable AI.87 Disengaging adversaries is important to maintain de-escalation measures in conflict. For NATO, interoperability between systems also relates to governable AI because allies must also consider how the interactions between the 30 Allies’ own AI-enabled and autonomous systems may result in unintended or emergent behaviour.88 This means that NATO has a responsibility to coordinate activities – be they technical exchanges, standardisation efforts, or training and exercises – to build confidence that the systems perform as humans intend.89 Without this coordination, the lack of interoperability of allied systems could lead to accidents, and separately, the potential loss of operational effectiveness also presents vulnerabilities for adversaries to exploit.
In addition to governability, NATO and its Allies are assessing the risks that bias, attacks, and lack of interpretability can introduce in relation to the anticipated uses of a given AI system.90 In security and defence, new and heightened risks include poisoning of the information environment, deception systems and techniques, uncertainty about the performance of systems in new and unknown environments, and the possibility that tensions or accidents escalate at a faster tempo than humans and institutions can process, among others. These risks can manifest either in motivated attacks or unintentional failure modes.91 In both cases, assuring and certifying that military assets are safe and secure is important given the inherently high risk in operational environments. These operational environments include the presumption that an adversary is disrupting one’s own systems, be it by directly attacking the AI systems themselves, or disrupting the broader command, control, and communications systems under which the AI systems are operating.92