Free beta access. 50 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

NC3s are dangerous and vulnerable because they are trained through machine learning

Vincent Boulanin, May 2019, "," No Publication, https://www.sipri.org/sites/default/files/2019-05/sipri1905-ai-strategic-stability-nuclear-risk.pdf

It is already known that a broad range of models trained through machine learning are susceptible to a new kind of vulnerability, termed ‘adversarial examples’: an adversary can craft a malicious input that reliably causes a trained model to produce the wrong behaviour (e.g. misclassify an object in an image or take an inappropriate action in the environment).7 While this vulnerability has been known and researched heavily for several years, no robust solution has yet been found. Nonetheless, given the promise of new capabilities that machine learning and automation offer, the pressure to deploy potentially insecure systems may present itself.8 When considering threats that might be introduced from increased autonomy and use of machine learning, it is important to consider the entire sprawling range of systems and functions that make up and support NC4ISR. Specific attention has been given to delivery systems and to nuclear command, control and communications (NC3).9 The awareness of potential threats to ‘core’ computer systems in NC3 has led to significantly improved security for such systems, and some reluctance to introduce autonomy and machine learning into them.10 However, more peripheral systems can also pose a threat, especially as they are more likely sites for the introduction of autonomy and machine learning. These include, for example, systems onboard satellites that relay communications and images or the simulators used to plan and test strategies. They can also extend as far as the vast computer systems and networks that provide news information to the public and to civilian officials, which may affect tactical or strategic decision-making. Admittedly, it is not always easy to chart a scenario that begins with a compromise of a particular peripheral system and ends with the unauthorized launch of a nuclear weapon.11 It is similarly difficult to describe a scenario whereby an adversary would intervene in the authorized launch of a nuclear weapon. However, these systems are present for the well-funded and patient adversary to explore and exploit. In particular, there is increasing concern about attacks that initially target command, control, communications, computers, intelligence, surveillance and reconnaissance (C4ISR) systems that are ‘entangled’—that is, used for both nuclear and conventional weapons—such as satellites, intelligence gathering and logistics.12 Entangled systems present two challenges here: first, they are often not considered ‘nuclear’ systems, so are subject to a lower level of security scrutiny than nuclear systems. Second, attacks on such systems may be considered by an adversary as unlikely to trigger a nuclear escalation, leading to a miscalculation—the adversary may not even know that the system has an NC4ISR purpose and may therefore consider the attack to be conventional while the targeted state may perceive the attack as an attack on its nuclear capabilities.

Sign in to recut, update, or save this card.