AI attacks destroy every military sector because of shared datasets.

A second major AI attack surface is the military. Military applications of AI are expected to be a critical component of the next major war. The U.S. Department of Defense has recently made the integration of artificial intelligence and machine learning into the military a high priority with its creation of the Joint Artificial Intelligence Center (JAIC). The JAIC has “the overarching goal of accelerating the delivery of AI-enabled capabilities, scaling the Department-wide impact of AI, and synchronizing DoD AI activities to expand Joint Force advantages.”35 The Pentagon’s Project Maven applies AI to the analysis of full motion video (FMV), highlighting the military’s desire to use AI to identify ground-based assets.36 The Defense Advanced Research Projects Agency (DARPA) and Air Force Research Laboratory’s (AFRL) release of the Moving and Stationary Target Acquisition and Recognition dataset is aimed at building AI techniques to classify and recognize targets of interest.37 Attacking these military AI systems is the logical successor of General Patton’s FUSAG. The contested environments in which the military operates creates a number of unique ways for adversaries to craft attacks against these military systems, and correspondingly, a number of unique challenges in defending against them. First, adversaries may capture the physical equipment, including drones and weapon systems, on which AI systems will live. The loss and capture of this equipment will be routine in future conflicts, and the threat this poses to AI systems will grow as more and more AI-enabled systems are deployed in the field or on equipment that can be captured by an adversary. This trend will further increase with the proliferation of “edge computing” in military contexts. In edge computing, rather than sending data to a centralized cloud infrastructure for processing, the data and AI algorithms are stored and run directly on the devices deployed in the field. The DoD has made the development of “edge computing” a priority, as the bandwidth needed to support a cloud-based AI paradigm is unlikely to be available in battlefield environments.38 This reality will require these systems to be treated with care. Just as the military recognizes the threat created when a plane, drone, or weapon system is captured by an enemy, these AI systems must be recognized and treated as a member of this same protected class so that the systems are not compromised if captured by an enemy. Second, the military’s unique domain necessitates the creation of similarly unique datasets and tools, both of which are likely to be shared within the military at-large. Because these datasets and systems will be expensive and difficult to create, there will be significant pressures to share them widely among different applications and branches. However, when multiple AI systems depend on this small set of shared assets, a single compromise of a dataset or system would expose all dependent systems to attack. Despite this risk, shared datasets are expected to become widespread within military AI operations. The DoD has already stated that the foundation for its AI efforts “includes shared data, reusable tools, frameworks, libraries, and standards…”39 The initial DoD AI applications, which focus on extracting information from aerial images and video, illustrate why sharing datasets is attractive. These datasets are critical to developing a set of powerful AI systems, but are expensive—both in terms of time and money—to collect and prepare.40 As a result, there is a logical desire to share and reuse these datasets across many different applications rather than creating a separate dataset for each application. However, this creates a single point of vulnerability for system-wide attacks. If this data is hacked or compromised, every application developed using this data would be potentially compromised. If a large number of applications depended on this same shared dataset, this could lead to widespread vulnerabilities throughout the military. In the case of input attacks, an adversary would then be easily able to find attack patterns to engineer an attack on any systems trained using the dataset. In the case of poisoning attacks, an adversary would only need to compromise one dataset in order to poison any downstream models that are later trained using this poisoned dataset.