An “Arms Race” mentality creates pressure to deploy brittle, untested AI – this causes weapons failures in real world environments
Sharre, 2018 - director of the technology and national security program at the Center for a New American Security [Paul, 12 September, “Ultrafast computing is critical to modern warfare. But it also ensures a lot could go very wrong, very quickly.” https://foreignpolicy.com/2018/09/12/a-million-mistakes-a-second-future-of-war/ ST]
The problem is that automated systems—at least those using current technology—tend to be brittle. Machines are good at handling routine tasks under predictable circumstances, such as flying a commercial airliner. But automation can sometimes fail dramatically in new situations, which is a major reason why self-driving cars, which must contend with extremely dynamic and uncontrolled environments, have proved so much harder to develop than self-flying planes. Learning systems, which focus on a set of rules for processing and incorporating data into behavior instead of strict mandates, exhibit more flexibility but are still limited by the quality of information. Machine learning can fail if the real world proves different from training models—a major problem in the military context, where adversaries are unlikely to offer easy access to their tactics and hardware. Human intelligence is robust and adaptable in ways that machine intelligence is not—yet. The most effective militaries will thus be those that find ways to successfully marry human and machine intelligence into joint cognitive systems—an approach that defense analysts call centaur warfighting. Despite humans’ advantages in decision-making, an arms race in speed may slowly push humans out of the OODA loop. Militaries are unlikely to knowingly field weapons they cannot control, but war is a hazardous environment and requires balancing competing risks. Faced with the choice of falling behind an adversary or deploying a new and not yet fully tested weapon, militaries are likely to do what they must to keep pace with their enemies. As mentioned above, automated stock trading provides a useful window into the perils of this dynamic. In 2010, the Dow Jones Industrial Average lost nearly 10 percent of its value in just minutes. The cause? A sudden shift in market prices driven in part by automated trading, or what’s come to be known as a flash crash. In the last decade, financial markets have started to suffer such crashes, or at least miniature versions of them, on a regular basis. The circuit breakers installed by regulators to pull a stock offline can’t prevent incidents from occurring, but they can stop flash crashes from spiraling out of control. Circuit breakers are still regularly tripped, though, and on Aug. 24, 2015, more than 1,200 of them went off across multiple exchanges after China suddenly devalued the yuan. In competitive environments such as stock markets and battlefields, unexpected interactions between algorithms are natural. The causes of the 2010 flash crash are still disputed. In all likelihood, there were a range of causes, including an automated sell algorithm interacting with extreme market volatility, exacerbated by high-frequency trading and deliberate spoofing of trading algorithms. To prevent the military equivalent of such crises, in which autonomous weapons become trapped in a cascade of escalating engagements, countries will have to balance advantages in speed with the risk of accidents. Yet growing competition will make that balancing act ever more difficult. In 2016, Robert Work, then-U.S. deputy defense secretary, colorfully summed up the problem this way: “If our competitors go to Terminators, and it turns out the Terminators are able to make decisions faster, even if they’re bad, how would we respond?”