Humans won’t detect failures---they’ll over-trust military A.I., prompting escalation.

An equal danger is what analysts call “automation bias,” or the tendency for stressed-out decision-makers to trust the information and advice supplied by advanced computers rather than their own considered judgment. For example, a U.S. president, when informed of sensor data indicating an enemy nuclear attack and under pressure to make an immediate decision, might choose to accept the computer’s advice to initiate a retaliatory strike rather than consider possible alternatives, such as with Petrov’s courageous Cold War action. Given that AI data systems can be expected to gain ever more analytical capacity over the coming decades, “it is likely that humans making command decisions will treat the AI system’s suggestions as on a par with or better than those of human advisers,” a 2018 RAND study noted. “This potentially unjustified trust presents new risks that must be considered.”19

Compounding all these risks is the likelihood that China, Russia, and the United States will all install automated NC3 systems but without informing each other of the nature and status of these systems. Under these circumstances, it is possible to imagine a “flash war,” roughly akin to a “flash crash” on Wall Street, that is triggered by the interaction of competing corporate investment algorithms. In such a scenario, the data assessment systems of each country could misinterpret signs of adversary moves and conclude an attack is imminent, leading other computers to order preparatory moves for a retaliatory strike, in turn prompting the similar moves on the other side, until both commence a rapid escalatory cycle ending in nuclear catastrophe.20