The best move is to not play in the arms race at all
Paul Scharre, 19, (Paul Scharre, 6-1-2019, Foreign Affairs, Killer Apps, https://www.foreignaffairs.com/articles/2019-04-16/killer-apps?check_logged_in=1&utm_medium=promo_email&utm_source=lo_flows&utm_campaign=registered_user_welcome&utm_term=email_1&utm_content=20220626, 6-26-2022) SCade
THE ONLY WINNING MOVE IS NOT TO PLAY Today’s AI technologies are powerful but unreliable. Rules-based systems cannot deal with circumstances their programmers did not anticipate. Learning systems are limited by the data on which they were trained. AI failures have already led to tragedy. Advanced autopilot features in cars, although they perform well in some circumstances, have driven cars without warning into trucks, concrete barriers, and parked cars. In the wrong situation, AI systems go from supersmart to superdumb in an instant. When an enemy is trying to manipulate and hack an AI system, the risks are even greater. Even when they don’t break down completely, learning systems sometimes learn to achieve their goals in the wrong way. In a research paper last year, a group of 52 AI researchers recounted dozens of times when AI systems showed surprising behavior. An algorithm learning to walk in a simulated environment discovered it could move fastest by repeatedly falling over. A Tetris-playing bot learned to pause the game before the last brick fell, so that it would never lose. One program deleted the files containing the answers against which it was being evaluated, causing it to be awarded a perfect score. As the researchers wrote, “It is often functionally simpler for evolution to exploit loopholes in the quantitative measure than it is to achieve the actual desired outcome.” Surprise seems to be a standard feature of learning systems. Machine-learning systems are only ever as good as their training data. If the data don’t represent the system’s operating environment well, the system can fail in the real world. In 2018, for example, researchers at the MIT Media Lab showed that three leading facial recognition systems were far worse at classifying dark-skinned faces than they were at classifying light-skinned ones. When they fail, machine-learning systems are also often frustratingly opaque. For rules-based systems, researchers can always explain the machine’s behavior, even if they can’t always predict it. For deep-learning systems, however, researchers are often unable to understand why a machine did what it did. Ali Rahimi, an AI researcher at Google, has argued that much like medieval alchemists, who discovered modern glassmaking techniques but did not understand the chemistry or physics behind their breakthroughs, modern machine-learning engineers can achieve powerful results but lack the underlying science to explain them. Every failing of an AI system also presents a vulnerability that can be exploited. In some cases, attackers can poison the training data. In 2016, Microsoft created a chatbot called Tay and gave it a Twitter account. Other users began tweeting offensive messages at it, and within 24 hours, Tay had begun parroting their racist and anti-Semitic language. In that case, the source of the bad data was obvious. But not all data-poisoning attacks are so visible. Some can be buried within the training data in a way that is undetectable to humans but still manipulates the machine. Even if the creators of a deep-learning system protect its data sources, the system can still be tricked using what are known as “adversarial examples,” in which an attacker feeds the system an input that is carefully tailored to get the machine to make a mistake. A neural network classifying satellite images might be tricked into identifying a subtly altered picture of a hospital as a military airfield or vice versa. The change in the image can be so small that the picture looks normal to a human but still fools the AI. Adversarial examples can even be placed in physical objects. In one case, researchers created a plastic turtle with subtle swirls embedded in the shell that made an object identification system think it was a rifle. In another, researchers placed a handful of small white and black squares on a stop sign, causing a neural network to classify it as a 45-mile-per-hour speed-limit sign. To make matters worse, attackers can develop these kinds of deceptive images and objects without access to the training data or the underlying algorithm of the system they are trying to defeat, and researchers have struggled to find effective defenses against the threat. Unlike with cybersecurity vulnerabilities, which can often be patched once they are uncovered, there is no known way of fully inoculating algorithms against these attacks. Governments already have plenty of experience testing military, cyber-, and surveillance tools, but no testing method can guarantee that complex systems won’t experience glitches once they’re out in the real world. The first time F-22 fighter jets crossed the International Date Line, their computers crashed and the aircraft were nearly stranded over the Pacific Ocean. Testing AI systems often takes even more time and money than testing traditional military hardware. Their complexity, which makes them more capable, also creates more opportunities for unexpected glitches. Imagine that a government develops an AI system that can hack into its adversaries’ computer networks while avoiding detection. The first government to deploy such a system would gain a huge advantage over its competitors. Worried that an adversary was developing a similar tool, the government might feel compelled to cut testing short and deploy the system early. This dynamic has already played out in other industries, such as self-driving cars. The consequences of accidents caused by national security AI tools could be far worse. AI wouldn’t be the first case of governments relying on a powerful but unsafe technology. That’s exactly what happened with computers, which play critical roles in everything from trading stocks to guiding missiles even though they suffer from enormous vulnerabilities. In 2018, investigators at the U.S. Government Accountability Office found that U.S. weapons systems were riddled with cybersecurity loopholes that could be exploited with “relatively simple tools and techniques.” Even worse, Defense Department program managers didn’t know about the problems and dismissed the GAO’s findings, claiming the tests were not realistic. Computer security vulnerabilities aren’t limited to government-run systems. Company after company has suffered major data breaches. Digital security is already too often an afterthought. A world of widespread, unprotected AI systems isn’t just a possibility; it’s the default setting.