Engineers extinction-level pathogens, spurs other x-risks

But Deep Blue can only beat humans at chess. Artificial General Intelligence (AGI), by contrast, refers to any machine that can match humans across the board in a wide range of intellectual tasks. General intelligence is a far harder problem to solve, and progress here has been less dramatic than experts predicted it would be 50 years ago. In the last five years, however, breakthroughs in new visual and speech recognition systems have surprised AI researchers, and the most promising sign is re- search led by Demis Hassabis, the founder of Google DeepMind. The team at DeepMind re- ported developing a system that taught itself how to play 49 different Atari arcade games, such as Space Invaders, learning not only the rules but how to maximize its score. The re- searchers programmed the artificial agent with no background knowledge about the games, beyond telling it to score as high as it could. After playing the games many times, the AI worked out the best strategies for itself, includ- ing some the designers themselves were un- aware of. Hassabis describes this as “the first significant rung of the ladder toward proving a general learning system can work.” Of course, this rung is hardly threatening, except to de- signers of video games.

The most serious AI safety concerns arise not from narrow intelligence or even general intelligence, but rather from what many ex- perts argue will quickly follow AGI: artificial superintelligence. Because a capacity to learn would be an integral component of any system that attains general intelligence, the AI would be able to boost its own capabilities—such an AI would be able to design an improved version of itself, and then that smarter version would do the same, and so on. This recursive self-im- provement could result in an “intelligence ex- plosion,” a scenario that the mathematician I.J. Good, one of Alan Turing’s colleagues, first identified in 1965. In this scenario, a system’s intelligence increases, potentially even from a sub-human level in most areas but with specific capabilities in fields such as mathematics and computer science, to a lev- el far beyond that of humans in a relatively short period of time.

Professor Nick Bostrom, director of the Fu- ture of Humanity Institute at Oxford University, argues that catastrophe is the default outcome of such an intelligence explosion because of two features of machine superintelligence.

First, there is no reason to believe that a superintelligent AI will have goals anything like that of intelligent humans. Whereas intelligence in humans is often associated with contemplation, empathy, and nonviolence, it is far easier to program an AI that places final value on nothing more than calculating the digits of pi than it is to construct one that values human welfare or that has some conception of moral goodness, if that can be codified at all.

Second, final values that seem benign could lead to disastrous unintended consequences, even when the AI’s programmers have good intentions. Many final goals, when interpreted precisely, would lead to the acquisition of as many resources as possible and the elimi- nation of threats. Humans may pose such a threat, and they certainly control resources. It is far more difficult than it may seem to ensure that a machine more powerful than its creators doesn’t interpret its goals in ways that run pro- foundly counter to the creators’ true intentions. Bostrom gives two simplified examples to il- lustrate this difficulty. An AI designed for the mundane task of maximizing the number of paperclips produced in a factory would, taken to its logical extreme, start monopolizing resources from elsewhere. Once it realized that humans had the power to shut it down and end its paperclip mission, it might view humans as a threat and take appropriate actions—chang- ing the electronic locks on the factory doors or reprogramming the braking system in the CEO’s Tesla. Or take the final goal of “make humans smile”—a suitably diligent AI may interpret this as “paralyze human facial musculature into constant grins.”

Speculating about more realistic paths to disaster in such detail is likely to be mislead- ing, given that superintelligence remains years away. But such scenarios are not hard to imag- ine. Even today, much of society operates in the digital realm. Servers store financial assets; Wikipedia and Google Books host the sum of human knowledge; and military and commer- cial drones are increasingly guided by software. To adapt a disaster scenario from Eliezer Yud- kowsky, a leading AI safety theorist, an AI with access to the Internet could scan through existing research papers on biotechnology, develop a particularly lethal pathogen, and email specifications of DNA strings to online laboratories that offer DNA synthesis and sequencing with FedEx delivery (such labs already exist). It would only have to find one human who can be paid or fooled into receiving FedEx’d vials and mixing their contents in a specified way to unleash a global pandemic.