Failure to act now ensures extinction even if they are right

Artificial Intelligence. An artificial intelligence (AI) program may broadly be defined as a computer program that automatically finds ways to achieve goals. For example, an AI chess program finds moves for achieving the goal of checkmating the opponent. The moves have not been explicitly defined by the programmer – instead the program finds them by “weighing” the pros and cons of different strategies. An AI question-answering system automatically generates the answer to a query by searching a knowledge base. A self-driving car finds a way to control throttle, break, and steering, in order to get to its goal destination.

An important distinction between different AI systems is generality. Most present-day AIs can only solve narrow sets of tasks in restricted environments. For example, most chess programs can only play chess, and most question answering systems can only answer specific type of queries from a particular kind of database. This is in contrast to humans, who can learn to perform a wide range of cognitive tasks.

However, a trend towards greater generality can be observed in AI systems. While most self-driving car algorithms would not perform well on tasks other than driving a car, driving a car is already a rather general task. It involves acting in an environment with varying weather, traffic, and lightning conditions, using multiple sources of perception. While most board-game AIs have traditionally been tailored towards a specific type of board game, AlphaZero is a single algorithm that achieves super human performance in Go, Chess, and Shogi (Silver, Hubert, et al., 2017). Another popular benchmark is video games. Here again a single algorithm called DQN recently managed to outperform humans on a majority of the ATARI games (Hessel et al., 2017; Mnih, Kavukcuoglu, et al., 2015).

Extrapolating this trend towards greater generality and intelligence, we may expect AI systems to eventually surpass humans on both these metrics. Such systems are sometimes called artificial general intelligences (AGIs). Indeed, many expect the creation of the first human-level AGI to occur within the next few decades, and greatly superhuman AGI soon after (Bostrom, 2014; and Section 2.2 below).

Controlling AGI. A superhuman AGI is a system that outperforms humans on most cognitive tasks. In order to control it, humans would need to control a system more intelligent than themselves. This may be nearly impossible if the difference in intelligence is large, and the AGI is trying to escape control.

Humans have one key advantage: As the designers of the system, we get to decide the AGI’s goals, and the way the AGI strives to achieve its goals.2 This may allow us design AGIs whose goals are aligned with ours, and that pursue them in a responsible way. Increased intelligence in an AGI is not a threat as long as the AGI only strives to help us achieve our own goals.

On a high level, this goal alignment problem is challenging for at least two reasons: Specificity of human values, and Goodhart’s law. First, human values are rather specific. For example, most humans prefer futures with happy sentient beings over futures with much suffering or no sentient beings at all. But even futures where happiness greatly outweighs suffering can still be undesirable. Yudkowsky (2009) gives an example where a happy, sentient experience is set to repeat everywhere until the heat-death of the universe. Most people find it repulsive, because it lacks important values such as variation and growth. However, listing all important values is not an easy task, which makes it hard to define a goal for an AGI that matches our human values even if the goal is optimized by a highly intelligent system. In this way, an AGI resembles a fairy-tale genie who grants you a wish, but interprets it over-literally (Wiener, 1960). In most fairy tales, the hero eventually wants his wish undone.

A second reason why alignment is hard is Goodhart’s law (Goodhart, 1975, 1984), which roughly states that:

“When a measure becomes a target, it ceases to be a good measure.” (Strathern, 1997).

In the context of AGI, it means that if we give our system a proxy-measure for how well it is satisfying a goal, then the proxy is likely to cease being a good measure once the AGI starts optimizing for it. An oft-mentioned example is that of a reinforcement learning (RL) agent that uses a reward signal as goal proxy. In most circumstances, the reward may correspond well to the agent’s performance. But an AGI optimizing the reward may find a way to short circuit the reward signal, obtaining maximum reward for behaviors considered undesirable by its human designers (Ring and Orseau, 2011). This is a rather extreme instance of Goodhart’s law. Section 4.5 of this thesis is devoted to various aspects of the goal alignment problem.

Other aspects. In addition to goal alignment, some further properties may also contribute to making an AGI safe. We may for example ask that the AI system is corrigible (Soares, Fallenstein, et al., 2015) in the sense that it lets us correct mistakes in its source that we have made while designing it, and that it lets us shut it down if we judge that to be the best option. Unfortunately, many AI designs give the agent an incentive to prevent corrections and shut down. This, of course, is a type of misalignment. There has been some debate about whether corrigibility should be treated as a property separate from goal alignment (Carey, 2018), or be made to follow from a more general approach to goal alignment (Hadfield-Menell, Dragan, et al., 2017; Milli et al., 2017). We will study some aspects of corrigibility in Chapter 11.

The way an AGI makes decisions is also important. Even if its goals are aligned, it may still do the wrong thing if it follows the wrong decision principle. While expected utility maximization is a principle that many agree on, there are subtleties in how the expectation should be computed when the agent is part of the environment that it is interacting with. Some aspects of decision theory will be considered in Section 6.4 and Chapter 10.

Why study AGI before it exists? Why study the safety of AGI before it exists, and before we even know whether it will ever exist? There are at least two types of reasons for this. The first is pragmatic. If AGI is created, and we do not know how to control it, then the outcome could be catastrophic (Bostrom, 2014). It is customary to take precautions not only against catastrophes we know will happen, but also against catastrophes that have only a slight chance of occurring (for example, a city may decide to build earthquake safe buildings, even if the probability of an earth quake occurring is fairly low). As discussed in Section 2.2 below, many believe that AGI has more than a small probability of occurring reasonably soon, and it can potentially cause very significant catastrophes. The second reason is scientific. Potential future AGIs are theoretically interesting objects, and the question of how a human can control machines more intelligent than him or herself is philosophically stimulating.