Free beta access. 50 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

Good AI invents solutions to present AND speculative future risks---but regs are key

Toby Ord 20, Senior Research Fellow in Philosophy at Oxford University, “5. Future Risks,” The Precipice: Existential Risk and the Future of Humanity, First edition, Hachette Books, 2020, pp. 121–158

Even if these arguments for risk are entirely wrong in the particulars, we should pay close attention to the development of AGI as it may bring other, unforeseen, risks. The transition to a world where humans are no longer the most intelligent entities on Earth could easily be the greatest ever change in humanity’s place in the universe. We shouldn’t be surprised if events surrounding this transition determine how our longterm future plays out— for better or worse.

One key way in which AI could help improve humanity’s longterm future is by offering protection from the other existential risks we face. For example, AI may enable us to find solutions to major risks or to identify new risks that would have blindsided us. AI may also help make our longterm future brighter than anything that could be achieved without it. So the idea that developments in AI may pose an existential risk is not an argument for abandoning AI, but an argument for proceeding with due caution.

The case for existential risk from AI is clearly speculative. Indeed, it is the most speculative case for a major risk in this book. Yet a speculative case that there is a large risk can be more important than a robust case for a very low-probability risk, such as that posed by asteroids. What we need are ways to judge just how speculative it really is, and a very useful starting point is to hear what those working in the field think about this risk.

Some outspoken AI researchers, like Professor Oren Etzioni, have painted it as “very much a fringe argument,” saying that while luminaries like Stephen Hawking, Elon Musk and Bill Gates may be deeply concerned, the people actually working in AI are not.103 If true, this would provide good reason to be skeptical of the risk. But even a cursory look at what the leading figures in AI are saying shows it is not.

For example, Stuart Russell, a professor at the University of California, Berkeley, and author of the most popular and widely respected textbook in AI, has strongly warned of the existential risk from AGI. He has gone so far as to set up the Center for Human-Compatible AI, to work on the alignment problem.104 In industry, Shane Legg (Chief Scientist at DeepMind) has warned of the existential dangers and helped to develop the field of alignment research.105 Indeed many other leading figures from the early days of AI to the present have made similar statements.106

There is actually less disagreement here than first appears. The main points of those who downplay the risks are that (1) we likely have decades left before AI matches or exceeds human abilities, and (2) attempting to immediately regulate research in AI would be a great mistake. Yet neither of these points is actually contested by those who counsel caution: they agree that the time frame to AGI is decades, not years, and typically suggest research on alignment, not regulation. So the substantive disagreement is not really over whether AGI is possible or whether it plausibly could be threat to humanity. It is over whether a potential existential threat that looks to be decades away should be of concern to us now. It seems to me that it should.

One of the underlying drivers of the apparent disagreement is a difference in viewpoint on what it means to be appropriately conservative. This is well illustrated by a much earlier case of speculative risk, when Leo Szilard and Enrico Fermi first talked about the possibility of an atomic bomb: “Fermi thought that the conservative thing was to play down the possibility that this may happen, and I thought the conservative thing was to assume that it would happen and take all the necessary precautions.”107 In 2015 I saw this same dynamic at the seminal Puerto Rico conference on the future of AI. Everyone acknowledged that the uncertainty and disagreement about timelines to AGI required us to use “conservative assumptions” about progress—but half used the term to allow for unfortunately slow scientific progress and half used it to allow for unfortunately quick onset of the risk. I believe much of the existing tension on whether to take risks from AGI seriously comes down to these disagreements about what it means to make responsible, conservative, guesses about future progress in AI.

That conference in Puerto Rico was a watershed moment for concern about existential risk from AI. Substantial agreement was reached and many participants signed an open letter about the need to begin working in earnest to make AI both robust and beneficial.108 Two years later an expanded conference reconvened at Asilomar, a location chosen to echo the famous genetics conference of 1975, where biologists came together to pre- emptively agree principles to govern the coming possibilities of genetic engineering. At Asilomar in 2017, the AI researchers agreed on a set of Asilomar AI Principles, to guide responsible longterm development of the field. These included principles specifically aimed at existential risk:

Capability Caution: There being no consensus, we should avoid strong assumptions regarding upper limits on future AI capabilities.

Importance: Advanced AI could represent a profound change in the history of life on Earth, and should be planned for and managed with commensurate care and resources.

Risks: Risks posed by AI systems, especially catastrophic or existential risks, must be subject to planning and mitigation efforts commensurate with their expected impact.109

Perhaps the best window into what those working on AI really believe comes from the 2016 survey of leading AI researchers. As well as asking if and when AGI might be developed, it asked about the risks: 70 percent of the researchers agreed with Stuart Russell’s broad argument about why advanced AI might pose a risk;110 48 percent thought society should prioritize AI safety research more (only 12 percent thought less). And half the respondents estimated that the probability of the longterm impact of AGI being “extremely bad (e.g., human extinction)” was at least 5 percent.111 I find this last point particularly remarkable—in how many other fields would the typical leading researcher think there is a one in twenty chance the field’s ultimate goal would be extremely bad for humanity?

Of course this doesn’t prove that the risks are real. But it shows that many AI researchers take seriously the possibilities that AGI will be developed within 50 years and that it could be an existential catastrophe. There is a lot of uncertainty and disagreement, but it is not at all a fringe position.

There is one interesting argument for skepticism about AI risk that gets stronger—not weaker—when more researchers acknowledge the risks. If researchers can see that building AI would be extremely dangerous, then why on earth would they go ahead with it? They are not simply going to build something that they know will destroy them.112

If we were all truly wise, altruistic and coordinated, then this argument would indeed work. But in the real world people tend to develop technologies as soon as the opportunity presents itself and deal with the consequences later. One reason for this comes from the variation in our beliefs: if even a small proportion of researchers don’t believe in the dangers (or welcome a world with machines in control), they will be the ones who take the final steps. This is an instance of the unilateralist’s curse (discussed here). Another reason involves incentives: even if some researchers thought the risk was as high as 10 percent, they may still want to take it if they thought they would reap most of the benefits. This may be rational in terms of their self-interest, yet terrible for the world.

In some cases like this, government can step in to resolve these coordination and incentive problems in the public interest. But here these exact same coordination and incentive problems arise between states and there are no easy mechanisms for resolving those. If one state were to take it slowly and safely, they may fear others would try to seize the prize. Treaties are made exceptionally difficult because verification that the others are complying is even more difficult here than for bioweapons.113

Whether we survive the development of AI with our longterm potential intact may depend on whether we can learn to align and control AI systems faster than we can develop systems capable enough to pose a threat. Thankfully, researchers are already working on a variety of the key issues, including making AI more secure, more robust and more interpretable. But there are still very few people working on the core issue of aligning AI with human values. This is a young field that is going to need to progress a very long way if we are to achieve our security.

Even though our current and foreseeable systems pose no threat to humanity at large, time is of the essence. In part this is because progress may come very suddenly: through unpredictable research breakthroughs, or by rapid scaling-up of the first intelligent systems (for example by rolling them out to thousands of times as much hardware, or allowing them to improve their own intelligence).114 And in part it is because such a momentous change in human affairs may require more than a couple of decades to adequately prepare for. In the words of Demis Hassabis, co- founder of DeepMind:

We need to use the downtime, when things are calm, to prepare for when things get serious in the decades to come. The time we have now is valuable, and we need to make use of it.115

DYSTOPIAN SCENARIOS

So far we have focused on two kinds of existential catastrophe: extinction and the unrecoverable collapse of civilization. But these are not the only possibilities. Recall that an existential catastrophe is the permanent destruction of humanity’s longterm potential, and that this is interpreted broadly, including outcomes where a small fragment of potential may remain.

Losing our potential means getting locked into a bad set of futures. We can categorize existential catastrophes by looking at which aspects of our future get locked in. This could be a world without humans (extinction) or a world without civilization (unrecoverable collapse). But it could also take the form of an unrecoverable dystopia—a world with civilization intact, but locked into a terrible form, with little or no value.116

This has not happened yet, but the past provides little comfort. For these kinds of catastrophes only became possible with the advent of civilization, so our track record is much shorter. And there is reason to think that the risks may increase over time as the world becomes more interconnected and experiments with new technologies and ideologies.

I won’t attempt to address these dystopian scenarios with the same level of scientific detail as the risks we’ve explored so far, for the scenarios are diverse and our present understanding of them very limited. Instead, my aim is just to take some early steps toward noticing and understanding these very different kinds of failure.

We can divide the unrecoverable dystopias we might face into three types, on the basis of whether they are desired by the people who live in them. There are possibilities where the people don’t want that world, yet the structure of society makes it almost impossible for them to coordinate to change it. There are possibilities where the people do want that world, yet they are misguided and the world falls far short of what they could have achieved. And in between there are possibilities where only a small group wants that world but enforces it against the wishes of the rest. Each of these types has different hurdles it would need to overcome in order to become truly locked in.

[FIGURE 5.2 OMITTED]

Note that to count as existential catastrophes, these outcomes don’t need to be impossible to break out of, nor to last millions of years. Instead, the defining feature is that entering that regime was a crucial negative turning point in the history of human potential, locking off almost all our potential for a worthy future. One way to look at this is that when they end (as they eventually must), we are much more likely than we were before to fall down to extinction or collapse than to rise up to fulfill our potential. For example, a dystopian society that lasted all the way until humanity was destroyed by external forces would be an existential catastrophe. However, if a dystopian outcome does not have this property, if it leaves open all our chances for success once it ends—it is a dark age in our story, but not a true existential catastrophe.

The most familiar type is the enforced dystopia. The rise of expansionist totalitarianism in the mid-twentieth century caused intellectuals such as George Orwell to raise the possibility of a totalitarian state achieving global dominance and absolute control, locking the world into a miserable condition.117 The regimes of Hitler and Stalin serve as a proof of principle, each scaling up to become imperial superpowers while maintaining extreme control over their citizens.118 However, it is unclear whether Hitler or Stalin had the expansionist aims to control the entire world, or the technical and social means to create truly lasting regimes.119

This may change. Technological progress has offered many new tools that could be used to detect and undermine dissent, and there is every reason to believe that this will continue over the next century. Advances in AI seem especially relevant, allowing automated, detailed monitoring of everything that happens in public places—both physical and online. Such advances may make it possible to have regimes that are far more stable than those of old.

That said, technology is also providing new tools for rebellion against authority, such as the internet and encrypted messages. Perhaps the forces will remain in balance, or shift in favor of freedom, but there is a credible chance that they will shift toward greater control over the populace, making enforced dystopias a realistic possibility.

A second kind of unrecoverable dystopia is a stable civilization that is desired by few (if any) people. It is easy to see how such an outcome could be dystopian, but not immediately obvious how we could arrive at it, or lock it in, if most (or all) people do not want it.120

The answer lies in the various population-level forces that can shape global outcomes. Well-known examples include market forces creating a race to the bottom, Malthusian population dynamics pushing down the average quality of life, or evolution optimizing us toward the spreading of our genes, regardless of the effects on what we value. These are all dynamics that push humanity toward a new equilibrium, where these forces are finally in balance. But there is no guarantee this equilibrium will be good.

For example, consider the tension between what is best for each and what is best for all. This is studied in the field of game theory through “games” like the prisoner’s dilemma and the tragedy of the commons, where each individual’s incentives push them toward producing a collectively terrible outcome. The Nash equilibrium (the outcome we reach if we follow individual incentives) may be much worse for everyone than some other outcome we could have achieved if we had overcome these local incentives.

The most famous example is environmental degradation, such as pollution. Because most of the costs of pollution aren’t borne by the person who causes it, we can end up in a situation where it is in the self-interest of each person to keep engaging in such activities, despite this making us all worse off. It took significant moral progress and significant political action to help us break out of this. We may end up in new traps that are even harder to coordinate our way out of. This could be at the level of individuals, or at the level of groups. We could have nations, ideological blocs, or even planets or descendent species of Homo sapiens locked in harmful competition—doing what is best for their group, but bad for groups on the whole.

I don’t know how likely it is that we suffer a sufficiently bad (and sufficiently intractable) tragedy of the commons like this. Or that we are degraded by evolutionary pressures, or driven to lives of very low quality by Malthusian population dynamics, or any other such situation. I’d like to hope that we could always see such things coming and coordinate to a solution. But it’s hard to be sure that we could.

The third possibility is the “desired dystopia.”121 Here it is easier to see how universal desire for an outcome might cause us to lock it in, though less clear how such an outcome could be dystopian. The problem is that there are many compelling ideas that can radically shape our future— especially ideologies and moral theories, as these make direct normative claims about the world we should strive to create. If combined with the technological or social means for instilling the same views in the next generation (indoctrination, surveillance), this has the potential to be disastrous.

The historical record is rife with examples of seriously defective ideologies and moral views that gripped large parts of the world. Moreover, even reasonable normative views often recommend that they be locked in— for otherwise a tempting rival view may take over, with (allegedly) disastrous results.122 Even though the most plausible moral views have a lot of agreement about which small changes to the world are good and which are bad, they tend to come strongly apart in their recommendations about what an optimal world would look like. This problem thus echoes that of AI alignment, where a strong push toward a mostly correct ideal could instead spell disaster.

Some plausible examples include: worlds that completely renounce further technological progress (which ensures our destruction at the hands of natural risks),123 worlds that forever fail to recognize some key form of harm or injustice (and thus perpetuate it blindly), worlds that lock in a single fundamentalist religion, and worlds where we deliberately replace ourselves with something that we didn’t realize was much less valuable (such as machines incapable of feeling).124

All of these unrecoverable dystopias can be understood in terms of lock-in. Key aspects of the future of the civilization are being locked in such that they are almost impossible to change. If we are locked into a sufficiently bad set of futures, we have an unrecoverable dystopia; an existential catastrophe.

Of course, we can also see lock-in on smaller scales. The Corwin Amendment to the US constitution provides a disturbing example of attempted lock-in. In an effort to placate the South and avoid civil war, the proposed Thirteenth Amendment aimed to lock in the institution of slavery by making it impossible for any future amendments to the constitution to ever abolish it.125

I cannot see how the world could be locked into a dystopian state in the near future.126 But as technology advances and the world becomes more and more interlinked, the probability of a locked-in dystopia would appear to rise, perhaps to appreciable levels within the next hundred years. Moreover, in the further future I think these kinds of outcomes may come to take up a high share of the remaining risk. For one thing, they are more subtle, so even if we got our act together and made preserving our longterm potential a high global priority, it may take remarkable wisdom and prudence to avoid some of these traps. And for another, our eventual spread beyond the Earth may make us nearly immune to natural catastrophes, but ideas travel at the speed of light and could still corrupt all that we hope to achieve.

A key problem is that the truth of an idea is only one contributor to its memetic potential—its ability to spread and to stick. But the more that rigorous and rational debate is encouraged, the more truth contributes to memetic success. So encouraging a culture of such debate may be one way we can now help avoid this fate. (For more on this, see the discussion of the Long Reflection in Chapter 7.)

The idea of lock-in also gives us another useful lens through which to think about existential risk in general. We might adopt the guiding principle of minimizing lock-in. Or to avoid the double negative, of preserving our options.127 This is closely related to the idea of preserving our longterm potential—the difference being that preserving our options takes no account of whether the options are good or bad. This is not because we intrinsically care about keeping options alive even if they are bad, but because we aren’t certain they are bad, so we risk making an irreversible catastrophic mistake if we forever foreclose an option that would turn out to be best.

OTHER RISKS

What other future risks are there that warrant our concern?

One of the most transformative technologies that might be developed this century is nanotechnology. We have already seen the advent of nanomaterials (such as carbon nanotubes) which are just a few atoms thick and structured with atomic precision. But much larger vistas would open up if we could develop machinery that operates with atomic precision. We have proof that some form of this is possible within our very own cells, where atomically precise machinery already performs their essential functions.

In the popular imagination nanotechnology is synonymous with building microscopic machines. But the bigger revolution may instead come from using nanomachinery to create macro-scale objects. In his foundational work on the topic, Eric Drexler describes how nanotechnology could allow desktop fabricators, capable of assembling anything from a diamond necklace to a new laptop. This atomically precise manufacturing would be the ultimate form of 3D printing: taking a digital blueprint for the object and the raw chemical elements, and producing an atomically precise instance. This may allow us to construct things beyond our current technological reach, as well as cutting prices of existing objects such as computers or solar cells to near the cost of their raw materials, granting the world vastly more computing power and clean energy.

Such a powerful technology may pose some existential risk. Most attention has so far focused on the possibility of creating tiny self- replicating machines that could spread to create an ecological catastrophe. This may be possible, but there are mundane dangers that appear more likely, since extreme manufacturing power and precision would probably also allow the production of new weapons of mass destruction.128 Indeed the problems resemble those of advanced biotechnology: the democratization of extremely powerful technology would allow individuals or small groups access to the kinds of power (both constructive and destructive) that was previously only available to powerful nations. Solutions to managing this technology may require digital controls on what can be fabricated or state control of fabrication (the path we took with nuclear power). While this technology is more speculative than advanced biotechnology or AI, it may also come to pose a significant risk.

A very different kind of risk may come from our explorations beyond the Earth. Space agencies are planning missions which would return soil samples from Mars to the Earth, with the chief aim of looking for signs of life. This raises the possibility of “back contamination” in which microbes from Mars might compromise the Earth’s biosphere. While there is a consensus that the risk is extremely small, it is taken very seriously.129 The plan is to return such samples to a new kind of BSL-4 facility, with safeguards to keep the chance of any unsterilized particle escaping into the environment below one in a million.130 While there are still many unknown factors, this anthropogenic risk appears comparatively small and well managed.131

The extra-terrestrial risk that looms largest in popular culture is conflict with a spacefaring alien civilization. While it is very difficult to definitively rule this out, it is widely regarded to be extremely unlikely (though becoming more plausible over the extreme long term).132 The main risk in popular depictions is from aliens traveling to Earth, though this is probably the least likely possibility and the one we could do the least about. But perhaps more public discussion should be had before we engage in active SETI (sending powerful signals to attract the attention of distant aliens). And even passive SETI (listening for their messages) could hold dangers, as the message could be designed to entrap us.133 These dangers are small, but poorly understood and not yet well managed.

Another kind of anthropogenic risk comes from our most radical scientific experiments—those which create truly unprecedented conditions.134 For example, the first nuclear explosion created temperatures that had never before occurred on Earth, opening up the theoretical possibility that it might ignite the atmosphere. Because these conditions were unprecedented we lost the reassuring argument that this kind of event has happened many times before without catastrophe. (We could view several of the risks we have already discussed—such as back contamination, gain of function research and AGI—through this lens of science experiments creating unprecedented conditions.)

In some cases, scientists confidently assert that it is impossible for the experiment to cause a disaster or extinction. But even core scientific certainties have been wrong before: for example, that objects have determinate locations, that space obeys Euclid’s axioms, and that atoms can’t be subdivided, created or destroyed. If pressed, the scientists would clarify that they really mean it couldn’t happen without a major change to our scientific theories. This is sufficient certainty from the usual perspective of seeking accurate knowledge, where 99.9 percent certainty is more than enough. But that is a standard which is independent of the stakes. Here the stakes are uniquely high and we need a standard that is sensitive to this.135

The usual approach would be to compare the expected gains to the expected losses. But that is challenging to apply, as a very low (and hard to quantify) chance of enormous catastrophe needs to be weighed against the tangible benefits that such experiments have brought and are likely to bring again. Furthermore, the knowledge or the technologies enabled by the experiments may help lower future existential risk, or may be necessary for fulfilling our potential.

For any given experiment that creates truly unprecedented conditions, the chance of catastrophe will generally be very small. But there may be exceptions, and the aggregate chance may build up. These risks are generally not well governed.136

These risks posed by future technologies are by their very nature more speculative than those from natural hazards or the most powerful technologies of the present day. And this is especially true as we moved from things that are just now becoming possible within biotechnology to those that are decades away, at best. But one doesn’t have to find all of these threats to be likely (or even plausible) to recognize that there are serious risks ahead. Even if we restrict our attention to engineered pandemics, I think there is more existential risk than in all risks of the last two chapters combined, and those risks were already sufficient to make safeguarding humanity a central priority of our time.

UNFORESEEN RISKS

Imagine if the scientific establishment of 1930 had been asked to compile a list of the existential risks humanity would face over the following hundred years. They would have missed most of the risks covered in this book—especially the anthropogenic risks.137 Some would have been on the edge of their awareness, while others would come as complete shocks. How much risk lies beyond the limits of our own vision?

We can get some inkling by considering that there has been no slow-down in the rate at which we’ve been discovering risks, nor the rate at which we’ve been producing them. It is thus likely we will face unforeseen risks over the next hundred years and beyond. Since humanity’s power is still rapidly growing, we shouldn’t be surprised if some of these novel threats pose a substantial amount of risk.

One might wonder what good can come of considering risks so far beyond our sight. While we cannot directly work on them, they may still be lowered through our broader efforts to create a world that takes its future seriously. Unforeseen risks are thus important to understanding the relative value of broad versus narrowly targeted efforts. And they are important for estimating the total risk we face.

Nick Bostrom has recently pointed to an important class of unforeseen risk.138 Every year as we invent new technologies, we may have a chance of stumbling across something that offers the destructive power of the atomic bomb or a deadly pandemic, but which turns out to be easy to produce from everyday materials. Discovering even one such technology might be enough to make the continued existence of human civilization impossible.

Sign in to recut, update, or save this card.