a. On magnitude---AI engineers a dystopia worse than extinction.
Toby Ord 20, senior research fellow in philosophy at Oxford University and world-renowned risk-assessment expert who has advised WHO, the World Bank, the WEF, and the US National Intelligence Council, 3/3/2020, The Precipice: Existential Risk and the Future of Humanity, pp. 194-196, cc
Losing our potential means getting locked into a bad set of futures. We can categorize existential catastrophes by looking at which aspects of our future get locked in. This could be a world without humans (extinction) or a world without civilization (unrecoverable collapse). But it could also take the form of an unrecoverable dystopia—a world with civilization intact, but locked into a terrible form, with little or no value.116
[Footnote 116]
In the worst cases, perhaps even negative value—an outcome worse than extinction.
[Footnote 116 ends]
This has not happened yet, but the past provides little comfort. For these kinds of catastrophes only became possible with the advent of civilization, so our track record is much shorter. And there is reason to think that the risks may increase over time as the world becomes more interconnected and experiments with new technologies and ideologies.
I won’t attempt to address these dystopian scenarios with the same level of scientific detail as the risks we’ve explored so far, for the scenarios are diverse and our present understanding of them very limited. Instead, my aim is just to take some early steps toward noticing and understanding these very different kinds of failure.
We can divide the unrecoverable dystopias we might face into three types, on the basis of whether they are desired by the people who live in them. There are possibilities where the people don’t want that world, yet the structure of society makes it almost impossible for them to coordinate to change it. There are possibilities where the people do want that world, yet they are misguided and the world falls far short of what they could have achieved. And in between there are possibilities where only a small group wants that world but enforces it against the wishes of the rest. Each of these types has different hurdles it would need to overcome in order to become truly locked in.
Note that to count as existential catastrophes, these outcomes don’t need to be impossible to break out of, nor to last millions of years. Instead, the defining feature is that entering that regime was a crucial negative turning point in the history of human potential, locking off almost all our potential for a worthy future. One way to look at this is that when they end (as they eventually must), we are much more likely than we were before to fall down to extinction or collapse than to rise up to fulfill our potential. For example, a dystopian society that lasted all the way until humanity was destroyed by external forces would be an existential catastrophe. However, if a dystopian outcome does not have this property, if it leaves open all our chances for success once it ends—it is a dark age in our story, but not a true existential catastrophe.
The most familiar type is the enforced dystopia. The rise of expansionist totalitarianism in the mid-twentieth century caused intellectuals such as George Orwell to raise the possibility of a totalitarian state achieving global dominance and absolute control, locking the world into a miserable condition.117 The regimes of Hitler and Stalin serve as a proof of principle, each scaling up to become imperial superpowers while maintaining extreme control over their citizens.118 However, it is unclear whether Hitler or Stalin had the expansionist aims to control the entire world, or the technical and social means to create truly lasting regimes.119
This may change. Technological progress has offered many new tools that could be used to detect and undermine dissent, and there is every reason to believe that this will continue over the next century. Advances in AI seem especially relevant, allowing automated, detailed monitoring of everything that happens in public places— both physical and online. Such advances may make it possible to have regimes that are far more stable than those of old.
That said, technology is also providing new tools for rebellion against authority, such as the internet and encrypted messages. Perhaps the forces will remain in balance, or shift in favor of freedom, but there is a credible chance that they will shift toward greater control over the populace, making enforced dystopias a realistic possibility.
A second kind of unrecoverable dystopia is a stable civilization that is desired by few (if any) people. It is easy to see how such an outcome could be dystopian, but not immediately obvious how we could arrive at it, or lock it in, if most (or all) people do not want it.120
The answer lies in the various population-level forces that can shape global outcomes. Well-known examples include market forces creating a race to the bottom, Malthusian population dynamics pushing down the average quality of life, or evolution optimizing us toward the spreading of our genes, regardless of the effects on what we value. These are all dynamics that push humanity toward a new equilibrium, where these forces are finally in balance. But there is no guarantee this equilibrium will be good.
For example, consider the tension between what is best for each and what is best for all. This is studied in the field of game theory through “games” like the prisoner’s dilemma and the tragedy of the commons, where each individual’s incentives push them toward producing a collectively terrible outcome. The Nash equilibrium (the outcome we reach if we follow individual incentives) may be much worse for everyone than some other outcome we could have achieved if we had overcome these local incentives.
The most famous example is environmental degradation, such as pollution. Because most of the costs of pollution aren’t borne by the person who causes it, we can end up in a situation where it is in the self-interest of each person to keep engaging in such activities, despite this making us all worse off. It took significant moral progress and significant political action to help us break out of this. We may end up in new traps that are even harder to coordinate our way out of. This could be at the level of individuals, or at the level of groups. We could have nations, ideological blocs, or even planets or descendent species of Homo sapiens locked in harmful competition—doing what is best for their group, but bad for groups on the whole.
I don’t know how likely it is that we suffer a sufficiently bad (and sufficiently intractable) tragedy of the commons like this. Or that we are degraded by evolutionary pressures, or driven to lives of very low quality by Malthusian population dynamics, or any other such situation. I’d like to hope that we could always see such things coming and coordinate to a solution. But it’s hard to be sure that we could.
The third possibility is the “desired dystopia.”121 Here it is easier to see how universal desire for an outcome might cause us to lock it in, though less clear how such an outcome could be dystopian. The problem is that there are many compelling ideas that can radically shape our future —especially ideologies and moral theories, as these make direct normative claims about the world we should strive to create. If combined with the technological or social means for instilling the same views in the next generation (indoctrination, surveillance), this has the potential to be disastrous.
The historical record is rife with examples of seriously defective ideologies and moral views that gripped large parts of the world. Moreover, even reasonable normative views often recommend that they be locked in—for otherwise a tempting rival view may take over, with (allegedly) disastrous results.122 Even though the most plausible moral views have a lot of agreement about which small changes to the world are good and which are bad, they tend to come strongly apart in their recommendations about what an optimal world would look like. This problem thus echoes that of AI alignment, where a strong push toward a mostly correct ideal could instead spell disaster.
Some plausible examples include: worlds that completely renounce further technological progress (which ensures our destruction at the hands of natural risks),123 worlds that forever fail to recognize some key form of harm or injustice (and thus perpetuate it blindly), worlds that lock in a single fundamentalist religion, and worlds where we deliberately replace ourselves with something that we didn’t realize was much less valuable (such as machines incapable of feeling).124
All of these unrecoverable dystopias can be understood in terms of lock-in. Key aspects of the future of the civilization are being locked in such that they are almost impossible to change. If we are locked into a sufficiently bad set of futures, we have an unrecoverable dystopia; an existential catastrophe.