Free beta access. 50 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

1. Unlike AI, nuke war’s not existential

Karina Vold & Daniel R. Harris 21, Vold is a philosopher of cognitive science and artificial intelligence & an assistant professor at the University of Toronto's Institute for the History and Philosophy of Science and Technology; Harris is a retired lawyer and Foreign Service Officer at the US Department of State, “How Does Artificial Intelligence Pose an Existential Risk?,” Oxford Handbook of Digital Ethics, Ed. C. Veliz., pp 1-34

The idea that AI might one day threaten humanity has been around for some time. In 1863, the novelist Samuel Butler (1863 ,185) suggested that machines may one day hold “supremacy over the world and its inhabitants”. By the mid-twentieth century, these concerns had left the realm of science fiction, as thinkers like Alan Turing (1951, 260) began to warn the public that we should expect intelligent machines to eventually “take control”. Still, for many years, academics did not spill much ink over these concerns, even while Hollywood filmmakers ran with them, producing countless blockbusters based on this “AI takeover” scenario (think: The Terminator or Battlestar Galactica). Over the last decade or so, however, many leading academics and entrepreneurs have notably increased their attention to existential risks from AI. These concerns are, as we will see, more subtle than those depicted in crude Hollywood-produced AI takeover scenarios. Indeed, those depictions have largely misrepresented the concrete issues scholars are concerned with by overly focusing on anthropomorphic concerns of conscious AI systems deciding to destroy humans.

This renewed scholarly interest in AI safety has been spurred on in part by the recent deep learning revolution. This period is defined by major advances in the accomplishments of deep neural networks— artificial neural networks with multiple layers between the input and output layers—across a wide range of areas, including game-playing, speech and facial recognition, and image generation. Even with these breakthroughs though, the cognitive capabilities of current AI systems remain limited to domain-specific applications. Nevertheless, many researchers are alarmed by the speed of progress in AI and worry that future systems, if not managed correctly, could present an existential threat.

Despite the renewed interest in this concern, there remains substantial disagreement over both the nature and the likelihood of the existential threats posed by AI. Hence, our aim in this chapter is to explicate the main arguments that have been given for thinking that AI does pose an existential risk, and to point out where there are disagreements and weakness in these arguments. The chapter has the following structure: in §2, we will introduce the concept of existential risk, the sources of such risks, and how these risks are typically assessed. In §3–5, we will critically examine three commonly cited reasons for thinking that AI poses an existential threat to humanity: the control problem, global disruption from an AI “arms race”, and the weaponization of AI. Our focus is on the first of these three, because it represents a kind of existential risk that is novel to AI as technology. While the latter two are equally important, they have commonalities with other kinds of technologies (e.g., nuclear weapons) discussed in the literature on existential risk, and so we will dedicate less time to them.

2. What Is an Existential Risk?

Many people believe that existential risks (henceforth, Xrisks) are the greatest threats facing humanity. And whilst there is much common ground amongst scholars about which scenarios constitute an Xrisk—the most commonly cited example is extinction risks1—there is not as much consensus on the precise definition of the concept (Beard et al., 2020; Torres, 2019). While most Xrisk scholars agree that a risk is existential if an adverse outcome would bring about human extinction, few endorse the narrower view that a risk is existential only if it would cause this outcome.2 Most definitions of Xrisk are broader, including at times the risk of global civilizational collapse (Rees, 2003; Ó hÉigeartaigh, 2017); scenarios in which the technological and moral potential of humanity is “permanently and drastically” curtailed (Bostrom, 2002, 2013); and suffering risks, defined as cases in which “an adverse outcome would bring about severe suffering on an astronomical scale, vastly exceeding all suffering that has existed on Earth so far” (Sotala & Gloor, 2017, 389).

Xrisks are typically distinguished from the broader category of global catastrophic risks. Bostrom (2013), for example, uses two dimensions—scope and severity—to make this distinction. Scope refers to the number of people at risk, while severity refers to how badly the population in question would be affected (ibid, 16). Xrisks are at the most extreme end of both of these spectrums: they are pan-generational in scope (i.e., “affecting humanity over all, or almost all, future generations”), and they are the severest kinds of threats, causing either “death or a permanent and drastic reduction of quality of life” (ibid, 17). Perhaps the clearest example of an Xrisk is an asteroid impact on the scale of that which hit the Earth 66 million years ago, wiping out the dinosaurs (Schulte et al., 2010; Ó hÉigeartaigh, 2017). Global catastrophic risks, by way of contrast, could be either just as severe but narrower in scope, or just as broad but less severe. Some examples include the destruction of cultural heritage, thinning of the ozone layer, or even a large-scale pandemic outbreak (Bostrom, 2013). In this chapter, we will focus mostly on the least controversial category of Xrisks— extinction risks—but will also at times discuss some of the other scenarios mentioned.

2.1 Sources of Xrisk

For most of human history, the only source of Xrisks facing humanity were natural causes, such as an asteroid hitting Earth or a global pandemic (Bostrom, 2002). But the creation of the first atomic bomb in 1945 introduced a new source of existential threat to humanity, one that was anthropogenic in nature. But since then, humanity has created numerous other kinds of threats to our own existence, including human- caused climate change, global biodiversity loss, biological warfare, and threats from artificial intelligence, for example. In fact, it is widely thought that most Xrisks today are anthropogenic and that, as a result of these new threats, this current century is the riskiest one that humanity has ever faced (Rees, 2003; Bostrom, 2013; Ó hÉigeartaigh, 2017; Ord, 2020).

Not all of these threats pose straightforward Xrisks. Let’s consider an extinction scenario to be the existential outcome in question, and then take nuclear fallout as an example. Today, the worldwide arsenal of nuclear weapons could lead to unprecedented death tolls and habitat destruction and, hence, it poses a clear global catastrophic risk. Still, experts assign a relatively low probability to human extinction from nuclear warfare (Martin, 1982; Sandberg & Bostrom, 2008; Shulman, 2012). This is in part because it seems more likely that extinction, if it follows at all, would occur indirectly from the effects of the war, rather than directly. This distinction has appeared in several discussions on Xrisks (e.g., Matheny, 2007, Liu et al., 2018; Zwetsloot & Dafoe, 2019), but it is made most explicitly in Cotton-Barratt et al. (2020, 6), who explain that a global catastrophe that causes human extinction can do so either directly by “killing everyone”, or indirectly, by “removing our ability to continue flourishing over a longer period.” A nuclear explosion itself is unlikely to kill everyone directly, but the resulting effects it has on the Earth could lead to lands becoming uninhabitable, in turn leading to a scarcity of essential resources, which could (over a number of years) lead to human extinction. Some of the simplest examples of direct risks of human extinction, by way of contrast, are “[i]f the entire planet is struck by a deadly gamma ray burst, or enough of a deadly toxin is dispersed through the atmosphere” (ibid, 6). What’s critical here is that for an Xrisk to be direct it has to be able to reach everyone.

Sign in to recut, update, or save this card.