AGI leads to extinction---outweighs nuclear war

Department of Psychology, York University) 2023. “Artificial General Intelligence, Existential Risk, and Human Risk Perception” https://arxiv.org/pdf/2311.08698 // ZX

As with any modern technology, contemporaneous societal and civilizational pressures and scientific breakthroughs continue to shape the evolution of AI. Its effects, much like those of nuclear technology, can and likely will change substantially over time. For instance, the perceived necessity of securing a decisive military advantage over the Axis powers during the Second World War motivated the US’s development of nuclear weapons, but this gave way to a military arms race once the US and Soviet Union were locked into the decades-long Cold War. Since then, there has been an expansion of nuclear-armed states and the pressures for expansion continue. While the threat of mutually-assured destruction serves as a potent first-use deterrent, there have been several close calls that raise the question of how long humans can avoid a global nuclear catastrophe (i.e., a nuclear winter) given that there appears to be no harmonious “end of history” in sight (Barrett et al., 2013; Baum, 2015). Attention to existential risk posed by AI invariably concerns possible future AI rather than the current state of AI. Although current AI carries significant risk—for example, algorithmic processes may be unexplainable (Angelov et al., 2021; Vilone & Longo, 2021) or biased and possibly unfair (Fazelpour & Danks, 2021)—and possibly biased for inexplicable reasons (Bostrom & Yudkowsky, 2018)—this is not what most experts regard as existential risk from AI. Current AI does not pose an existential threat because its development has not reached the stage where machines have parity with human intelligence across a wide range of tasks and domains.

However, many forecasters believe that parity is not far off. Consider forecasting data from Metaculus, a reputation-based online forecasting and aggregation platform devoted to predicting the scientific and technological outcomes and their consequences (Aguirre, 2016). As of July 20, 2023, the median probability estimate of 1,041 Metaculus forecasters that human-machine parity in intelligence will be reached by 2040 was a staggering 92%.1The criterion for scoring parity as having occurred is based on the requirement that the machine system outperform at least two out of three graduate students drawn from elite universities and having backgrounds in physics, mathematics, and computer science, respectively, on a two-hour intelligence test. The machine and three humans could access the Internet but could not consult with other humans. Certainly, more stringent criterion for scoring intelligence parity would be expected to lower the pooled estimate. Nevertheless, other estimates based on takeoff speed models (e.g., Davidson, 2023) agree that by 2040, AGI will be capable of performing virtually all human tasks regardless of their intelligence requirements. Such models rest on many assumptions, however, and their coherence with forecasters’ estimates is just information, not proof. The prospect of intelligence parity is not necessarily bad. AGI that could think as intelligently as humans but at much faster speeds—what Vinge (1993) calls weak superhumanity —could accelerate scientific and technological progress that humans might have eventually made but over much longer times. This process has even been given the sticky name, PASTA—Process for Automating Scientific and Technological Advancement (Karnofsky, 2021). If AGI (or PASTA) were necessary to avert an existential calamity (or worse, a mass extinction), having it would be a very good thing. The differential clock speeds of humans and machines, however, casts doubt on the parity notion. As soon as AGI could perform as well on any intelligence test (broadly defined) as the best performing human, it will have exceeded human intelligence since, if nothing else, it will have faster-than-human speed.

Unlike humans, AGI would likely be able to access its source code, modify it, and replicate either original or modified copies. These abilities would set off a positive feedback loop constituting what Good (1966) referred to as an intelligence explosion. As a result, AGI would evolve very quickly to be strongly superintelligent; i.e., it would not only think faster, it would think smarter than the smartest human (Yukdowsky, 2008).It is the prospect of superintelligent AGI that raises concern regarding the continued existence our species and that has caused many AI experts and others alike to call for a pause in advanced AI development (e.g., Future of Life Institute, 2023). If AGI were more powerful than humans, a misalignment of human and AGI goals would likely not end well for humanity. This alignment problem—the question of whether AGI will become misaligned with human goals and values—is a threat analysis problem concerning a set of possible futures having devastating consequences for humans as a species. Since there are no tightly coupled reference classes to draw from in determining the probability of human extinction (or catastrophic demise) from AGI misalignment, estimating existential risk from AGI constitutes an extreme version of the reference class problem (Hájek, 2007; Reichenbach, 1949) with the various pro and con arguments drawing heavily on loose analogies. Therefore, it is of little surprise that estimates of risk vary greatly. A recent study by Karger et al. (2023) compared the forecasts of 32 AI experts, 48 experts on areas of existential risk other than from AI, and 89 “superforecasters” who demonstrated top tier forecasting skill on a wide spectrum of short-range topics (Mellers et al., 2015) and who tend to exhibit higher than average coherence across a range of probabilistic judgment tasks (Mellers et al., 2017).

The authors found that among various existential threats considered (i.e., AI, nuclear, biological, and climate), existential risk from AI prompted the greatest level of disagreement. The inter-quartile range of estimates of human extinction from AI by 2100, remarkably, was greater among AI experts than among other experts from other areas or superforecasters. As well, Karger et al. (2023) found that the largest absolute disagreement between superforecasters and area-relevant experts (i.e., AI experts for AI topics, nuclear experts for nuclear topics, and so on) focused on AI risk. However, this was largely attributable to the fact that both groups regarded the probability of extinction from AI to be larger than for other risks considered. For instance, whereas superforecasters’ median estimated risk of human extinction by 2100 due to AI was 0.38%, the comparable estimated risk due to nuclear technologies was 0.074%. Likewise, whereas AI experts’ median estimated risk of extinction by 2100 was 3.0%, the comparable risk from nuclear technologies forecasted by nuclear experts was 0.55%. Remarkably, among both superforecasters and domain experts, the judged probability of human extinction by 2100 due to AI is just over five times higher than the perceived probability of extinction due to nuclear technology. Other existential risks considered by the forecasters in Karger et al.’s study were estimated to be even lower than for nuclear technology, placing AI at the top of the “most probable” list of existential risks among experts and competent generalists (i.e., superforecasters).