Upside and downside AI impacts are the most probable x-risks.

Ilan Noy and Tomáš Uher, “Four New Horsemen of an Apocalypse? Solar Flares, Super-volcanoes, Pandemics, and Artificial Intelligence,” Economics of Disasters and Climate Change, https://link.springer.com/article/10.1007/s41885-022-00105-x

Misaligned Artificial Intelligence

Artificial intelligence (AI) refers to computer programs or machines which exhibit behaviour humans would perceive to be intelligent (Kaplan 2016). The significant scientific progress of the field in the last decades has caused many to wonder about the future of this technology and its societal implications. AI may create significant risks, depending on what kinds of systems we create and how we deploy them.

With respect to advances in the field of AI, considerable improvements have been made within specific narrow domains of intelligence (narrow AI) such as computer vision or natural language processing. Notable progress has also been made towards achieving a more general kind of intelligence, such that may eventually be able to employ multiple cognitive abilities, abstraction, and reasoning, as is typical for humans (Strogatz 2018; Badia et al. 2020; Silver et al. 2018). A system with this general reasoning ability is termed artificial general intelligence (AGI).

Broadly speaking, the use of AI can lead to harmful outcomes either if the AI is programmed to achieve a harmful goal, or if the AI is programmed to achieve a beneficial goal but employs a harmful method for achieving it (Future of Life Institute, n.d.; Turchin and Denkenberger 2018a). The latter case is especially relevant for AGI, as it is argued that application of such systems could lead to catastrophic outcomes without any bad intentions or development of harmful methods by its creators (Omohundro 2008).

The development of machines that may potentially become smarter and more powerful than humans could mark the end of an era characterized by humanity’s control of its future (Russell 2019). If such a powerful agent does not share our values, the result could be catastrophic (Bostrom 2014; Ord 2020). To prevent the potential disastrous outcomes of future AI, researchers argue it is crucial to align the value and motivation systems of AI systems with human values, a task that is referred to as the alignment problem (Bostrom 2014; Yudkowsky 2016; Critch and Krueger 2020). However, objectively formulating and programming human values into a computer is a complicated task. At present, we do not seem to know how to do it.

Predicting the potential catastrophic impacts of AI is made difficult by several factors. Firstly, the risk posed by AI is unprecedented and cannot be reliably assessed using historical data and extrapolation, unlike the other types of existential risks explored in this review (space weather, super-volcanoes, and pandemics). Secondly, with respect to general and super-intelligence, it may be practically or even inherently impossible for us to predict how a system more intelligent than us will act (Yampolskiy 2020). Considering the difficulty of predicting catastrophic societal impacts associated with AI, we are limited here to hypothetical scenarios of basic pathways describing how the disastrous outcomes could be manifested.

When considering potential global catastrophic or existential risks stemming from AI, it is useful to distinguish between narrow AI and AGI, as the speculated possible outcomes associated with each type can differ greatly. For narrow AI systems to cause catastrophic outcomes, the potential scenarios include events such as software viruses affecting hardware or critical infrastructure globally, AI systems serving as weapons of mass destruction (such as slaughter-bots), or AI-caused biotechnological or nuclear catastrophe (Turchin and Denkenberger 2018a, b; Tegmark 2017; Freitas 2000). Interestingly, Turchin and Denkenberger (2018a) argue that the catastrophic risks stemming from narrow AI are relatively neglected despite their potential to materialize sooner than the risks from AGI. Still, the probability of narrow AI to cause an existential catastrophe appears to be relatively lower than in the case of AGI (Ord 2020).

With respect to the global catastrophic and existential risk of misaligned AGI, much of the expected risk seems to lie in an AGI system’s extraordinary ability to pursue its goals. According to Bostrom’s instrumental convergence thesis, instrumental aims such as developing more resources and/or power for gaining control over humans would be beneficial for achieving almost any final goal the AGI system might have (Bostrom 2012). Hence, it can be argued that almost any misaligned AGI system would be motivated to gain control over humans (often described in the literature as a ‘decisive strategic advantage’) to eliminate the possibility of human interference with the system’s pursuit of its goals (Bostrom 2014; Russell 2019). Once humans have been controlled, the system would be free to pursue its main goal, whatever that might be.

In line with this instrumental convergence thesis, an AGI system whose values are not perfectly aligned with human values would be likely to pursue harmful instrumental goals, including seizing control and thus potentially creating catastrophic outcomes for humanity (Russell and Norvig 2016; Bostrom 2002, 2003a; Taylor et al. 2016; Urban 2015; Ord 2020; Muehlhauser 2014). A popular example of such a scenario is the paperclip maximizer, which firstly appeared in a mailing list of AI researchers in the early 2000’s (Harris 2018); a later version is included in Bostrom (2003a). Most versions of this scenario involve an AGI system with an arbitrary goal of manufacturing paperclips. In pursuit of this goal, it will inevitably transform Earth into a giant paperclip factory and therefore destroy all life on it. There are other scenarios that end up with potential extinction. Ord (2020) presents one in which the system increases its computational resources by hacking other systems, which enables it to gain financial and human resources to further increase its power in pursuit of its defined goal. 6

Another possible scenario with respect to the future of AI technology is AI merging with humans and creating artificially enhanced beings. This may indeed be a likely future outcome, considering that the development of these emerging technologies is already ongoing. The potential for this to create globally catastrophic outcomes is relatively less explored. We have not been able to identify any studies focusing on it. Could this be a path that may eventually lead us to great harm?

The hypothetical scenarios discussed here demonstrate the possible pathways through which development and use of AI technology could lead to global or existential catastrophes. However, they do not inform us about concrete societal effects associated with such outcomes, or much on how one can invest in prevention. The catastrophic risk of AI appears to be so speculative and unpredictable that we were not able to identify any estimates of more specific socio-economic impacts of this technology, or any associated risk reduction costs and benefits.

Estimating the likelihood of a catastrophic AI scenario is equally fraught (Armstrong and Sotala 2015). At present, estimates are predominantly based on expert opinion surveys. In these, experts estimate that the probability that an AI system will be capable of performing most human tasks by 2040–2050 to be around 50% (Müller and Bostrom 2016). Similarly, a survey of Grace et al. (2018) suggests a 50% probability of AI systems outperforming humans in all tasks by 2063. The respondents in Müller and Bostrom (2016) predict that such systems would become super-intelligent less than 30 years after their development, with an estimated chance of one in three that machine super-intelligence will be bad or extremely bad for humanity. A survey described in Turchin (2019) suggests that there is an 10% chance of AGI appearing relatively early in the next 10–15 years. By combining this polling with extrapolations of current technological progress, Turchin (2019) estimates that AI capable of causing global catastrophic outcomes (narrow AI included) could be possible as early as 2030–2040.

When questioned directly about existential risk from AI, an informal survey of participants in the Global Catastrophic Risk Conference in Oxford in 2008 indicated that the risk of human extinction due to a super-intelligent AI before 2100 may be as high as 5%; the highest probability out of all those considered in that event (Sandberg and Bostrom 2008). Ord (2020) puts the probability of an AI-caused existential catastrophe even higher at 10% within the next 100 years. Sandberg and Bostrom (2008), Turchin and Denkenberger (2018c) and Ord (2020) all suggest that the existential risk probability of AI is the highest out of the risks described herein.

Conclusion

The previous section argued that Artificial Intelligence (AI) systems most likely pose the highest global catastrophic and existential risk to humanity from the four risks we described here, including solar-flares and space weather, engineered and natural pandemics, and super-volcanic eruptions. While the risk assessments are difficult for all these four hazards, the difficulty of predicting the potential disastrous dynamics of AI is most acute, even as the risk is also likely the highest.

AI may also be very relevant for the other existential risks we described here. Counter-intuitively, maybe, AI may potentially assist us in decreasing the risk of other global catastrophic and existential events such as those associated with a nuclear war, an engineered pandemic, or a solar-flare (Liu et al. 2018; Akiyama 2021; Boulanin 2019; Bostrom 2002). There are undeniably tremendous benefits in well-aligned AI systems in general, but also particularly in terms of their capabilities to reduce existential and other risks (Bostrom 2003b; Yudkowsky 2008) or increase our resilience to these shocks.7 We should note that AI systems pose other less extreme risks, and some observers view these are more concerning than the extinction risk we focus on here (e.g., Acemoglu 2021; Lu and Zhou 2021).