It's the biggest impact. Regulation fails. We don’t need intent.

The existential risks posed by artificial intelligence (AI) are now widely recognized.

After hundreds of industry and science leaders warned that “mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war,” the U.N. Secretary-General recently echoed their concern. So did the prime minister of the U.K., who is also investing 100 million pounds into AI safety research that is mostly meant to prevent existential risk. Other leaders are likely to follow in recognizing AI’s ultimate threat.

In the scientific field of existential risk, which studies the most likely causes of human extinction, AI is consistently ranked at the top of the list. In The Precipice, a book by Oxford existential risk researcher Toby Ord that aims to quantify human extinction risks, the likeliness of AI leading to human extinction exceeds that of climate change, pandemics, asteroid strikes, supervolcanoes, and nuclear war combined. One would expect that even for severe global problems, the risk that they lead to full human extinction is relatively small, and this is indeed true for most of the above risks. AI, however, may cause human extinction if only a few conditions are met. Among them is human-level AI, defined as an AI that can perform a broad range of cognitive tasks at least as well as we can. Studies outlining these ideas were previously known, but new AI breakthroughs have underlined their urgency: AI may be getting close to human level already.

Recursive self-improvement is one of the reasons why existential-risk academics think human-level AI is so dangerous. Because human-level AI could do almost all tasks at our level, and since doing AI research is one of those tasks, advanced AI should therefore be able to improve the state of AI. Constantly improving AI would create a positive feedback loop with no scientifically established limits: an intelligence explosion. The endpoint of this intelligence explosion could be a superintelligence: a godlike AI that outsmarts us the way humans often outsmart insects. We would be no match for it.

A godlike, superintelligent AI

A superintelligent AI could therefore likely execute any goal it is given. Such a goal would be initially introduced by humans, but might come from a malicious actor, or not have been thought through carefully, or might get corrupted during training or deployment. If the resulting goal conflicts with what is in the best interest of humanity, a superintelligence would aim to execute it regardless. To do so, it could first hack large parts of the internet and then use any hardware connected to it. Or it could use its intelligence to construct narratives that are extremely convincing to us. Combined with hacked access to our social media timelines, it could create a fake reality on a massive scale. As Yuval Harari recently put it: “If we are not careful, we might be trapped behind a curtain of illusions, which we could not tear away—or even realise is there.” As a third option, after either legally making money or hacking our financial system, a superintelligence could simply pay us to perform any actions it needs from us. And these are just some of the strategies a superintelligent AI could use in order to achieve its goals. There are likely many more. Like playing chess against grandmaster Magnus Carlsen, we cannot predict the moves he will play, but we can predict the outcome: we lose.

But why would a superintelligence want to make humanity extinct? In fact, it might not explicitly want that at all. Humanity’s extinction might be a mere side effect of executing another goal to its extremity. When we are executing our own goals as humans, such as food production or growing our economy, we tend to not pay much attention to the effects this has on other species. This routinely leads to the extinction of animal species as a mere side effect of us maximising our goals. In a similar fashion, an uncontrollable superintelligence could lead to our extinction as a side effect of the AI pushing its goals, whatever they may be, to their limits.

So how can we avoid AI existential risk? Humanity now faces a stark choice: pause AI at a safe level, or continue development indefinitely. Unsurprisingly, the AI labs want the latter. To continue, however, means literally betting our lives on these labs being able to solve an open scientific problem called AI Alignment.

AI Alignment is the approach that leading labs such as OpenAI, DeepMind, and Anthropic are currently taking to prevent human extinction. AI Alignment does not attempt to control how powerful an AI gets, does not attempt to control what exactly the AI will be doing, and does not even necessarily attempt to prevent a potential takeover from happening. Since the labs anticipate they will not be able to control the superintelligent AI that they are creating, they accept that such an AI will act autonomously and unstoppably, but aim to make it act according to our values. As Richard Ngo, a leading alignment researcher working at OpenAI, puts it, “I doubt we’ll be able to make every single deployment environment secure against the full intellectual powers of a superintelligent AGI.” There are large problems with their alignment approach, however, both fundamentally and technically.