Even the top AI labs admit that they can’t secure everything. And, they can’t solve inner misalignment.

So how can we avoid AI existential risk? Humanity now faces a stark choice: pause AI at a safe level, or continue development indefinitely. Unsurprisingly, the AI labs want the latter. To continue, however, means literally betting our lives on these labs being able to solve an open scientific problem called AI Alignment.

AI Alignment is the approach that leading labs such as OpenAI, DeepMind, and Anthropic are currently taking to prevent human extinction. AI Alignment does not attempt to control how powerful an AI gets, does not attempt to control what exactly the AI will be doing, and does not even necessarily attempt to prevent a potential takeover from happening. Since the labs anticipate they will not be able to control the superintelligent AI that they are creating, they accept that such an AI will act autonomously and unstoppably, but aim to make it act according to our values. As Richard Ngo, a leading alignment researcher working at OpenAI, puts it, “I doubt we’ll be able to make every single deployment environment secure against the full intellectual powers of a superintelligent AGI.” There are large problems with their alignment approach, however, both fundamentally and technically.

What values should an AI have?

Fundamentally, there is of course no agreement on what values we have. What OpenAI founder Sam Altman wants a superintelligence to do may well be completely different from what, say, a textile worker in Bangladesh wants it to do. Aggregating human preferences has always been a thorny issue even before AI, but will get even more complicated when we have a superintelligence around that might be able to quickly, completely, and globally implement the wishes of a handful of CEOs. Democracy, however imperfect, has been the best solution we can find for value aggregation so far. It is, however, unclear whether our current combination of national political systems can control a godlike AI. In fact, it is unclear whether any organisation or system would be up to the task.

Beyond disagreement over current values, humans have historically not been very good at predicting the future externalities of new technologies. The climate crisis, for one, can be seen as an unforeseen outcome of technologies such as steam and internal combustion engines. Ever more powerful technology has the potential to create ever larger externalities. It is impossible to foresee what the negative side effects will be of actually implementing some interpretation of humanity’s current values. They might make climate change pale in comparison.

There are more unsolved fundamental issues with aligning a superintelligence, but currently, AI companies don’t even get that far. It seems that even making current large neural networks reliably do what anyone wants is a technical problem that we currently cannot solve. This is called inner misalignment and has been observed already in current AI: the model pursues the wrong goal when it is released into the real world after being trained on a limited dataset. For a superintelligence, this could have a catastrophic outcome. This problem is, according to OpenAI’s recent document Governance of Superintelligence, an “open research question.” But not to worry: they say they are “putting a lot of effort” into it.