Regulation determines alignment---current efforts are insufficient.

Technical experts and policy-makers have increasingly emphasized the need to address extinction risk from artificial intelligence (AI) systems that might circumvent safeguards and thwart attempts to control them (1). Reinforcement learning (RL) agents that plan over a long time horizon far more effectively than humans present particular risks. Giving an advanced AI system the objective to maximize its reward and, at some point, withholding reward from it, strongly incentivizes the AI system to take humans out of the loop, if it has the opportunity. The incentive to deceive humans and thwart human control arises not only for RL agents but for long-term planning agents (LTPAs) more generally.

Because empirical testing of sufficiently capable LTPAs is unlikely to uncover these dangerous tendencies, our core regulatory proposal is simple: Developers should not be permitted to build sufficiently capable LTPAs, and the resources required to build them should be subject to stringent controls. Governments are turning their attention to these risks, alongside current and anticipated risks arising from algorithmic bias, privacy concerns, and misuse. At a 2023 global summit on AI safety, the attending countries, including the United States, United Kingdom, Canada, China, India, and members of the European Union (EU), issued a joint statement warning that, as AI continues to advance, “Substantial risks may arise from…unintended issues of control relating to alignment with human intent” (2). This broad consensus concerning the potential inability to keep advanced AI under control is also reflected in President Biden’s 2023 executive order that introduces reporting requirements for AI that could “eva[de] human control or oversight through means of deception or obfuscation” (3). Building on these efforts, now is the time for governments to develop regulatory institutions and frameworks that specifically target the existential risks from advanced artificial agents. RISKS FROM LTPAs RL agents function as follows: They receive perceptual inputs and take actions, and certain inputs are typically designated as “rewards.”

An RL agent then aims to select actions that it expects will lead to higher rewards. For example, by designating money as a reward, one could train an RL agent to maximize profit on an online retail platform (4). Highly capable and far-sighted RL agents are likely to accrue reward very successfully. If plan A leads to more expected reward than plan B, sufficiently advanced RL agents would favor the former. Crucially, securing the ongoing receipt of maximal rewards with very high probability would require the agent to achieve extensive control over its environment, which could have catastrophic consequences (5–8). One path to maximizing long-term reward involves an RL agent acquiring extensive resources and taking control over all human infrastructure (5, 6), which would allow it to manipulate its own reward free from human interference (5). Additionally, because being shut down by humans would reduce the expected reward, sufficiently capable and far-sighted agents are likely to take steps to preclude that possibility (7) or if feasible, create new agents (unimpeded by monitoring or shutdown) to act on their behalf (5). Progress in AI could enable such advanced behavior. So long as an agent’s rewards can be controlled, it can be incentivized to achieve complex goals by conditioning the rewards appropriately. But a sufficiently capable RL agent could take control of its rewards, which would give it the incentive to secure maximal reward single-mindedly. Constraining the influence that highly competent agents learn to exert over their environment is likely to prove extremely difficult; an intelligent agent could, for example, persuade or pay unwitting human actors to execute important actions on its behalf (5, 7). Critically, far-sighted RL agents face an incentive to develop and execute arbitrarily competent long-term plans. Many AI systems are trained only to achieve certain immediate outcomes, like correctly classifying an image.

Although such short-sighted agents could certainly cause harm, they would likely lack the incentive to execute protracted schemes to subvert human control. Accordingly, we define an LTPA as an algorithm designed to produce plans, and to prefer plan A to plan B, when it expects that plan A is more conducive to a given goal over a long time horizon. For example, an agent trained to maximize profit on an online retail platform, as proposed by Suleyman’s “new Turing test” (4), might productively use such an algorithm and hinder attempts to interfere with its profit making. LTPAs include all long-horizon RL algorithms, including so-called “policy gradient” methods, which lack an explicit planning subroutine but are trained to be as competent as possible. LTPAs also include algorithms that imitate trained LTPAs, but not algorithms that merely imitate humans. In the latter case, if plan A is more competent than any plan a human could develop, and plan B is a human plan, an algorithm imitating a human would not prefer plan A to plan B. The supplementary materials include a taxonomy situating LTPAs among other machine learning systems. Notably, there is no recognizable horizon length at which risk increases sharply; accordingly, regulators will have to define the length of a long time horizon according to their risk tolerance. Losing control of advanced LTPAs, although not the only existential risk from AI, is the class of risk that we aim to address here—and one that necessitates new forms of government intervention. A GOVERNANCE PROPOSAL Although governments have expressed concern about existential risks from AI, regulatory proposals do not adequately address this class of risk (9). The EU AI Act (10) canvasses a broad array of risks from AI but does not single out loss of control of advanced LTPAs. We see promising first steps from the US and UK—President Biden’s executive order on AI (3) requires reports on potentially uncontrollable AI systems, but it does not seek to constrain their development or proliferation; the US and UK AI Safety Institutes are building capacity for regulators to understand cutting-edge AI but lack the authority to control it (11).