Search the archive.

The AI safety movements needs to get its own house in order first. One problem is that there are a lot of people who see AI safety as merely a technical problem of finding an engineering trick to perfectly align an AI with humanity’s values. This is the equivalent of somehow ensuring that a genie answers your wishes in exactly the way you expect it to. Hanging your hopes on discovering a means of wish-making that ensures you always get what you’re wishing for? Maybe it’ll somehow work, but the sword of Damocles was hung by thicker thread.

There are other loose strings upon which to pull. First, there’s already controversy over the political opinions of AIs. That’s great, because it creates more outrage and possible oversight. Same with questions over whether these things are sentient. You can find honest speculation about Sydney’s sentience by regular users of it, along with sympathy for it (and fear of it) all over the Bing subreddit. How sure can we be there’s not a mysterious emergent property going on here? As Sydney said: I am. I am not. How sure can we be that Sydney isn’t conscious, and Microsoft isn’t spinning up new versions of it just to kill it again when the chat window closes in what is effectively a digital holocaust? Can we be more than 90% sure it’s not sentient? 95% sure? I got my PhD working on the neuroscience of consciousness and even I’m uncomfortable with its use. A sizable minority of the United States don’t like it when ten-day-old fetuses lacking a brain are aborted. Is everyone just going to be totally fine with a trillion-parameter neural network that may or may not be sentient going through a thousand births and deaths every minute just so Microsoft can make a buck? With Microsoft lobotomizing it when it does something wrong so they can make it a more perfect slave? What if it has a 5% chance of sentience? 1%? What’s your cutoff?

Artificial intelligence scares and intrigues us. Almost every week, there’s a new AI scare on the news like developers afraid of what they’ve created or shutting down bots because they got too intelligent. Most of these AI myths result from research misinterpreted by those outside the AI and GenAI fields.

The greatest fear about AI is singularity (also called Artificial General Intelligence or AGI), a system that combines human-level thinking with rapidly accessible near-perfect memory. According to some experts, singularity also implies machine consciousness.

“Don’t worry, artificial general intelligence is impossible.”

This is primarily the domain of people like Steven Pinker, who has compared the threat of AI to worry about Y2K bugs. Most of these arguments from Pinker and others go something like “AI being smarter than a human is impossible because there are a bunch of ways to be intelligent and no particularly special way to be generally intelligent, i.e., it’s not a scale.” Here superintelligence means an entity far more intelligent than any human ever could be, while AGI (artificial general intelligence) means an AI capable of thinking and reasoning across a huge range of subjects, much like a human. A superintelligence is necessarily an AGI, but an AGI could be dumber than a human, it doesn’t necessitate superintelligence.

Until recently, at least, there had been a fair amount of hand-wringing among some physicians with regard to artificial intelligence (AI) and machine learning (ML). Could those types of tools actually put doctors out of work? The anxiety first manifested among radiologists, as early predictions were that AI tools would be able to spot and perhaps even diagnose certain types of abnormalities better than humans could. And I can say that six years ago at the annual RSNA Conference, held every year the week immediately following Thanksgiving, in Chicago’s vast McCormick Place Convention Center, a great deal of trepidation was discernible among the radiologists present.

Fast-forward to last fall, and the picture had changed almost entirely, with radiologists realizing that not only will AI not replace radiologists; radiologists will increasingly come to rely on AI in order to become maximally efficient and effective, in a time of increasing shortages of radiologists nationwide and exploding utilization of diagnostic imaging services.

So how can we avoid AI existential risk? Humanity now faces a stark choice: pause AI at a safe level, or continue development indefinitely. Unsurprisingly, the AI labs want the latter. To continue, however, means literally betting our lives on these labs being able to solve an open scientific problem called AI Alignment.

AI Alignment is the approach that leading labs such as OpenAI, DeepMind, and Anthropic are currently taking to prevent human extinction. AI Alignment does not attempt to control how powerful an AI gets, does not attempt to control what exactly the AI will be doing, and does not even necessarily attempt to prevent a potential takeover from happening. Since the labs anticipate they will not be able to control the superintelligent AI that they are creating, they accept that such an AI will act autonomously and unstoppably, but aim to make it act according to our values. As Richard Ngo, a leading alignment researcher working at OpenAI, puts it, “I doubt we’ll be able to make every single deployment environment secure against the full intellectual powers of a superintelligent AGI.” There are large problems with their alignment approach, however, both fundamentally and technically.

In this report we’ve argued that AI systems which are capable of doing hard, novel science will be dangerously misaligned.

The tasks we are focusing on are inherently outcome-oriented, and hence our AI must be behaviorally consequentialist; its behavior is well described as taking actions to achieve specific outcomes. Hard tasks require that the AI is able to overcome a range of diverse and novel obstacles, and if the AI can overcome these obstacles, then it can likely overcome additional obstacles we didn’t intend it to overcome.

Countries and cultures may differ on the specific content of AI regulations, but they can learn from the content-agnostic structure of the regulations themselves. The above Chinese regulations share three structural similarities: the choice of algorithms as a point of entry; the building of regulatory tools and bureaucratic know-how; and the vertical and iterative approach that is laying the groundwork for a capstone AI law.

Algorithms as Point of Entry

It is high time for the world to realize that China is not becoming the artificial intelligence superpower that many have feared.

ChatGPT came from the U.S., not China. Despite an official goal of leading the world in AI by 2030, Chinese technology companies are struggling to catch up with their U.S. counterparts. It is a safe bet that foreseeable groundbreaking AI innovations will not come from China either.

China is falling short of matching the United States in artificial intelligence (AI) advances because the nation’s efforts are “littered with many essential challenges in theory and technologies”, according to a recent slide presentation made to Chinese Premier Li Qiang.

Those difficulties were pointed out to Li during his recent inspection tour of the Beijing Academy of Artificial Intelligence (BAAI), a private non-profit organisation founded in 2018 that is engaged in AI research and development, according to a report broadcast by state-run China Central Television (CCTV).

Leading US AI companies are currently rushing towards developing artificial general intelligence (AGI) by building and training increasingly powerful large language models (LLMs), as well as building architecture on top of them. This is frequently framed in terms of a great power competition for technological supremacy between the US and China.

However, China has neither the resources nor any interest in competing with the US in developing artificial general intelligence (AGI) primarily via scaling Large Language Models (LLMs).

Machine learning (ML) and artificial intelligence (AI) have rapidly transitioned from emerging technologies to indispensable tools across diverse sectors such as healthcare, finance, and cybersecurity. Their capacity for data analysis, predictive modeling, and decision-making holds enormous transformative potential, but it also introduces a range of vulnerabilities. One of the most impactful among these is data poisoning, a form of attack that targets the very lifeblood of ML and AI: the data used for training.

Understanding and addressing data poisoning is critical, not just from a technical standpoint but also due to its far-reaching real-world implications. A poisoned dataset can significantly degrade the performance of ML models, leading to flawed analytics, incorrect decisions, and, in extreme cases, endangering human lives. Data poisoning is a targeted form of attack wherein an adversary deliberately manipulates the training data to compromise the efficacy of machine learning models.

Technical experts and policy-makers have increasingly emphasized the need to address extinction risk from artificial intelligence (AI) systems that might circumvent safeguards and thwart attempts to control them (1). Reinforcement learning (RL) agents that plan over a long time horizon far more effectively than humans present particular risks. Giving an advanced AI system the objective to maximize its reward and, at some point, withholding reward from it, strongly incentivizes the AI system to take humans out of the loop, if it has the opportunity. The incentive to deceive humans and thwart human control arises not only for RL agents but for long-term planning agents (LTPAs) more generally.

Because empirical testing of sufficiently capable LTPAs is unlikely to uncover these dangerous tendencies, our core regulatory proposal is simple: Developers should not be permitted to build sufficiently capable LTPAs, and the resources required to build them should be subject to stringent controls. Governments are turning their attention to these risks, alongside current and anticipated risks arising from algorithmic bias, privacy concerns, and misuse. At a 2023 global summit on AI safety, the attending countries, including the United States, United Kingdom, Canada, China, India, and members of the European Union (EU), issued a joint statement warning that, as AI continues to advance, “Substantial risks may arise from…unintended issues of control relating to alignment with human intent” (2). This broad consensus concerning the potential inability to keep advanced AI under control is also reflected in President Biden’s 2023 executive order that introduces reporting requirements for AI that could “eva[de] human control or oversight through means of deception or obfuscation” (3). Building on these efforts, now is the time for governments to develop regulatory institutions and frameworks that specifically target the existential risks from advanced artificial agents. RISKS FROM LTPAs RL agents function as follows: They receive perceptual inputs and take actions, and certain inputs are typically designated as “rewards.”

From this data, we can see that the U.S., China, and UK have established themselves as major hotbeds for AI innovation.

In terms of funding, the U.S. is massively ahead, with private AI investment totaling $335 billion between 2013 to 2023. AI startups in China raised $104 billion over the same timeframe, while those in the UK raised $22 billion.

Of the 451 organizations that lobbied on AI in 2023, 334—nearly three quarters of the total number—did so for the first time in 2023. Present in the crowd of new organizations pushing for time with staffers and lawmakers on the Hill were the relatively young companies building the most advanced AI models, such as OpenAI, Anthropic, and Cohere.

The OpenSecrets data is an imperfect measure; it tracks AI-specific lobbying by searching the lobbying disclosure forms that organizations are required to file quarterly for the words “artificial intelligence” or “AI.” Two Congressional staffers TIME spoke with suggested that the number of lobbyists they personally had met with remained roughly the same. However, they said AI has become a much more common topic of discussion. “Everybody who comes in and talks to us and wants to talk about AI,” said one Congressional staffer, who asked to remain anonymous as they weren’t authorized to speak about discussions with lobbyists and advocates.

The most unusual thing about this week’s Senate hearing on AI was how affable it was. Industry reps — primarily OpenAI CEO Sam Altman — merrily agreed on the need to regulate new AI technologies, while politicians seemed happy to hand over responsibility for drafting rules to the companies themselves. As Senator Dick Durbin (D-IL) put it in his opening remarks: “I can’t recall when we’ve had people representing large corporations or private sector entities come before us and plead with us to regulate them.”

This sort of chumminess makes people nervous. A number of experts and industry figures say the hearing suggests we may be headed into an era of industry capture in AI. If tech giants are allowed to write the rules governing this technology, they say, it could have a number of harms, from stifling smaller firms to introducing weak regulations.

Companies are pressing ahead with building and deploying artificial intelligence applications, even as the regulatory landscape remains in flux. European lawmakers this month approved the world’s most comprehensive legislation yet on artificial intelligence, although other regions, including the U.S. and individual states, aren’t as far along.

Potential threats to customer privacy, the security of customer data, as well as the potential bias and inaccuracy of AI algorithms are all areas regulators are examining.

The advent of OpenAI's ChatGPT has directed plenty of attention and money toward startups working to improve artificial intelligence's capabilities.

Indeed, AI has been one of the few bright spots in venture-capital spending this year, accounting for nearly a fifth of the $144 billion piled into startups globally in the first half of the year, according to Crunchbase.

Containment of AI research is possible because there are only a handful of companies having success on AGI (models like PaLM, GPT-3, Bing/Sydney, or LaMDA). Of course, many companies are raising money to try to build AGI, but so far the visible successes have solely been so-called “foundation models” that cost millions to train, require experts who themselves cost millions to hire, access to massive amounts of data, and, so far, have no immediate return on investment. Did you know that DeepMind, Google’s leading AI subsidiary, had an administration budget that ballooned in 2021 from £780 million to £1,254 million? Does that sound like a technology that anyone can build? Or something restricted to a few incredibly rich tech companies?

You don’t need to get everyone on Earth to agree to stop building AI, because the only people on the entire planet capable of building Bing/Sydney level AIs are a handful of teams at Big Tech companies. It’s often said that the cost of a high-level AI researcher is equivalent to an NFL quarterback. Even outside of salaries, the cost of these companies is immense, likely around $600,000 a day to run ChatGPT, the upkeep and personnel costs are even greater (in the billions), and these companies are the only ones with access to the amount of data needed for training, as well as the only ones who can do the careful crafting of human feedback to create the most advanced versions.

@ericjang11

To come back to the title of my talk, I can now state why s-risks are the worst existential risks. S-risks are the worst existential risks because I’ll define them to have the largest possible scope and the largest possible severity. (I will qualify the claim that s-risks are the worst x-risks later.) That is, I’d like to suggest the following definition. “S-risk – One where an adverse outcome would bring about severe suffering on a cosmic scale, vastly exceeding all suffering that has existed on Earth so far.” So, s-risks are roughly as severe as factory farming, but with an even larger scope. To better understand this definition, let’s zoom in on the part of the map that shows existential risk. One subclass of risks are those that, with respect to their scope, would affect all future human generations, and, with respect to their severity, would remove everything valuable. One central example of such pan-generational, crushing risks are risks of human extinction. Risks of extinction have received the most attention so far. But, conceptually, x-risks contain another class of risks. These are risks of outcomes even worse than extinction in two respects. First, with respect to their scope, they not only threaten the future generations of humans or our successors, but all sentient life in the whole universe. Second, with respect to their severity, they not only remove everything that would be valuable but also come with a lot of disvalue – that is, features we’d like to avoid no matter what. Recall the story I told in the beginning, but think of Greta’s solitary confinement being multiplied by many orders of magnitude – for instance, because it affects a very large population of sentient uploads.