Containment and oversight now—fear of AI risk growing

DeepMind founder Demis Hassabis, in a recent interview about the promise and perils of AI, offered a note of caution. “I think a lot of times, especially in Silicon Valley, there’s this sort of hacker mentality of like ‘We’ll just hack it and put it out there and then see what happens.’ And I think that’s exactly the wrong approach for technologies as impactful and potentially powerful as AI. … I think it’s going to be the most beneficial thing ever to humanity, things like curing diseases, helping with climate, all of this stuff. But it’s a dual-use technology — it depends on how, as a society, we decide to deploy it — and what we use it for.”

Other leading AI labs are simply skeptical of the idea that there’s anything to worry about at all. Yann LeCun, the head of Facebook/Meta’s AI team, recently published a paper describing his preferred approach to building machines that can “reason and plan” and “learn as efficiently as humans and animals.” He has argued in Scientific American that Turing, Good, and Hawking’s concerns are no real worry: “Why would a sentient AI want to take over the world? It wouldn’t.”

But while divides remain over what to expect from AI — and even many leading experts are highly uncertain — there’s a growing consensus that things could go really, really badly. In a summer 2022 survey of machine learning researchers, the median respondent thought that AI was more likely to be good than bad but had a genuine risk of being catastrophic. Forty-eight percent of respondents said they thought there was a 10 percent or greater chance that the effects of AI would be “extremely bad (e.g., human extinction).”

It’s worth pausing on that for a moment. Nearly half of the smartest people working on AI believe there is a 1 in 10 chance or greater that their life’s work could end up contributing to the annihilation of humanity.

It might seem bizarre, given the stakes, that the industry has been basically left to self-regulate. If nearly half of researchers say there’s a 10 percent chance their work will lead to human extinction, why is it proceeding practically without oversight? It’s not legal for a tech company to build a nuclear weapon on its own. But private companies are building systems that they themselves acknowledge will likely become much more dangerous than nuclear weapons.

The problem is that progress in AI has happened extraordinarily fast, leaving regulators behind the ball. The regulation that might be most helpful — slowing down the development of extremely powerful new systems — would be incredibly unpopular with Big Tech, and it’s not clear what the best regulations short of that are.

Furthermore, while a growing share of ML researchers — 69 percent in the above survey — think that more attention should be paid to AI safety, that position isn’t unanimous. In an interesting, if somewhat unfortunate dynamic, people who think that AI will never be powerful have often ended up allied with tech companies against AI safety work and AI safety regulations: the former opposing regulations because they think it’s pointless and the latter because they think it’ll slow them down.

At the same time, many in Washington are worried that slowing down US AI progress could enable China to get there first, a Cold War mentality which isn’t entirely unjustified — China is certainly pursuing powerful AI systems, and its leadership is actively engaged in human rights abuses — but which puts us at very serious risk of rushing systems into production that are pursuing their own goals without our knowledge.

But as the potential of AI grows, the perils are becoming much harder to ignore. Former Google executive Mo Gawdat tells the story of how he became concerned about general AI like this: robotics researchers had been working on an AI that could pick up a ball. After many failures, the AI grabbed the ball and held it up to the researchers, eerily humanlike. “And I suddenly realized this is really scary,” Gawdat said. “It completely froze me. … The reality is we’re creating God.”

For me, the moment of realization — that this is something different, this is unlike emerging technologies we’ve seen before — came from talking with GPT-3, telling it to answer the questions as an extremely intelligent and thoughtful person, and watching its responses immediately improve in quality.

For Blake Lemoine, the eccentric Google engineer who turned whistleblower when he came to believe Google’s LaMDA language model was sentient, it was when LaMDA started talking about rights and personhood. For some people, it’s the chatbot Replika, whose customer service representatives are sick of hearing that the customers think their Replika is alive and sentient. For others, that moment might come from DALL-E or Stable Diffusion, or the systems released next year, or next month, or next week that are more powerful than any of these.

For a long time, AI safety faced the difficulty of being a research field about a far-off problem, which is why only a small number of researchers were even trying to figure out how to make it safe. Now, it has the opposite problem: The challenge is here, and it’s just not clear if we’ll solve it in time.