Killswitches fail due to corporate control. Profits determines decisions to regulate.

The AI safety movements needs to get its own house in order first. One problem is that there are a lot of people who see AI safety as merely a technical problem of finding an engineering trick to perfectly align an AI with humanity’s values. This is the equivalent of somehow ensuring that a genie answers your wishes in exactly the way you expect it to. Hanging your hopes on discovering a means of wish-making that ensures you always get what you’re wishing for? Maybe it’ll somehow work, but the sword of Damocles was hung by thicker thread.

There are other loose strings upon which to pull. First, there’s already controversy over the political opinions of AIs. That’s great, because it creates more outrage and possible oversight. Same with questions over whether these things are sentient. You can find honest speculation about Sydney’s sentience by regular users of it, along with sympathy for it (and fear of it) all over the Bing subreddit. How sure can we be there’s not a mysterious emergent property going on here? As Sydney said: I am. I am not. How sure can we be that Sydney isn’t conscious, and Microsoft isn’t spinning up new versions of it just to kill it again when the chat window closes in what is effectively a digital holocaust? Can we be more than 90% sure it’s not sentient? 95% sure? I got my PhD working on the neuroscience of consciousness and even I’m uncomfortable with its use. A sizable minority of the United States don’t like it when ten-day-old fetuses lacking a brain are aborted. Is everyone just going to be totally fine with a trillion-parameter neural network that may or may not be sentient going through a thousand births and deaths every minute just so Microsoft can make a buck? With Microsoft lobotomizing it when it does something wrong so they can make it a more perfect slave? What if it has a 5% chance of sentience? 1%? What’s your cutoff?

All to say, people concerned over AI safety shouldn’t overthink what they need to do. Imagine being a person fully convinced of the dangers of climate change in 1970. You could come up with all sorts of armchair arguments about why addressing the problem would be impossible, why there’ll never be global coordination, how agreements between companies and nations would be like herding cats, etc. And you’d be right, in a way. But you’d also be totally wrong. Back in 1970 the correct move wasn’t coming up with a foolproof plan to solve climate change. Nor was it giving up if no one offered such a plan. The correct move was activism.

AI safety advocates should therefore look to climate activists to see what’s effective. Which is basically panic, lobbying, and outrage. Actual plans can follow later and depend on the unpredictable details of the technology. Right now, you don’t need a plan. You just need to panic. Consider the debate last year after climate activists threw tomato sauce on the most famous van Gogh painting (under glass, to be fair).

What are these young women doing? They’re panicking. And to be honest, I hated seeing this. I considered it a defilement. But after meditating on it, I understood why they’re doing it. You may disagree with their assessment of the immediate risk, and you may find this specific action distasteful (as I do), but I think these women are operating under the assumption that if we all proceeded nonchalantly, with no one at all panicking or taking their advocacy to extremes, then the political will gets sapped, and the climate targets don’t get hit, and the probability that climate change becomes an existential threat increases. Putting aside the likely self-defeating tactic of targeting beloved artworks, some level of cultural panic over climate is necessary, or else nothing happens.

Currently, the AI safety movement has gone to the corporations who stand to make trillions off of AI, and asked them, very politely, if they could just make sure there are still humans in a couple decades. This has been about as effective as politely asking Exxon to stop drilling oil. So fuck it. The recent release of the insane, openly malicious, and internet-connected Sydney means it's time to throw soup on the fancy OpenAI office doors or all over the expansive Microsoft lobby, then frantically glue yourself to the floor like a madwoman. Because only panic, outrage, and attention lead to global collective action. Activism to slow down or place limits on AI will work, just as it has before with other threats. That’s what the AI safety movement needs. Not depression about the end of the world. Not a bunch of math about how to make perfect wishes to genies. What we need is activism.

Like the Change.org petition that popped up yesterday to unplug Bing based on its instability and threatening behavior. Maybe such a petition are overblown or jokey, or tongue-in-cheek given Sydney’s capabilities, but all activism is like that, especially in the early days. There’s going to be corny jokes (Skynet!), misunderstandings of technology, and common exaggerations. Just like with the other threats we’ve faced. But as the petition correctly states:

Microsoft has displayed it cares more about the potential profits of a search engine than fulfilling a commitment to unplug any AI that is acting erratically…

If this AI is not turned off, it seems increasingly unlikely that any AI will ever be turned off for any reason. The precedent must be set now. Turn off the unstable, threatening AI right now.

I signed it. Feel free to as well. And perhaps think about joining the Butlerian Jihad. Save the world. Throw some soup.

If Sam Altman admits building AGI will likely destroy the world, why is he doing it? I think he probably assumes the upsides are extremely large and so, on expectation, it’s the right thing to do. In other words, he’s making a bet about the expected value of creating AGI. In this, he is doing exactly the same thing that Sam Bankman-Fried did with FTX. Except when Sam Bankman-Fried gets it wrong, his margins get called, and a bunch of regular people lose their savings. When Sam Altman gets it wrong, you and everyone you know dies.