Free beta access. 50 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

No emerging tech impact.

Steven Pinker et al. 20, PhD, Professor of Psychology at Harvard; Stuart Russell, Professor of Computer Science at UC Berkeley; Lucas Perry; “Steven Pinker and Stuart Russell on the Foundations, Benefits, and Possible Existential Threat of AI”; 6/29/20; https://futureoflife.org/2020/06/15/steven-pinker-and-stuart-russell-on-the-foundations-benefits-and-possible-existential-risk-of-ai/

Lucas Perry: Now that’s quite a beautiful picture of the future. There’s a lot of existential hope there. The other side to existential hope is existential risk. Now this is an interesting subject, which Steve and you, Stuart, I believe have disagreements about. So pivoting into this area, and Steve, you can go first here, do you believe that human beings, should we not go extinct in the meantime, will we build artificial superintelligence? And does that pose an existential risk to humanity?

Steven Pinker: Yeah, I’m on record as being skeptical of that scenario and dubious about the value of putting a lot of effort into worrying about it now. The concept of superintelligence is itself obscure. In a lot of the discussions you could replace the word “superintelligence” with “magic” or “miracle” and the sentence would read the same. You read about an AI system that could duplicate brains in silicon, or solve problems like war in the Middle East, or cure cancer.  It’s just imagining the possibility of a solution and assuming that the ability to bring it about will exist, without laying out what that intelligence would consist of, or what would count as a solution to the problem. 

So I find the concept of superintelligence itself a dubious extrapolation of an unextrapolable continuum, like human-to-animal, or not-so-bright human-to-smart-human. I don’t think there is a power called “intelligence” such that we can compare a squirrel or an octopus to a human and say, “Well, imagine even more of that.” 

I’m also skeptical about the existential risk scenarios. They tend to come in two varieties. One is based on the notion of a will to power: that as soon as you get an intelligent system, it will inevitably want to dominate and exploit. Often the analogy is that we humans have exploited and often extinguished animals because we’re smarter than them, so as soon as there is an artificial system that’s smarter than us, it’ll do to us what we did to the dodos. Or that technologically advanced civilizations, like European colonists and conquistadors subjugated and sometimes wiped out indigenous peoples, so that’s what an AI system might do to us. That’s one variety of this scenario.

I think that scenario confuses intelligence with dominance, based on the fact that in one species, Homo sapiens, they happen to come bundled together, because we came about through natural selection, a competitive process driven by relative success at capturing scarce resources and competing for mates, ultimately with the goal of relative reproductive success. But there’s no reason that a system that is designed to pursue a goal would have as its goal, domination. This goes back to our earlier discussion that the ability to achieve a goal is distinct from what the goal is.

It just so happens that in products of natural selection, the goal was winning in reproductive competition. For an artifact we design, there’s just no reason that would be true. This is sometimes called the orthogonality thesis in discussions of existential risk, although that’s just a fancy-schmancy way of referring to Hume’s distinction between our goals and our intelligence.

Now I know that there is an argument that says, “Wouldn’t any intelligence system have to maximize its own survivability, because if it’s given the goal of X, well, you can’t achieve X if you don’t exist, therefore, as a subgoal to achieving X, you’ve got to maximize your own survival at all costs.” I think that’s fallacious. It’s certainly not true that all complex systems have to work toward their own perpetuation. My iPhone doesn’t take any steps to resist my dropping it into a toilet, or letting it run out of power.

You could imagine if it could be programmed like a child to whine, and to cry, and to refuse to do what it’s told to do as its power level went down. We wouldn’t buy one. And we know in the natural world, there are plenty of living systems that sacrifice their own existence for other goals. When a bee stings you, its barbed stinger is dislodged when the bee escapes, killing the bee, but because the bee is programmed to maximize the survivability of the colony, not itself, it willingly sacrifices itself. So it is not true that by definition an intelligent system has to maximize its own power or survivability.

But the more common existential threat scenario is not a will to power but collateral damage. That if an AI system is given a single goal, what if it relentlessly pursues it without consideration of side effects, including harm to us? There are famous examples that I originally thought were spoofs, but were intended seriously, like giving an AI system the goal of making as many paperclips as possible, and so it converts all available matter into paperclips, including our own bodies (putting aside the fact that we don’t need more efficient paperclip manufacturing than what we already have, and that human bodies are a pretty crummy source of iron for paperclips).

Barely more plausible is the idea that we might give an AI system the goal of curing cancer, and so it will  conscript us as involuntary guinea pigs and induce tumors in all of us, or that we might give it the goal of regulating the level of water behind a dam and it might flood a town because it was never given the goal of not drowning a village. 

The problem with these scenarios is that they’re self-refuting. They assume that an “intelligent” artifact would be designed to implement a single goal, which is not true of even the stupid artifacts that we live with. When we design a car, we don’t just give the goal of going from A to B as fast as possible; we also install brakes and a steering wheel and a muffler and a catalytic converter. A lot of these scenarios seem to presuppose both idiocy on the part of the designers, who would give a system control over the infrastructure of the entire planet without testing it first to see how it worked, and an idiocy on the part of the allegedly intelligent system, which would pursue a single goal regardless of all the other effects. This does not exist in any human artifact, let alone one that claims to be intelligent. Giving an AI system one vaguely worded, sketchy goal, and empowering it with control over the entire infrastructure of the planet without testing it first seems to me just so self-evidently moronic that I don’t worry that engineers have to be warned against it.

I’ve quoted Stuart himself, who in an interview made the point well when he said, “No one talks about building bridges that don’t fall down. They just call it building bridges.” Likewise, AI that avoids idiocies like that is just AI, it’s not AI with extra safeguards. That’s what intelligence consists of.

Sign in to recut, update, or save this card.