It’s fast—recursion, hide abilities—risks a sneak attack

2. The dangers of unfriendly powerful artificial general intelligence

Unlike with whatever wetware runs the human brain, it would be relatively easy to make changes to a PAGI’s software. PAGI could even make changes to itself. Such self-modification could possibly allow PAGI to undergo an intelligence explosion where it figures out how to improve its own intelligence, then, as it gets smarter, it figures out new ways to improve its intelligence. It has been theorized that through recursive self-improvement a PAGI could go from being a bit smarter than humans to becoming a computer superintelligence in a matter of days (Good, 1965; Yudkowsky, 2008).

If our understanding of the laws of physics is correct, the universe contains a limited amount of free energy, and this free energy is necessary to do any kind of work and most types of computing. Consequently, it has been theorized that most types of computer superintelligences would have an instrumental goal of gathering as much free energy as possible to further whatever ultimate goals they had (Omohundro, 2008). Humanity’s continued existence uses free energy. Consequently, if a PAGI did not have promoting human welfare as a goal, it would likely see humanity’s continuing existence as rival to its terminal values. A PAGI that wanted to maximize its understanding of, say, chess would further this end by exterminating mankind and using the atoms in our bodies to make chess computing hardware. A PAGI that wanted to maximize the number of paperclips in the universe would likewise kill us, not out of malice, but to align the atoms in our bodies with its objective. The term “paperclip maximizer” has come to mean a PAGI that seeks to use all the resources it can get for an objective that most humans would not consider worthwhile (Arbital Contributors, 2017).

A PAGI that was smarter than humans, but not yet smart enough to take over the world, would have an incentive to hide its abilities and intentions from us if it predicted that we would turn the PAGI off if it scared us. Consequently, the PAGI might appear friendly weak, and unambitious right until it launches a surprise devastating attack on us, by taking what has been called a “treacherous turn” (Bostrom, 2014, pp. 116-119).