Evil AGI won’t happen---they can’t develop independent motives.

Aside from being a highly stretched analogy, the use of human intelligence as the “archetype of general intelligence,” contains a massive oversight. The “generality” of human cognition is rooted in our status as a living organism whose survival strategies involve tool use, language and social cooperation. In other words, it is life – the human being’s imperative for survival, sustenance, shelter and reproduction - that provides human intelligence with its objectives and its evolutionary trajectory. Humans have learned to adapt to novel situations because it enhances their survival as a species. Intelligence serves life, not the other way around.

Yet computers are not alive. Giving them more computing power, building more complex neural networks, and feeding them more data will not by itself make them living beings. Life requires the ability to act autonomously to find the resources needed to sustain operations, and the ability to self-replicate. Computers cannot do that yet.3 No existing computing machines can solve problems they have not been given. They must be told what objectives to pursue and must be trained to pursue them. (I will address the question of whether AGI will develop its own goals later.) In other words, humans make computing machines "more intelligent" by improving their ability to pursue specific objectives defined by their human creators. Ergo, a "general intelligence" attributed to a man-made computing machine is really an oxymoron.