Self-improvement makes super-intelligent AI inevitable.

*Problematic language edited in brackets; PAGI = Powerful Artificial General Intelligence

If the human brain lacks supernatural components such as a soul, then it is just a machine. But it is a mere three-pound machine that must fit inside of a human skull, be powered by relatively tiny amounts of energy, and be made entirely of biological materials. Engineers, who do not face any of these constrains when building their computers, should be able to eventually construct much more powerful machines than the human brain since these engineers should eventually be able to do everything that nature does plus much more. Furthermore, it seems extraordinarily unlikely that evolution just happened to hit upon the smartest possible minds when it formed the brains of Einstein, von Neumann, and Newton, especially given the constrains that evolution faced. If our high technology civilization survives long enough, it will someday likely be capable of making machines much smarter than any human who has ever existed.

Intelligence propels economic growth and military power (Miller 2012, 47-53). Enormous riches and influence will go to whatever organization creates smarter-than-human general intelligence. Even if most people agreed that PAGI is dangerous and we should avoid building it, competition among countries, organizations, and programmers will likely push someone, somewhere to create PAGI if PAGI can indeed be created.Unlike with whatever wetware runs the human brain, it would be relatively easy to make changes to a PAGI’s software. The PAGI could even make changes to itself. Such self-modification could possibly allow a PAGI to undergo an intelligence explosion where it figures out how to improve its own intelligence, then, as it gets smarter, figures out new ways to improve its intelligence. It has been theorized that through recursive self-improvement a PAGI could go from being a bit smarter than humans to becoming a computer superintelligence in a matter of days (Good 1965, Yudkowsky 2008).