Causes Extinction.

The events from 2045 onwards will start to evolve very rapidly. Very soon after it has been created, Superintelligence, embodied in whatever form, will be capable of redesigning itself. Repetitions of this cycle will result in the so called runaway effect, or an intelligence explosion, where such a smart machine designs successive generations of increasingly improved, powerful copies of itself, creating intelligence far exceeding human intellectual capacity. Once Superintelligence has reached that capability it may be impossible for a human to comprehend and even less so, to control it. Humanity will have reached the point known as Technological Singularity. Ray Kurzweil predicts that Technological Singularity event will happen by 2045.

If we manage to turn Superintelligence into our friend and let it arrive at the Technological Singularity event, assuming we would still have control over its evolution (e.g. via linking its goals to our most important human values), then the question is what will be our choice. For example, we can let it evolve itself and, so to speak, let it fly off and leave us alone. That may be possible. However, it is unlikely we will choose this option, since as someone said, ‘one cannot uninvent the atomic bomb’. We are inquisitive and innovative beings and it is difficult to imagine we would like to return to ‘less developed’ world if it was at all possible.

We have to assume that once the Technological Singularity is there, it will become our Master by default, even if our values will have been embedded in its overall decision-making pattern. Its knowledge, choices of important decisions for humanity and overall comprehension of the world around us and the Universe in general, will be unimaginably greater than our own capabilities.

So, we shall have two options (assuming there are both possible). The first one is to upload (copy) our personality, memories, and consciousness by reading our brain (although that may not be enough) onto a digital platform (a chip embedded in some material resembling our bodies). The second option is to merge our minds in a digital form with Superintelligence. From then on it is a pure speculation about how Humanity might evolve into the long, long future. However, in the next few decades we may be forced to make that biggest decision in the history of Humanity on how we want to evolve as a species. Therefore, let me take you on my own journey into the unknown because the conclusions may be useful for making such a decision.

Chapter 2

The risk arising from a hostile Superintelligence

I would now like to concentrate on the risk for Humanity arising from the advent of Superintelligence. The key aspect of the assessment of that risk for Humanity is to ask a basic question about key values that define Humanity, what is good and what is right. I will develop this argument further on. So, paradoxically, Superintelligence forces us to answer these questions more meaningfully than ever before.

In my view, Superintelligence represents the most dangerous risk. Why? Because it is almost certain to happen, if we do nothing, unlike natural pandemics that may not happen at all, even if we do not apply any counter measures, since it is a lottery type risk. The second reason why it is so dangerous is that it may happen much earlier than the risk mostly talked about in recent years – the climatic catastrophe. The risk coming from Superintelligence is more likely to happen in the next 50 years rather than next century. On the other hand, I believe that if we manage to produce the so called “friendly” Superintelligence, then instead of becoming the biggest risk, it itself can help us reduce other anthropogenic risks, such as climate change.

The Superintelligence is defined as a type of artificial intelligence that would surpass even the smartest humans. The threat stems, from even the slightest misalignment of our values and Superintelligence’s objectives, or its “values”. If this happens, even when the corresponding goals appear benign, it could be disastrous. Nick Bostrom quotes a scaring example that involves a Superintelligence programmed to “maximize” the abundance of some objects, like paperclips. This could lead Superintelligence to harvest all available atoms, including those in human bodies, thereby destroying humanity (and perhaps the entire biosphere (77). In addition, there are multiple ways that Superintelligence could become outright malevolent toward humanity, as University of Louisville computer scientist Roman Yampolskiy outlines in a recent paper (78).

Preventing humans from using resources such as money, land, water, rare elements, organic matter, internet service or computer hardware;

Subverting the functions of local and federal governments, international corporations, professional societies, and charitable organizations to pursue its own ends, rather than their human-designed purposes;

Constructing a total surveillance state (or exploitation of an existing one), reducing any notion of privacy to zero – including privacy of thought;

Enslaving humankind, restricting our freedom to move or otherwise choose what to do with our bodies and minds, as through forced cryonics or concentration camps;

Abusing and torturing humankind with perfect insight into our physiology to maximize amount of physical or emotional pain, perhaps combining it with a simulated model of us to make the process infinitely long.

It would be impossible to provide a complete list of negative outcomes of an AI agent with general reasoning ability that he would be able to inflict. We can expect a lot of these sorts of attacks in the future. The situation is even more complicated once we consider systems that exceed human capacity. Superintelligence may be capable of inventing dangers we are not even capable of predicting or imagining.

So, we have to accept that the creation of Superintelligence poses perhaps the most difficult long-term risks to the future of Humanity. Phil Torres identifies several issues here, saying that the first one is the amity-enmity problem: the AI could dislike us for whatever reason, and therefore try to kill us. The second risk is the indifference problem: the AI could simply not care about our well-being, and thus destroy us because we happen to be in the way. And finally, there is yet another problem, which he calls “the clumsy fingers problem”: the AI could inadvertently nudge us over the cliff of extinction rather than intentionally pushing us. This possibility is based on the assumptions, which states that higher levels of intelligence aren’t necessarily correlated with the avoidance of certain kinds of mistakes. He warns that the fruits of our ingenuity — namely, dual-use technologies — have introduced brand new existential risk scenarios never before encountered by Earth-originating life. Given the immense power of Superintelligence, e.g. it could manipulate matter in ways that appear to us as pure magic, it would be enough to make a single error for such a being to trip humanity into the eternal grave of extinction (79).