Absent innovation – unregulated emerging tech causes extinction

Historically, the risks that have arisen from emerging technologies have been small when compared with their benefits. The potential exceptions are unprecedented risks that could threaten large parts of the globe, or even our very survival1. Technology has significantly improved lives in the United Kingdom and the rest of the world. Over the past 150 years, we have become much more prosperous. During this time, the UK average income rose by more than a factor of seven in real terms, much of this driven by improving technology. This increased prosperity has taken millions of people out of absolute poverty and has given everyone many more freedoms in their lives. The past 150 years also saw historically unprecedented improvements in health, with life expectancy in the United Kingdom steadily increasing by two to three years each decade. From a starting point of about 40 years, it has doubled to 80 years2. These improvements are not entirely due to technological advances, of course, but a large fraction of them are. We have seen the cost of goods fall dramatically due to mass production, domestic time freed up via labour saving machines at home, and people connected by automobiles, railroads, airplanes, telephones, television, and the Internet. Health has improved through widespread improvements in sanitation, vaccines, antibiotics, blood transfusions, pharmaceuticals, and surgical techniques. These benefits significantly outweigh many kinds of risks that emerging technologies bring, such as those that could threaten workers in industry, local communities, consumers, or the environment. After all, the dramatic improvements in prosperity and health already include all the economic and health costs of accidents and inadvertent consequences during technological development and deployment, and the balance is still overwhelmingly positive. This is not to say that governance does or should ignore mundane risks from new technologies in the future. Good governance may have substantially decreased the risks that we faced over the previous two centuries, and if through careful policy choices we can reduce future risks without much negative impact on these emerging technologies, then we certainly should do so. However, we may not yet have seen the effects of the most important risks from technological innovation. Over the next few decades, certain technological advances may pose significant and unprecedented global risks. Advances in the biosciences and biotechnology may make it possible to create bioweapons more dangerous than any disease humanity has faced so far; geoengineering technologies could give individual countries the ability to unilaterally alter the global climate (see case study); rapid advances in artificial intelligence could give a single country a decisive strategic advantage. These scenarios are extreme, but they are recognized as potential low-probability high-impact events by relevant experts. To safely navigate these risks, and harness the potentially great benefits of these new technologies, we must continue to develop our understanding of them and ensure that the institutions responsible for monitoring them and developing policy responses are fit for purpose. This chapter explores the high-consequence risks that we can already anticipate; explains market and political challenges to adequately managing these risks; and discusses what we can do today to ensure that we achieve the potential of these technologies while keeping catastrophic threats to an acceptably low level. We need to be on our guard to ensure we are equipped to deal with these risks, have the regulatory vocabulary to manage them appropriately, and continue to develop the adaptive institutions necessary for mounting reasonable responses. Anthropogenic existential risks vs. natural existential risks An existential risk is defined as a risk that threatens the premature extinction of humanity, or the permanent and drastic destruction of its potential for desirable future development. These risks could originate in nature (as in a large asteroid impact, gamma-ray burst, supernova, supervolcano eruption, or pandemic) or through human action (as in a nuclear war, or in other cases we discuss below). This chapter focuses on anthropogenic existential risks because — as we will now argue — the probability of these risks appears significantly greater. Historical evidence shows that species like ours are not destroyed by natural catastrophes very often. Humans have existed for 200,000 years. Our closest ancestor, Homo erectus, survived for about 1.8 million years. The median mammalian species lasts for about 2.2 million years3. Assuming that the distribution of natural existential catastrophes has not changed, we would have been unlikely to survive as long as we have if the chance of natural extinction in a given century were greater than 1 in 500 or 1 in 5,000 (since (1 – 1/500)2,000 and (1 – 1/5,000)18,000 are both less than 2%). Consistent with this general argument, all natural existential risks are believed to have very small probabilities of destroying humanity in the coming century4. Case study omitted. In contrast, the tentative historical evidence we do have points in the opposite direction for anthropogenic risks. The development of nuclear fission, and the atomic bomb, was the first time in history that a technology created the possibility of destroying most or all of the world’s population. Fortunately we have not yet seen a global nuclear catastrophe, but we have come extremely close. US President John F. Kennedy later confessed that during the Cuban missile crisis, the chances of a nuclear war with Russia seemed to him at the time to be “somewhere between one out of three and even”. In light of this evidence, it is intuitively rather unclear that we could survive 500 or 5,000 centuries without facing a technologically-driven global catastrophe such as a nuclear war. We argue that in the coming decades, the world can expect to see several powerful new technologies that — by accident or design — may pose equal or greater risks for humanity. 1. Engineered Pathogens Pandemics such as Spanish flu and HIV have killed tens of millions of people. Smallpox alone was responsible for more than 300 million deaths in the first half of the twentieth century. As the ongoing Ebola epidemic reminds us, disease outbreaks remain a potent threat today. However, pressures from natural selection limit the destructive potential of pathogens because a sufficiently virulent, transmissible pathogen would eliminate the host population. As others have argued, and we reiterate below, bioengineering could be used to overcome natural limits on virulence and transmissibility, allowing pandemics of unprecedented scale and severity. For an example of an increase in fatality rates, consider mousepox, a disease that is normally non-lethal in mice. In 2001, Australian researchers modified mousepox, accidentally increasing its fatality rate to 60%, even in mice with immunity to the original version5 . By 2003, researchers led by Mark Buller found a way to increase the fatality rate to 100%, although the team also found therapies that could protect mice from the engineered version6 . For an example of an increase in transmissibility, consider the ‘gain of function’ experiments on influenza that have enabled airborne transmission of modified strains of H5N1 between ferrets7 . Proponents of such experiments argue that further efforts building on their research “have contributed to our understanding of host adaptation by influenza viruses, the development of vaccines and therapeutics, and improved [disease] surveillance”8 . However, opponents argue that enhancing the transmissibility of H5N1 does little to aid in vaccine development; that long lag times between capturing and sequencing natural flu samples limits the value of this work for surveillance; and that epistasis — in which interactions between genes modulate their overall effects — limits our ability to infer the likely consequences of other genetic mutations in influenza from what we have observed in gainof-function research so far9 . Many concerns have been expressed about the catastrophic and existential risks associated with engineered pathogens. For example, George Church, a pioneer in the field of synthetic biology, has said: “While the likelihood of misuse of oligos to gain access to nearly extinct human viruses (e.g. polio) or novel pathogens (like IL4-poxvirus) is small, the consequences loom larger than chemical and nuclear weapons, since biohazards are inexpensive, can spread rapidly world-wide and evolve on their own.”10 Similarly, Richard Posner11, Nathan Myhrvold12, and Martin Rees13 have argued that in the future, an engineered pathogen with the appropriate combination of virulence, transmissibility and delay of onset in symptoms would pose an existential threat to humanity. Unfortunately, developments in this field will be much more challenging to control than nuclear weapons because the knowledge and equipment needed to engineer viruses is modest in comparison with what is required to create a nuclear weapon14. It is possible that once the field has matured over the next few decades, a single undetected terrorist group would be able to develop and deploy engineered pathogens. By the time the field is mature and its knowledge and tools are distributed across the world, it may be very challenging to defend against such a risk. This argues for the continuing development of active policy-oriented research, an intelligence service to ensure that we know what misuse some technologies are being put to, and a mature and adaptive regulatory structure in order to ensure that civilian use of materials can be appropriately developed to maximize benefit and minimize risk. We raise these potential risks to highlight some worst-case scenarios that deserve further consideration. Advances in these fields are likely to have significant positive consequences in medicine, energy, and agriculture. They may even play an important role in reducing the risk of pandemics, which currently pose a greater threat than the risks described here. 2. Artificial Intelligence Artificial intelligence (AI) is the science and engineering of intelligent machines. Narrow AI systems — such as Deep Blue, stock trading algorithms, or IBM’s Watson — work only in specific domains. In contrast, some researchers are working on AI with general capabilities, which aim to think and plan across all the domains that humans can. This general sort of AI only exists in very primitive forms today15. Many people have argued that long-term developments in artificial intelligence could have catastrophic consequences for humanity in the coming century16, while others are more skeptical17. AI researchers have differing views about when AI systems with advanced general capabilities might be developed, whether such development poses significant risks, and how seriously radical scenarios should be taken. As we’ll see, there are even differing views about how to characterize the distribution of opinion in the field. In 2012, Müller and Bostrom surveyed the 100 most-cited AI researchers to ask them when advanced AI systems might be developed, and what the likely consequences would be. The survey defined a “high-level machine intelligence” (HLMI) as a machine “that can carry out most human professions at least as well as a typical human”, and asked the researchers about which year they would assign a 10%, 50% or 90% subjective probability to such AI being developed. They also asked whether the overall consequences for humanity would be “extremely good”, “on balance good”, “more or less neutral”, “on balance bad”, or “extremely bad (existential catastrophe)”. The researchers received 29 responses: the median respondent assigned a 10% chance of HLMI by 2024, a 50% chance of HLMI by 2050, and a 90% chance of HLMI by 2070. For the impact on humanity, the median respondent assigned 20% to “extremely good”, 40% to “on balance good”, 19% to “more or less neutral”, 13% to “on balance bad”, and 8% to “extremely bad (existential catastrophe)”18. In our view, it would be a mistake to take these researchers’ probability estimates at face value, for several reasons. First, the AI researchers’ true expertise is in developing AI systems, not forecasting the consequences for society from radical developments in the field. Second, predictions about the future of AI have a mixed historical track record19. Third, these ‘subjective probabilities’ represent individuals’ personal degrees of confidence, and cannot be taken to be any kind of precise estimate of an objective chance. Fourth, only 29 out of 100 researchers responded to the survey, which therefore may not be representative of the field as a whole. The difficulty in assessing risks from AI is brought out further by a report from the Association for the Advancement of Artificial Intelligence (AAAI), which came to a different conclusion. In February 2009, about 20 leading researchers in AI met to discuss the social impacts of advances in their field. One of three sub-groups focused on potentially radical long-term implications of progress in artificial intelligence. They discussed the possibility of rapid increases in the capabilities of intelligent systems, as well as the possibility of humans losing control of machine intelligences that they had created. The overall perspective and recommendations were summarized as follows: • “The first focus group explored concerns expressed by lay people — and as popularized in science fiction for decades — about the long-term outcomes of AI research. Panelists reviewed and assessed popular expectations and concerns. The focus group noted a tendency for the general public, science-fiction writers, and futurists to dwell on radical long-term outcomes of AI research, while overlooking the broad spectrum of opportunities and challenges with developing and fielding applications that leverage different aspects of machine intelligence.” • “There was overall skepticism about the prospect of an intelligence explosion as well as of a “coming singularity,” and also about the large-scale loss of control of intelligent systems. Nevertheless, there was a shared sense that additional research would be valuable on methods for understanding and verifying the range of behaviors of complex computational systems to minimize unexpected outcomes.” • “The group suggested outreach and communication to people and organizations about the low likelihood of the radical outcomes, sharing the rationale for the overall comfort of scientists in this realm, and for the need to educate people outside the AI research community about the promise of AI for enhancing the quality of human life in numerous ways, coupled with a re-focusing of attention on actionable, shorter-term challenges.”20 This panel gathered prominent people in the field to discuss the social implications of advances in AI in response to concerns from the public and other researchers. They reported on their views about the concerns, recommended plausible avenues for deeper investigation, and highlighted the possible upsides of progress in addition to discussing the downsides. These were valuable contributions. However, the event had shortcomings as well. First, there is reason to doubt that the AAAI panel succeeded in accurately reporting the field’s level of concern about future developments in AI. Recent commentary on these issues from AI researchers has struck a different tone. For instance, the survey discussed above seems to indicate more widespread concern. Moreover, Stuart Russell — a leader in the field and author of the most-used textbook in AI — has begun publicly discussing AI as a potential existential risk21.