Even if a single technology can’t cause extinction---the accumulation of seemingly benign innovations is an existential risk.

2. Novel Technologies Reframed as an Explore/Exploit tradeoff

Bostrom's recent “Vulnerable World Hypothesis” lists several ways in which “there is some level of technological development at which civilization almost certainly gets devastated by default, i.e. unless it has exited the 'semi-anarchic default condition.”’ He illustrates this with an analogy modeled on a classic type of explanation in probability theory, that of an urn from which we draw balls. In his analogy, each ball represents a technological advance, and the balls can be white, black, or any shade of gray – representing the impact on humanity. Since we do not know what the future of technology will bring, the distribution of the colors of the balls inside the urn is unknown. He then discusses four scenarios which posit the existence of a “black ball” technology drawn from the urn, which, unlike those found so far with positive effects or mixed effects of humanity, “invariably or by default destroys the civilization that invents it.”

Bostrom admits that his analogy to an urn is simplistic, though it is compelling. However, for our argument it is insufficient. Instead of an urn containing discrete technologies, the process of scientific discovery that occurs when certain advances pose risks can be understood with a different parallel from probability theory, one more often used in machine learning, that of the explore-exploit tradeoff. To explain this, we can consider the multi-armed bandit problem, where a gambler possesses some finite number of coins, and is faced with a row of slot machines, each of which has a potentially different payout distribution. For each coin, the gambler is faced with a choice of which machine to play. A conservative gambler might “explore” a bit and try a few machines, and notice a machine that pays out 2:1 just over half the time, and stay there to “exploit” the machine, slowly but steadily making money, uninterested in further exploration. A more adventurous gambler would instead spend more time exploring the other machines, perhaps seeking a machine that pays out less often, but with a far higher reward. If no such machine exists, they lose out from their continued exploration - but if there is a machine they find which pays out, say, 50:1 25% of the time, they will make far more money.

Explore-exploit tradeoffs are also discussed in somewhat different terms in machine learning, by analogy to a landscape, rather than an urn or a multi-armed bandit. Here, imagine a robot, which in our analogy will parallel human civilization, in a potentially dangerous environment. There are rewards, that is, new technologies, scattered about. At each stage, the robot can explore in any direction to search the landscape for a new reward (novel technologies), or focus on places where it knows rewards can be found. Because the payoffs of future exploration are unknown, optimal decision making in explore-exploit situations is notoriously hard to analyze. In fact, when the multi-armed bandit problem was first proposed during World War II, allied analysts suggested that perhaps the most useful application was to give the problem to the Axis, so that they would also waste their time on the seemingly intractable problem.Gittins (1979)

Unlike the gambler, whose worst outcome is sub-optimal returns on their investment, in our scenario we suppose that the robot might accidentally become trapped or be destroyed by some previously unknown danger. If Bostrom's hypothesis is correct, civilization's exploration of new technologies comes with some probability of a fatal result, and some probability of discovering a highly rewarding new technology. In this new context, the goal is to find a search strategy that maximizes the total benefit humanity ever receives, what Bostrom has called humanity's cosmic endowment. Bostrom (2003) Any strategy, however, risks what Bostrom called astronomical waste - either embracing “technological relinquishment,” by being so risk-averse that advanced technologies that greatly benefit humanity are never explored or developed, or being so risk-accepting that Bostrom's postulated Vulnerable World becomes inevitable, losing the entire potential future benefit to humanity. The first critical question addressed by Bostrom - “Is there a black ball in the urn of possible inventions?” is, to reframe the question, about the existence of fatal dangers in our Robot's exploration landscape. If we could answer that question in the negative, this would seem to refute the informal hypothesis he proposes of a vulnerable world. This is not the only refutation he suggests, however. Bostrom's suggestion of “differential technological development” is, in our terms, to cordon-off or minimize exploration of sections of the landscape or directions most likely to contain fatal traps. If this can be done, it would refute the suggestion that technological development inevitably leads to increased risk of choosing a devastating technology.

Falsifying Bostrom's initial hypothesis does not, however, show that the world is not vulnerable in other ways. The fuller statement of the hypothesis in Bostrom's paper is that “[i]f technological development continues then a set of capabilities will at some point be attained that make the devastation of civilization extremely likely, unless civilization sufficiently exits the semi-anarchic default condition.“ This statement, I will argue, cannot be falsified even by proving the impossibility of “a technology that invariably or by default destroys the civilization that invents it”1 . In addition to Bostrom's four vulnerable worlds, I suggest a fifth possibility - that the simple accumulation of white and/or gray balls drawn from the urn can itself lead to fragility and, without strong forces pushing in the opposite direction, collapse of civilization by default. To frame it in our explore/exploit terms, the fifth possibility is that as the robot explores, it may dig too deep, or alter the landscape itself incautiously so that further movement could cause a collapse. The primary way for this to happen which this paper discusses is fragility - in the analogy, our exploration could lead to the equivalent of an avalanche or mine-collapse brought on by incautious behavior. That is, the creation of fragility in the landscape means that a lack of caution when exploring further could end up burying us under the weight of accumulated technologies.