Existential risk framing PRODUCES existential threats by reifying capitalist control over the universe. Attempts to “manage” existential risk backfire and justify endless atrocity

(Phil Torresis a PhD candidate in philosophy at Leibniz Universität Hannover in Germany. https://aeon.co/essays/why-longtermism-is-the-worlds-most-dangerous-secular-credo)

Yet the implications of longtermism are far more worrisome. If our top four priorities are to avoid an existential catastrophe – ie, to fulfil ‘our potential’ – then what’s not on the table for making this happen? Consider Thomas Nagel’s comment about how the notion of what we might call the ‘greater good’ has been used to ‘justify’ certain atrocities (eg, during war). If the ends ‘justify’ the means, he argues, and the ends are thought to be sufficiently large (eg, national security), then this ‘can be brought to bear to ease the consciences of those responsible for a certain number of charred babies’. Now imagine what might be ‘justified’ if the ‘greater good’ isn’t national security but the cosmic potential of Earth-originating intelligent life over the coming trillions of years? During the Second World War, 40 million civilians perished, but compare this number to the 1054 or more people (in Bostrom’s estimate) who could come to exist if we can avoid an existential catastrophe. What shouldn’t we do to ‘protect’ and ‘preserve’ this potential? To ensure that these unborn people come to exist? What means can’t be ‘justified’ by this cosmically significant moral end? Bostrom himself argued that we should seriously consider establishing a global, invasive surveillance system that monitors every person on the planet in realtime, to amplify the ‘capacities for preventive policing’ (eg, to prevent omnicidal terrorist attacks that could devastate civilisation). Elsewhere, he’s written that states should use preemptive violence/war to avoid existential catastrophes, and argued that saving billions of actual people is the moral equivalent of reducing existential risk by utterly minuscule amounts. In his words, even if there is ‘a mere 1 per cent chance’ of 1054 people existing in the future, then ‘the expected value of reducing existential risk by a mere one billionth of one billionth of one percentage point is worth 100 billion times as much as a billion human lives.’ Such fanaticism – a word that some longtermists embrace – has led a growing number of critics to worry about what might happen if political leaders in the real world were to take Bostrom’s view seriously. To quote the mathematical statistician Olle Häggström, who – perplexingly – tends otherwise to speak favourably of longtermism: I feel extremely uneasy about the prospect that [the calculations above] might become recognised among politicians and decision-makers as a guide to policy worth taking literally. It is simply too reminiscent of the old saying ‘If you want to make an omelette, you must be willing to break a few eggs,’ which has typically been used to explain that a bit of genocide or so might be a good thing, if it can contribute to the goal of creating a future utopia. Imagine a situation where the head of the CIA explains to the US president that they have credible evidence that somewhere in Germany, there is a lunatic who is working on a doomsday weapon and intends to use it to wipe out humanity, and that this lunatic has a one-in-a-million chance of succeeding. They have no further information on the identity or whereabouts of this lunatic. If the president has taken Bostrom’s argument to heart, and if he knows how to do the arithmetic, he may conclude that it is worthwhile conducting a full-scale nuclear assault on Germany to kill every single person within its borders. Here, then, are a few reasons I find longtermism to be profoundly dangerous. Yet there are additional, fundamental problems with this worldview that no one, to my knowledge, has previously noted in writing. For example, there’s a good case to make that the underlying commitments of longtermism are a major reason why humanity faces so many unprecedented risks to its survival in the first place. Longtermism might, in other words, be incompatible with the attainment of ‘existential security’, meaning that the only way to genuinely reduce the probability of extinction or collapse in the future might be to abandon the longtermist ideology entirely. To Bostrom and Ord, failing to become posthuman would prevent us from realising our vast, glorious potential To understand the argument, let’s first unpack what longtermists mean by our ‘longterm potential’, an expression that I have so far used without defining. We can analyse this concept into three main components: transhumanism, space expansionism, and a moral view closely associated with what philosophers call ‘total utilitarianism’. The first refers to the idea that we should use advanced technologies to reengineer our bodies and brains to create a ‘superior’ race of radically enhanced posthumans (which, confusingly, longtermists place within the category of ‘humanity’). Although Bostrom is perhaps the most prominent transhumanist today, longtermists have shied away from using the term ‘transhumanism’, probably because of its negative associations. Susan Levin, for example, points out that contemporary transhumanism has its roots in the Anglo-American eugenics movement, and transhumanists such as Julian Savulescu, who co-edited the book Human Enhancement (2009) with Bostrom, have literally argued for the consumption of ‘morality-boosting’ chemicals such as oxytocin to avoid an existential catastrophe (which he calls ‘ultimate harm’). As Savulescu writes with a colleague, ‘it is a matter of such urgency to improve humanity morally … that we should seek whatever means there are to effect this.’ Such claims are not only controversial but for many quite disturbing, and hence longtermists have attempted to distance themselves from such ideas, while nonetheless championing the ideology. Transhumanism claims that there are various ‘posthuman modes of being’ that are far better than our current human mode. We could, for instance, genetically alter ourselves to gain perfect control over our emotions, or access the internet via neural implants, or maybe even upload our minds to computer hardware to achieve ‘digital immortality’. As Ord urges in The Precipice, think of how awesome it would be to perceive the world via echolocation, like bats and dolphins, or magnetoreception, like red foxes and homing pigeons. ‘Such uncharted experiences,’ Ord writes, ‘exist in minds much less sophisticated than our own. What experiences, possibly of immense value, could be accessible, then, to minds much greater?’ Bostrom’s most fantastical exploration of these possibilities comes from his evocative ‘Letter from Utopia’ (2008), which depicts a techno-Utopian world full of superintelligent posthumans awash in so much ‘pleasure’ that, as the letter’s fictional posthuman writes, ‘we sprinkle it in our tea.’ The connection with longtermism is that, according to Bostrom and Ord, failing to become posthuman would seemingly prevent us from realising our vast and glorious potential, which would be existentially catastrophic. As Bostrom put it in 2012, ‘the permanent foreclosure of any possibility of this kind of transformative change of human biological nature may itself constitute an existential catastrophe.’ Similarly, Ord asserts that ‘forever preserving humanity as it is now may also squander our legacy, relinquishing the greater part of our potential.’ The second component of our potential – space expansionism – refers to the idea that we must colonise as much of our future light cone as possible: that is, the region of spacetime that is theoretically accessible to us. According to longtermists, our future light cone contains a huge quantity of exploitable resources, which they refer to as our ‘cosmic endowment’ of negentropy (or reverse entropy). The Milky Way alone, Ord writes, is ‘150,000 light years across, encompassing more than 100 billion stars, most with their own planets.’ Attaining humanity’s longterm potential, he continues, ‘requires only that [we] eventually travel to a nearby star and establish enough of a foothold to create a new flourishing society from which we could venture further.’ By spreading ‘just six light years at a time’, our posthuman descendants could make ‘almost all the stars of our galaxy … reachable’ since ‘each star system, including our own, would need to settle just the few nearest stars [for] the entire galaxy [to] eventually fill with life.’ The process could be exponential, resulting in ever-more ‘flourishing’ societies with each additional second our descendants hop from star to star. But why exactly would we want to do this? What’s so important about flooding the Universe with new posthuman civilisations? This leads to the third component: total utilitarianism, which I will refer to as ‘utilitarianism’ for short. Although some longtermists insist that they aren’t utilitarians, we should right away note that this is mostly a smoke-and-mirrors act to deflect criticisms that longtermism – and, more generally, the effective altruism (EA) movement from which it emerged – is nothing more than utilitarianism repackaged. The fact is that the EA movement is deeply utilitarian, at least in practice, and indeed, before it decided upon a name, the movement’s early members, including Ord, seriously considered calling it the ‘effective utilitarian community’. This being said, utilitarianism is an ethical theory that specifies our sole moral obligation as being to maximise the total amount of ‘intrinsic value’ in the world, as tallied up from a disembodied, impartial, cosmic vantage point called ‘the point of view of the Universe’. From this view, it doesn’t matter how value – which utilitarian hedonists equate with pleasure – is distributed among people across space and time. All that matters is the total net sum. For example, imagine that there are 1 trillion people who have lives of value ‘1’, meaning that they are just barely worth living. This gives a total value of 1 trillion. Now consider an alternative universe in which 1 billion people have lives with a value of ‘999’, meaning that their lives are extremely good. This gives a total value of 999 billion. Since 999 billion is less than 1 trillion, the first world full of lives hardly worth living would be morally better than the second world, and hence, if a utilitarian were forced to choose between these, she would pick the former. (This is called the ‘repugnant conclusion’, which longtermists such as Ord, MacAskill and Greaves recently argued shouldn’t be taken very seriously. For them, the first world really might be better!) Beckstead argued that we should prioritise the lives of people in rich countries over those in poor countries The underlying reasoning here is based on the idea that people – you and I – are nothing more than means to an end. We don’t matter in ourselves; we have no inherent value of our own. Instead, people are understood as the ‘containers’ of value, and hence we matter only insofar as we ‘contain’ value, and therefore contribute to the overall net amount of value in the Universe between the Big Bang and the heat death. Since utilitarianism tells us to maximise value, it follows that the more people (value containers) who exist with net-positive amounts of value (pleasure), the better the Universe will become, morally speaking. In a phrase: people exist for the sake of maximising value, rather than value existing for the sake of benefitting people. This is why longtermists are obsessed with calculating how many people could exist in the future if we were to colonise space and create vast computer simulations around stars in which unfathomably huge numbers of people live net-positive lives in virtual-reality environments. I already mentioned Bostrom’s estimate of 1054 future people, which includes many of these ‘digital people’, but in his bestseller Superintelligence (2014) he puts the number even higher at 1058 people, nearly all of whom would ‘live rich and happy lives while interacting with one another in virtual environments’. Greaves and MacAskill are similarly excited about this possibility, estimating that some 1045 conscious beings in computer simulations could exist within the Milky Way alone. That is what our ‘vast and glorious’ potential consists of: massive numbers of technologically enhanced digital posthumans inside huge computer simulations spread throughout our future light cone. It is for this goal that, in Häggström’s scenario, a longtermist politician would annihilate Germany. It is for this goal that we must not ‘fritter … away’ our resources on such things as solving global poverty. It is for this goal that we should consider implementing a global surveillance system, keep pre-emptive war on the table, and focus more on superintelligent machines than saving people in the Global South from the devastating effects of climate change (mostly caused by the Global North). In fact, Beckstead has even argued that, for the sake of attaining this goal, we should actually prioritise the lives of people in rich countries over those in poor countries, since influencing the long-term future is of ‘overwhelming importance’, and the former are more likely to influence the long-term future than the latter. To quote a passage from Beckstead’s 2013 PhD dissertation, which Ord enthusiastically praises as one of the most important contributions to the longtermist literature: Saving lives in poor countries may have significantly smaller ripple effects than saving and improving lives in rich countries. Why? Richer countries have substantially more innovation, and their workers are much more economically productive. [Consequently,] it now seems more plausible to me that saving a life in a rich country is substantially more important than saving a life in a poor country, other things being equal. This is just the tip of the iceberg. Consider the implications of this conception of ‘our potential’ for the development of technology and creation of new risks. Since realising our potential is the ultimate moral goal for humanity, and since our descendants cannot become posthuman, colonise space and create ~1058 people in computer simulations without technologies far more advanced than those around today, failing to develop more technology would itself constitute an existential catastrophe – a failure mode (comparable to Ramsey neglecting his talents by spending his days playing pool and drinking) that Bostrom calls ‘plateauing’. Indeed, Bostrom places this idea front-and-centre in his canonical definition of ‘existential risk’, which denotes any future event that would prevent humanity from reaching and/or sustaining a state of ‘technological maturity’, meaning ‘the attainment of capabilities affording a level of economic productivity and control over nature close to the maximum that could feasibly be achieved.’ Technological maturity is the linchpin here because controlling nature and increasing economic productivity to the absolute physical limits are ostensibly necessary for creating the maximum quantity of ‘value’ within our future light cone. But reflect for a moment on how humanity got itself into the current climatic and ecological crisis. Behind the extraction and burning of fossil fuels, decimation of ecosystems and extermination of species has been the notion that nature is something to be controlled, subjugated, exploited, vanquished, plundered, transformed, reconfigured and manipulated. As the technology theorist Langdon Winner writes in Autonomous Technology (1977), since the time of Francis Bacon our view of technology has been ‘inextricably bound to a single conception of the manner in which power is used – the style of absolute mastery, the despotic, one-way control of the master over the slave.’ He adds: There are seldom any reservations about man’s rightful role in conquering, vanquishing, and subjugating everything natural. This is his power and his glory. What would in other situations seem [to be] rather tawdry and despicable intentions are here the most honourable of virtues. Nature is the universal prey, to manipulate as humans see fit. This is precisely what we find in Bostrom’s account of existential risks and its associated normative futurology: nature, the entire Universe, our ‘cosmic endowment’ is there for the plundering, to be manipulated, transformed and converted into ‘value-structures, such as sentient beings living worthwhile lives’ in vast computer simulations, quoting Bostrom’s essay ‘Astronomical Waste’ (2003). Yet this Baconian, capitalist view is one of the most fundamental root causes of the unprecedented environmental crisis that now threatens to destroy large regions of the biosphere, Indigenous communities around the world, and perhaps even Western technological civilisation itself. While other longtermists have not been as explicit as Bostrom, there is a clear tendency to see the natural world the way utilitarianism sees people: as means to some abstract, impersonal end, and nothing more. MacAskill and a colleague, for example, write that the EA movement, and by implication longtermism, is ‘tentatively welfarist in that its tentative aim in doing good concerns promoting wellbeing only and not, say, protecting biodiversity or conserving natural beauty for their own sakes.’