Growth-oriented AI ensures extinction---but degrowth solves.

The challenges of sustainability and of superintelligence are not independent. The changing 84 fluxes of energy, matter, and information can be interpreted as different faces of a general acceleration2 85 . More directly, it is argued below that superintelligence would deeply affect 86 production technologies and also economic decisions, and could in turn be affected by the 87 socioeconomic and ecological context in which it develops. Along the lines of Pueyo (2014, p. 88 3454), this paper presents an approach that integrates these topics. It employs insights from a 89 variety of sources, such as ecological theory and several schools of economic theory. 90 The next section presents a thought experiment, in which superintelligence emerges after the 91 technical aspects of goal alignment have been resolved, and this occurs specifically in a neoliberal 92 scenario. Neoliberalism is a major force shaping current policies on a global level, which urges 93 governments to assume as their main role the creation and support of capitalist markets, and to 94 avoid interfering in their functioning (Mirowski, 2009). Neoliberal policies stand in sharp contrast 95 to degrowth views: the first are largely rationalized as a way to enhance efficiency and production 96 (Plehwe, 2009), and represent the maximum expression of capitalist values. 97 The thought experiment illustrates how superintelligence perfectly aligned with capitalist 98 markets could have very undesirable consequences for humanity and the whole biosphere. It also 99 suggests that there is little reason to expect that the wealthiest and most powerful people would be 100 exempt from these consequences, which, as argued below, gives reason for hope. Section 3 raises 101 the possibility of a broad social consensus to respond to this challenge along the lines of degrowth, 102 thus tackling major technological, environmental, and social problems simultaneously. The 103 uncertainty involved in these scenarios is vast, but, if a non-negligible probability is assigned to 104 these two futures, little room is left for either complacency or resignation. 105 106 2. Thought experiment: Superintelligence in a neoliberal scenario 107 108 Neoliberalism is creating a very special breeding ground for superintelligence, because it strives 109 to reduce the role of human agency in collective affairs. The neoliberal pioneer Friedrich Hayek 110 argued that the spontaneous order of markets was preferable over conscious plans, because markets, 111 he thought, have more capacity than humans to process information (Mirowski, 2009). Neoliberal 112 policies are actively transferring decisions to markets (Mirowski, 2009), while firms' automated 113 decision systems become an integral part of the market's information processing machinery 114 (Davenport and Harris, 2005). Neoliberal globalization is locking governments in the role of mere 115 players competing in the global market (Swank, 2016). Furthermore, automated governance is a 116 foundational tenet of neoliberal ideology (Plehwe, 2009, p. 23). 117 In the neoliberal scenario, most technological development can be expected to take place either in the context of firms or in support of firms3 118 . A number of institutionalist (Galbraith, 1985), post119 Keynesian (Lavoie, 2014; and references therein) and evolutionary (Metcalfe, 2008) economists 120 concur that, in capitalist markets, firms tend to maximize their growth rates (this principle is related 121 but not identical to the neoclassical assumption that firms maximize profits; Lavoie, 2014). Growth 122 maximization might be interpreted as expressing the goals of people in key positions, but, from an 123 evolutionary perspective, it is thought to result from a mechanism akin to natural selection 124 (Metcalfe, 2008). The first interpretation is insufficient if we accept that: (1) in big corporations, the 125 managerial bureaucracy is a coherent social-psychological system with motives and preferences of 126 its own (Gordon, 1968, p. 639; for an insider view, see Nace, 2005, pp. 1-10), (2) this system is 127 becoming techno-social-psychological with the progressive incorporation of decision-making 128 algorithms and the increasing opacity of such algorithms (Danaher, 2016), and (3) human mentality 129 and goals are partly shaped by firms themselves (Galbraith, 1985). 130 The type of AI best suited to participate in firms' decisions in this context is described in a 131 recent review in Science: AI researchers aim to construct a synthetic homo economicus, the 132 mythical perfectly rational agent of neoclassical economics. We review progress toward creating 133 this new species of machine, machina economicus (Parkes and Wellman, 2015, p. 267; a more 134 orthodox denomination would be Machina oeconomica). 135 Firm growth is thought to rely critically on retained earnings (Galbraith, 1985; Lavoie, 2014, p. 136 134-141). Therefore, economic selection can be generally expected to favor firms in which these are greater. The aggregate retained earnings4 137 RE of all firms in an economy can be expressed as: 138 RE=FE(R,L,K)-w⋅L-(i+δ)⋅K-g. (1) 139 Bold symbols represent vectors (to indicate multidimensionality). F is an aggregate production 140 function, relying on inputs of various types of natural resources R, labor L and capital K (including intelligent machines), and being affected by environmental factors5 141 E; w are wages, i are returns to 142 capital (dividends, interests) paid to households, δ is depreciation and g are the net taxes paid to 143 governments. 144 Increases in retained earnings face constraints, such as trade-offs among different parameters of 145 Eq. 1. The present thought experiment explores the consequences of economic selection in a 146 scenario in which two sets of constraints are nearly absent: sociopolitical constraints on market 147 dynamics are averted by a neoliberal institutional setting, while technical constraints are overcome 148 by asymptotically advanced technology (with extreme AI allowing for extreme technological 149 development also in other fields). The environmental and the social implications are discussed in 150 turn. Note that this scenario is not defined by some contingent choice of AIs' goals by their 151 programmers: The goals of maximizing each firm's growth and retained earnings are assumed to 152 emerge from the collective dynamics of large sets of entities subject to capitalistic rules of 153 interaction and, therefore, to economic selection.