Machine learning techniques create better AI – reward hacking, epistemological limitations, and distributions problems
Heather M. Roff , 19 July 2022, Senior Research Analyst, Johns Hopkins Applied Physics Laboratory Fellow, Foreign Policy, Brookings Institution Associate Research Fellow, Leverhulme Centre for the Future of Intelligence, University of Cambridge https://arxiv.org/pdf/2008.07321.pdf - Maren Lien
We want artificial intelligence (AI) to be beneficial.1 This is the grounding assumption of most of the attitudes towards AI research. We want AI to be “good” for humanity. We want it to help, not hinder, humans. Yet what exactly this entails in theory and in practice is not immediately apparent. Theoretically, this declarative statement subtly implies a commitment to a consequentialist ethics. Practically, some of the more promising machine learning techniques to create a robust AI, and perhaps even an artificial general intelligence (AGI) also commit one to a form of utilitarianism. In both dimensions, the logic of the beneficial AI movement may not in fact create “beneficial AI” in either narrow applications or in the form of AGI if the ethical assumptions are not made explicit and clear. Additionally, as it is likely that reinforcement learning (RL) will be an important technique for machine learning in this area, it is also important to interrogate how RL smuggles in a particular type of consequentialist reasoning into the AI: particularly, a brute form of hedonistic act utilitarianism. Since the mathematical logic commits one to a maximization function, the result is that an AI will inevitably be seeking more and more rewards. We have two conclusions that arise from this. First, is that if one believes that a beneficial AI is an ethical AI, then one is committed to a framework that posits ‘benefit’ is tantamount to the greatest good for the greatest number Second, if the AI relies on RL, then the way it reasons about itself, the environment, and other agents, will be through an act utilitarian morality. This proposition may, or may not, in fact be actually beneficial for humanity, (at least for the majority of reasons cited against utilitarianism over the past three hundred years). Indeed, as I will attempt to show here, much of the philosophical insights about the deficiencies of utilitarianism apply directly to many of the currently cited concrete problems in AI safety, such as specifying the wrong objective function, avoiding negative externalities, reward hacking, distribution problems, epistemological limitations, and distributional shift2 . The paper is organized into four sections. The first section lays out how RL could be seen as an implementation of hedonistic act utilitarianism. Section two discusses the various kinds of utilitarianism, as well as many of its well-cited criticisms, particularly how most of its forms ultimately collapse back into a brute act utilitarian evaluation. Moreover, this section argues that the two ways that utilitarianism is often described, as either an evaluative theory of right or as a decision-procedure for individual action, is the same description of how machine learning researchers are approaching the problem of creating beneficial AI. This presents us with a useful intervention point, both within policy but also within research communities to assess whether this is a solvable problem for AI. The third section argues that many of the problems noted in classic arguments about utilitarianism manifest themselves in concrete AI research safety problems, especially in reward hacking. Finally, section four argues that it may be useful for AI researchers to look towards philosophers to see how they have previously attempted to solve such problems. In particular, it may be more simple to look to various constraints on an AI’s action, or as Mackie notes as “device[s] for countering such specific evils” that may result.