Reinforced Learners (RL) rewards function logic mirrors classic utilitarian framing
Heather M. Roff , 19 July 2022, Senior Research Analyst, Johns Hopkins Applied Physics Laboratory Fellow, Foreign Policy, Brookings Institution Associate Research Fellow, Leverhulme Centre for the Future of Intelligence, University of Cambridge https://arxiv.org/pdf/2008.07321.pdf - Maren Lien
While RL is premised on a feedback loop, where an agent tries a particular action or behavior to reach a (preselected/identified) goal and receives a reward signal from the environment providing information about whether it is acting correctly, the environment does not fundamentally change. The environment may produce varying signals to the agent, but the environment as such, does not ontologically change, and so this is not a co-constituting ontology. The feedback loop only goes in one direction, not two. Thus, for the RL agent, a goal remains something external or exogenous to it, and the agent directs its actions towards it. It cannot modify or change its goal. This is important because as AIs become more powerful, the model for learning may need to change. RL agents may be able to reshape not only their environments, but rewards as well. An example may be helpful here. An agent playing chess receives various signals from its environment (such as another agent’s move on the board). The signal tells the agent which actions are not only permissible to take, but which actions may preferable or best to take, given the available knowledge about the game of chess at the time, with the goal of winning the game. This is a feedback loop. However, it is not a co-constituting agent-structure problem. The agent playing chess cannot, once its adversary moves to e5, claim that the board itself changed and now is a 9x9 matrix. The board is the structure; the agents move within it. The conceptual terrain is important here to understanding how the logic of reinforcement learning and the billiard ball model of learning work to produce a particular kind of utility maximizing agent: a hedonistic act utilitarian. While one may object, and claim that the RL utility maximizing agent is a different kind of “utilitarian” than that of the moral philosophers, this is in fact, not wholly true. One is merely the mathematical representation of the logic of the other, and indeed implicitly carries in certain assumptions about broader normative aims. For example, as Sutton and Barto (1998) explain with regards to reinforcement learning, Rewards are in a sense primary, whereas values, as predictions of rewards, are secondary. Without rewards there could be no values, and the only purpose of estimating values is to achieve more reward. Nevertheless, it is values with which we are more concerned when making and evaluating decisions. Action choices are made based on value judgments. 7 Here, a value judgment is a judgment about what sorts of behaviors or actions contain “worth”. But what “worth” entails, is actually about what is “good” all things considered for the decision or task at hand. 8 It is, therefore a teleological or goal-directed action. Moreover, if one desires to avoid circularity, then one cannot define “good” or “value” in terms of itself, so one must find some other thing to use as a signal or content for the “good.” In the RL scheme, the signal is the reward function, and this is used to estimate or “judge” which actions or states will yield the long term best act/choice/decision; that is, what will most likely yield the “value” function. Yet if we grant that this conceptualization of RL as true, then we must also grant that it maps — almost identically — to the logic of the moral theory of utilitarianism as well. Utilitarianism is a consequentialist moral theory that argues that normative properties are determined by some sort of end state (or goal). In easy terms, the best consequences — here operationalized as the most utility — determine the morally right act. Utility can be defined in any number of ways, as I will explain in the next section, however it is most often associated with an aggregate welfare account, usually seen in shorthand as “the greatest good for the greatest number.” One ought to immediately see that RL is very much like utilitarianism because both the RL agent and the utilitarian moral agent seek to determine some present action by a judgment about maximizing the value — or good — of some future state/goal/consequence. Additionally, utilitarianism also seeks coherence and avoiding circularity. Thus, one cannot define the good in terms of itself, but must instead define it in some other (nonmoral) term. Classically, this is done through a reward signal (pleasure or pain).9