Human control is necessary, because judgement is essential for complex military decisions.
Goldfarb and Lindsay, 2022 – Chair in AI and health care at the Univ of Toronto, Prof of Cybersecurity at the Georgia Tech [Avi, Jon, 2/25/22, https://doi.org/10.1162/isec_a_00425, “Prediction and Judgement: Why Artificial Intelligence Increases the Importance of Humans in War” 6/18/22, LND]
There are three types of machine learning algorithms. All require human judgment. First, in “supervised learning,” the human tells the machine what to predict. Second, “unsupervised learning” requires judgment about what to classify and what to do with the classifications. Third, “reinforcement learning” requires advance specification of a reward function. A reward function assigns a numerical score to the perceived state of the world to enable a machine to maximize a goal. More complicated strategies may combine these approaches by establishing instrumental goals in pursuit of the main objective. In every case, a human ultimately codes the algorithm and defines the payoffs for the machine.
In economic terms, judgment is the specification of the utility function. The preferences and valuations that determine utility are distinct from the strategies that maximize it. To take a trivial example, people who do not mind getting wet and dislike carrying umbrellas will not carry one, regardless of the weather forecast. People who dislike getting wet and do not mind carrying umbrellas might always have an umbrella in their bag. Others might carry an umbrella if the chance of rain is 75 percent but not if it is 25 percent. The prediction of rain is independent of preferences about getting wet or being prepared to get wet. Similarly, the AI variation on the notorious “trolley problem” poses an ethical dilemma about life-or-death choices. For example, should a self-driving car swerve to avoid running over four children at the risk of killing its human passenger? If the AI predicts even odds that someone will die either way, the car should swerve if all lives are equally valuable, but it should not swerve if the passenger's life is worth at least four times as much as that of a random child. This somewhat contrived dilemma understates the complexity of the judgment involved. Indeed, the ethical dilemmas of AI reinvigorate longstanding critiques of utilitarian reasoning. As Heather Roff points out, “We cannot speak about ethical AI because all AI is based on empirical observations; we cannot get an ‘ought’ from an ‘is.’ If we are clear eyed about how we build, design, and deploy AI, we will conclude that all of the normative questions surrounding its development and deployment are those that humans have posed for millennia.” If the trolley problem seems far-fetched, consider the case of a self-driving Uber car that killed a cyclist in Tempe, Arizona. The AI had predicted a low but nonzero probability that a human was in its path. The car was designed with a threshold for ignoring low-probability risks. The priority of not hitting humans was obvious enough. Yet, with an error tolerance set to zero, the car would not be able to drive. The question of where to set the tolerance was a judgment call. In this case, it appears that the prespecified judgment was tragically inappropriate for the context, but the prediction machine had absolutely no concept of what was at stake. A well-specified AI utility function has two characteristics. First, goals are clearly defined in advance. If designers cannot formally specify payoffs and priorities for all situations, then each prediction will require a customized judgment. This is often the case in medical applications. When there are many possible situations, human judgment is often needed upon seeing the diagnosis. The judgment cannot be determined in advance because it would take too much time to specify all possible contingencies. Such dynamic or nuanced situations require, in effect, incomplete contracts that leave out complex, situation-specific details to be negotiated later. Because all situations cannot be stipulated in advance, judgment is needed after seeing the prediction to interpret the spirit of the agreement. The military version of incomplete contracting is “mission command,” which specifies the military objective and rules of engagement but empowers local personnel to interpret guidance, coordinate support, and tailor operations as the situation develops. The opposite of mission command, sometimes described as “task orders,” is more like a complete contract that tells a unit exactly what to do and how to do it. Standard operating procedures, doctrinal templates, and explicit protocols help to improve the predictability of operations by detailing instructions for operations and equipment handling. In turbulent environments with unpredictable adversaries, however, standardized task orders may be inappropriate. The greater the potential for uncertainty and accident in military operations, the greater the need for local commanders to exercise initiative and discretion. In Clausewitzian terms, “fog” on the battlefield and “friction” in the organization require commanders to exercise “genius,” which is “a power of judgment raised to a marvelous pitch of vision, which easily grasps and dismisses a thousand remote possibilities which an ordinary mind would labor to identify and wear itself out in so doing.” The role of “genius” in mission command becomes particularly important, and particularly challenging, in modern combined arms warfare and multi-domain operations. When all possible combinations of factors cannot possibly be specified in advance, personnel have to exercise creativity and initiative in the field. Modern military operations tend to mix elements of both styles by giving local commanders latitude in how they interpret, implement, and combine the tools, tactics, and procedures that have been standardized, institutionalized, and exercised in advance.