AI undermines command structure because machines cannot exercise leadership when commanders disagree with AI recommendations.
Goldfarb and Lindsay, 2022 – Chair in AI and health care at the Univ of Toronto, Prof of Cybersecurity at the Georgia Tech [Avi, Jon, 2/25/22, https://doi.org/10.1162/isec_a_00425, “Prediction and Judgement: Why Artificial Intelligence Increases the Importance of Humans in War” 6/18/22, LND]
The second characteristic of a well-specified AI utility function is that all stakeholders should agree on what goals to pursue. When it is difficult for people to agree on what to optimize, transparent institutional processes for evaluating or aggregating different preferences may help to validate or legitimate decisions that guide AI systems. Unfortunately, consensus becomes elusive as “genius” becomes more geographically distributed, socially collaborative, and technically exacting. In an ethnography of divisional command in Afghanistan, Anthony King writes that “a general must define a mission, manage the tasks of which it is comprised and motivate the troops.” The first of these three factors—specifying positive objectives and negative limitations— is the consummate function of judgment; AI offers little help here. AI might provide some support for the second factor, oversight and administration, which involves a mixture of judgment and prediction. The third factor is leadership, which is fundamentally a judgment problem insofar as leaders attempt to socialize common purposes, values, and interpretations throughout an organization. Again, AI is of little use for issues of leadership, which become more important as organizations become geographically and functionally distributed: “The bureaucratic expertise of the staff has been improved and their cohesiveness has been condensed so that they are now bound in dense solidarity, even when they are not co-present.” Indeed, “decision-making has proliferated” in all three areas—strategy, management, leadership— because a “commander can no longer direct operations alone.” According to King, the commanding general is now less of a central controller and more of a social focal point for coordinating the complex interactions of “the command collective.” Collective command, however, is a collective action problem. In some cases, standard operating procedures and socialization rituals can simplify judgment tasks. King finds that “command teams, command boards, principal planning groups and deputies have appeared to assist and to support the commander and to manage discrete decision cycles to which the commander cannot attend.” Yet, in other cases, personnel from different services, branches, or units may disagree over how to interpret even basic tactics, techniques, and procedures. Disagreement may turn into controversy when mission assignments fall outside the scope of what professionals deem right or appropriate, as when armies are tasked with counterinsurgency, air forces are tasked with close air support, or cultural preferences clash. More serious disagreements about war aims and military methods can emerge within the civil-military chain of command or among coalition partners. Just as data availability and bias vary for any given decision task, we also expect variability in the clarity and consensus of judgment. Any factors that exacerbate confusion or disagreement in military institutions should be expected to make judgment more difficult for AI automation.