Free beta access. 5 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

Human control fails – our soldiers are not adequately trained for human/machine teams

Goldfarb and Lindsay, 2022 – Chair in AI and health care at the Univ of Toronto, Prof of Cybersecurity at the Georgia Tech [Avi, Jon, 2/25/22, https://doi.org/10.1162/isec_a_00425, “Prediction and Judgement: Why Artificial Intelligence Increases the Importance of Humans in War” 6/18/22, LND]

A fundamental organizational challenge is to recruit, train, and retain the human talent required for human-machine teaming. We anticipate that AI systems will increase the influence of junior personnel, giving more leverage to their judgment and decisions. Yet, we also expect that the junior officers, noncommissioned officers, civilian employees, and government contractors who maintain and operate AI systems will struggle to understand the consequences of their actions in complex political situations. Gen. Charles Krulak highlights the role of “the strategic corporal” on twenty-first-century battlefields. Krulak argues that operational complexity makes tactical actions more strategically consequential, for better or worse, which places a premium on the character and leadership ability of junior personnel. AI will further increase the burden of judgment on them. Forward personnel will have to see the predictions from AI systems, assess whether the data that created the predictions are reliable, and make value judgments about how and why automated systems can advance the mission. Furthermore, AI systems will require constant reconfiguration and repair as the context of human-machine teaming changes during actual operations. Military personnel have long engaged in field-expedient, bottom-up innovation. We expect personnel will likewise hack AI systems to improve mission performance, as they understand it, even as unauthorized modifications put them into conflict with system configuration managers elsewhere in the bureaucracy. It is important to emphasize the human capital requirements of combining a sophisticated understanding of the politico-military situation with the technical savvy to engineer AI in the field. The strategic corporal in the AI era must be not only a Clausewitzian genius but also a talented hacker. This may not be a realistic requirement. The importance of human-machine teaming is increasingly appreciated in organizations that implement AI systems. Amid all the hype about AI and war, plenty of thoughtful work seeks to discern the relative advantages of humans and machines and to devise methods of pairing them together in order to improve decision-making. As the U.S. Department of Defense AI strategy states, “The women and men in the U.S. armed forces remain our enduring source of strength; we will use AI-enabled information, tools, and systems to empower, not replace, those who serve.” Yet, the strategy's stated goal of “creating a common foundation of shared data, reusable tools, frameworks and standards, and cloud and edge services” is more of a description of the magnitude of the problem than a blueprint for a solution. As AI creates potential for large-scale efficiency improvements, it also creates potential for large-scale collective action problems. New military staff specialties are sure to emerge to manage data and judgment resources, creating new institutional equities and integration challenges. Perhaps even more challenging is the problem of nurturing trust among all the engineers, administrators, analysts, operators, and lawyers involved in designing, using, and repairing AI systems. As cheap prediction makes human judgment more vital in a wide variety of tasks, and as more judgment is needed to coordinate human-machine teaming, we anticipate that military bureaucracies will face complicated command decisions about why, and how, to conjoin humans and machines. Commercial firms that embrace AI often adjust their boundaries and business models by contracting out tasks involving data, prediction, and action (e.g., manufacturing, transportation, advertising, and service provision) while developing in-house judgment capacities that are too difficult to outsource. Military organizations, likewise, may find it advantageous to share specialized resources (sensors, shooters, intelligence products, and logistics) across a decentralized network of units, even as they struggle to make sense of it all. AI is thus part of a broader historical trend that has been described with terms like “networkcentric warfare,” “joint force operations,” “integrated multi-domain operations,” and “interagency cross-functional teams.” The whole is more than the sum of its parts, but each part must exercise excellent judgment in how it leverages shared assets. Historical experience suggests that military interoperability and shared sensemaking are difficult, but not necessarily impossible, to achieve. We thus expect military and political judgment will become even more difficult, diffused, and geographically distributed. Indeed, the ongoing involvement of the “strategic corporal” in conversations about politico-military ends could end up politicizing the military. In the United States, as Risa Brooks argues, the normative separation of political ends from military means has some paradoxically adverse consequences: it enables service parochialism, undermines civilian oversight, and degrades strategic deliberation. Greater reliance on AI could exacerbate all these problems, precisely because AI is a force multiplier that requires military personnel to exercise greater judgment. Brooks's argument implies that an AI-intensive defense bureaucracy could become both more powerful and more politically savvy. If machines perform the bulk of data gathering, prediction, and tactical warfighting, then the judgments of human engineers, managers, and operators will be highly consequential, even as ethical questions of accountability become harder to answer. Some military personnel may be unable to perform at such a high level of excellence, as attested by the many scandals during the wars in Iraq and Afghanistan (from targeting errors to prisoner abuse). Increasing reliance on AI will magnify the importance of leadership throughout the chain of command, from civilian elites to enlisted service members.

Sign in to recut, update, or save this card.