Free beta access. 5 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

Human control is key to prevent accidents with autonomous weapons – Empirical examples prove

Atherton 2022 – Military Technology Journalist [Kelsey, 5/6/22, “Understanding the errors introduced by military AI applications”, https://www.brookings.edu/techstream/understanding-the-errors-introduced-by-military-ai-applications/, 6/18/22, LND]

Automation is a compelling feature for an anti-air and, especially, for an anti-missile system. The calculations involved in shooting down aircraft and missiles are hard and require immediate translation of sensor information. Both interceptors and targets are traveling exceptionally fast. It’s the kind of task in which the involvement of a human introduces lag, slows down the process, and makes it less likely a missile is going to successfully shoot down an incoming projectile or aircraft. But human operators also serve an essential role: preventing accidental, incorrect shootdowns. And this requires a balance between human and machine decisionmaking that is difficult to achieve. When the Pentagon investigated the causes of the Tornado shootdown, as well as incidents of friendly fire involving Patriot systems, the missile system’s automated functions were identified as contributing factors in misidentifying friend as foe. U.S. Patriot batteries deployed to Iraq under the assumption that they would face heavy missile attacks, which would require the batteries to operate with a relative degree of autonomy in order to respond with sufficient speed. As a 2005 by the Defense Science Board Task Force on the Patriot system’s performance observed, operating autonomously required U.S. forces to trust that the automated features of the system were functioning properly. So when the assumptions underlying the decision to allow the Patriot system to autonomously identify and sometimes fire on targets no longer applied, the soldiers operating the system were not in a position to question what the weapon’s sensors were telling them. Had U.S. and coalition forces faced heavy missile attacks in the war, automating such defenses would have made more sense. Instead, U.S. and allied forces quickly established air superiority, enough to drastically shift the balance of what was in the sky. Instead of facing large amounts of incoming missiles, Patriot batteries were observing large numbers of allied planes operating in the sky above them and sometimes struggling to identify friend from foe. According to the Defense Science Board’s task force, the first 30 days of combat in Iraq saw nine ballistic missile attacks that Patriot batteries might have been expected to counter, compared to 41,000 aircraft sorties, amounting to a “4,000-to-1 friendly-to-enemy ratio.” Picking out the correct targets against the background of a large number of potential false positives proved highly challenging. In the case of the Tornado shootdown, automation—and the speed with which automated action was taken—was likely sufficient on its own to cause the tragedy, but it might have been prevented if other systems hadn’t failed. As the UK Ministry of Defence concluded in its examining the incident, the battery culpable for the shootdown was without its communications suite, which was still in transit from the United States. Contact with battalion headquarters occurred through a radio relay with another battery equipped with voice and data links to headquarters. “The lack of communications equipment meant that the Patriot crew did not have access to the widest possible ‘picture’ of the airspace around them to build situational awareness,” the report found. Another system that failed and that might have prevented the shootdown was the identification-as-friend-or-foe system, a safety measure designed to avoid such deadly mistakes. That kind of information, transmitted securely and immediately, could have prevented an automated system from shooting down the jet. If the information was communicated to the human crew operating the Patriot battery, it would have been a signal to call off the attack. Tragically, the IFF transponder or the Patriot battery’s ability to receive such a signal failed. While it is tempting to focus on the automated features of the Patriot system when examining the shootdown—or autonomous and semi-autonomous systems more broadly—it is important to consider such weapons as part of broader systems. As policymakers consider how to evaluate the deployment of increasingly autonomous weapons and military systems, the complexity of such systems, the ways in which they might fail, and how human operators oversee them are key issues to consider. Failures in communication, identification, and fire-control can occur at different points of a chain of events, and it can be difficult to predict how failures will interact with one another and produce a potentially lethal outcome. The Defense Science Board’s examination of the Patriot concluded that future conflicts will likely be “more stressing” and involve “simultaneous missile and air defense engagements.” In such a scenario, “a protocol that allows more operator oversight and control of major system actions will be needed,” the task force argued. 

Sign in to recut, update, or save this card.