Free beta access. 50 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

The patriot system is faulty---Iraq proves.

Atherton, 22 [Kelsey Atherton is a military technology journalist based in Albuquerque, New Mexico. His reporting has appeared in Popular Science, Breaking Defense, and The New York Times, 5-6-2022, accessed on 6-28-2022, Brookings, "Understanding the errors introduced by military AI applications", https://www.brookings.edu/techstream/understanding-the-errors-introduced-by-military-ai-applications/]/ISEE

On March 22, 2003, two days into the U.S.-led invasion of Iraq, American troops fired a Patriot interceptor missile at what they assumed was an Iraqi anti-radiation missile designed to destroy air-defense systems. Acting on the recommendation of their computer-powered weapon, the Americans fired in self-defense, thinking they were shooting down a missile coming to destroy their outpost. What the Patriot missile system had identified as an incoming missile, was in fact a UK Tornado fighter jet, and when the Patriot struck the aircraft, it killed two crew on board instantly. The deaths were the first losses suffered by the Royal Air Force in the war and the tragic result of friendly fire. A subsequent RAF Board of Inquiry investigation concluded that the shoot-down was the result of a combination of factors: how the Patriot missile classified targets, rules for firing the missiles, autonomous operation of Patriot missile batteries, and several other technical and procedural factors, like the Tornado not broadcasting its “friend or foe” identifier at the time of the friendly fire. The destruction of Tornado ZG710, the report concluded, represented a tragic error enabled by the missile’s computer routines. The shoot-down of the Tornado happened nearly 20 years ago, but it offers an insight into how AI-enabled systems or automated tools on the battlefield will affect the kinds of errors that happen in war. Today, human decisionmaking is shifting toward machines. With this shift comes the potential to reduce human error, but also to introduce new and novel types of mistakes. Where humans might have once misidentified a civilian as a combatant, computers are expected to step in and provide more accurate judgment. Across a range of military functions, from the movement of autonomous planes and cars to identifying tanks on a battlefield, computers are expected to provide quick, accurate decisions. But the embrace of AI in military applications also comes with immense risk. New systems introduce the possibility of new types of error, and understanding how autonomous machines will fail is important when crafting policy for buying and overseeing this new generation of autonomous weapons. What went wrong in 2003 The Patriot missile began development in the 1960s, when the U.S. Army sought a means to reliably shoot down enemy airplanes. Later, the missile would gain the ability to also intercept other missiles, and as the roles assigned to the missile expanded, its autonomous capabilities increased. Patriot missile batteries use a phased array radar to detect and identify targets. This information is then fed into a computer control station to manage how the missiles are launched in response. Once fired, the missiles fly toward an identified intercept point calculated before firing, directions that can be altered by sending updated sensor readings over radio signal to the fired missile. As it approaches for impact, the missile’s own radar tracks the target. Raytheon, which manufactures the Patriot, has described the system as having “automated operations” with “man-in-the-loop (human) override” capabilities—technology that allows the weapon to quickly engage targets with the necessary speed to carry out its missile defense mission. Automation is a compelling feature for an anti-air and, especially, for an anti-missile system. The calculations involved in shooting down aircraft and missiles are hard and require immediate translation of sensor information. Both interceptors and targets are traveling exceptionally fast. It’s the kind of task in which the involvement of a human introduces lag, slows down the process, and makes it less likely a missile is going to successfully shoot down an incoming projectile or aircraft. But human operators also serve an essential role: preventing accidental, incorrect shootdowns. And this requires a balance between human and machine decisionmaking that is difficult to achieve. When the Pentagon investigated the causes of the Tornado shootdown, as well as two other incidents of friendly fire involving Patriot systems, the missile system’s automated functions were identified as contributing factors in misidentifying friend as foe. U.S. Patriot batteries deployed to Iraq under the assumption that they would face heavy missile attacks, which would require the batteries to operate with a relative degree of autonomy in order to respond with sufficient speed. As a 2005 report by the Defense Science Board Task Force on the Patriot system’s performance observed, operating autonomously required U.S. forces to trust that the automated features of the system were functioning properly. So when the assumptions underlying the decision to allow the Patriot system to autonomously identify and sometimes fire on targets no longer applied, the soldiers operating the system were not in a position to question what the weapon’s sensors were telling them. Had U.S. and coalition forces faced heavy missile attacks in the war, automating such defenses would have made more sense. Instead, U.S. and allied forces quickly established air superiority, enough to drastically shift the balance of what was in the sky. Instead of facing large amounts of incoming missiles, Patriot batteries were observing large numbers of allied planes operating in the sky above them and sometimes struggling to identify friend from foe. According to the Defense Science Board’s task force, the first 30 days of combat in Iraq saw nine ballistic missile attacks that Patriot batteries might have been expected to counter, compared to 41,000 aircraft sorties, amounting to a “4,000-to-1 friendly-to-enemy ratio.” Picking out the correct targets against the background of a large number of potential false positives proved highly challenging.

Sign in to recut, update, or save this card.