Automation Bias – AI causes humans to depend Too Much on automation, undermining effectiveness military.
Horowitz, 2019 - Professor of Political Science, University of Pennsylvania [Michael C. May 2“When Speed Kills: Autonomous Weapon Systems, Deterrence, and Stability” https://ssrn.com/abstract=3348356]
Autonomous Weapon System Deployment The history of cruise and ballistic missile development – and specifically the desire for faster and more accurate missiles – illustrates why countries might seek to accelerate the development and deployment of LAWS. In the Cold War, for example, increased accuracy was mostly viewed as a good in itself, or a natural development.27 The promise of being able to operate faster than adversaries on the battlefield, or get inside their decision loops, are also constant goals of planners, as is the ability to use force more precisely. In combination, at the outset, these autonomous characteristics of weapon systems should make their development and deployment more likely. A key factor that will influence the deployment of LAWS is trust in the system. As referenced above, the automatic mode on the Phalanx and related CIWS systems provides an approximation of a lethal autonomous weapon system. Moreover, Iron Dome (Israel) and the PAC- 3 Patriot (US and others) integrate autonomy into targeting and firing decisions. A track record of effectiveness can lead operators to trust them too much through a process called automation bias, as they outsource judgment to algorithms.28 For example, in 2003, the Patriot missile system led to two fratricides in the US-led invasion of Iraq. In both cases, the autonomous characteristics of the system proved too brittle for the real ambiguities of combat, and operators trusted the sensors too much.29