Autonomous systems face inherent failure risks from malfunctions and bugs
Scharre, 2016 -- Vice President and Director of Studies at CNAS [Paul, Feb 2016, Center for New American Security, “Autonomous weapons and operational risk”, https://s3.us-east-1.amazonaws.com/files.cnas.org/documents/CNAS_Autonomous-weapons-operational-risk.pdf?mtime=20160906080515&focal=none, Acc 6/21/22, M. A.]
The risk in employing an autonomous system is that the system might not perform the task in a manner that the human operator intended. There are a number of reasons why an autonomous system might begin performing inappropriately, from simple malfunctions and software bugs to more complex system failures, changing environmental conditions, hacking, and human error. When these failures can be anticipated in advance, human operators can account for these limitations. When failures are unanticipated, however, the result can be autonomous systems that slip out of control. Understanding the likelihood and consequences of a loss of control is essential to assessing the risk in employing autonomous systems.