Free beta access. 5 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

AI systems need human control – only human correction can solve for unanticipated failures

Scharre, 2016 -- Vice President and Director of Studies at CNAS [Paul, Feb 2016, Center for New American Security, “Autonomous weapons and operational risk”, https://s3.us-east-1.amazonaws.com/files.cnas.org/documents/CNAS_Autonomous-weapons-operational-risk.pdf?mtime=20160906080515&focal=none, Acc 6/21/22, M. A.]

Maintaining effective control over autonomous systems Human operators will want to ensure that autonomous systems perform in a way consistent with their intentions. Because autonomous systems in many cases do not have real-time human supervision, maintaining effective control over the system has two components: 1. The ability of the human operator to accurately predict the autonomous system’s behavior in the environment in which it is being used. This includes its limitations and the conditions under which it will fail. This allows the human operator to employ the autonomous system only in situations where it will perform appropriately. 2. The ability of the human operator to undertake corrective action if/when the autonomous system fails to behave in accordance with the human operator’s intentions. A failure with an autonomous system is a loss of effective control—a situation in which the autonomous system no longer is behaving in accordance with human operator intention. Risk, in this context, refers to the risk of failure, both the probability and consequences of a loss of control: • The probability of failure is the likelihood of the system behaving in a manner inconsistent with human operator intentions in a particular environment. • The consequence of failure is the potential damage the autonomous system could do in that environment until such time as the human operator can undertake corrective action to bring the system back in line with human operator intentions or the autonomous system ceases operation. Autonomous systems can vary in the type of task they perform, their level of complexity, and degree of the human operator’s interaction with the system. As these aspects of the system change, the risks of employing an autonomous system change as well. The inherent hazard of a system depends on the task being performed Autonomous systems can perform a wide variety of tasks, from driving cars to regulating temperature or making toast. The inherent hazard of a system is the potential consequence if the autonomous system performs that task incorrectly. This depends on both the task being performed and its operating environment. The consequences of a failure with an autonomous car are far more potentially severe than a toaster failing to properly cook bread. The environment in which the system is operating is also a key component of the inherent hazard of the system. The hazard associated with an autonomous car driving on a closed-circuit track is much less severe than one driving through crowded city streets with pedestrians.

Sign in to recut, update, or save this card.