Free beta access. 5 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

Human control cannot solve all risks – delay in response times increases the chance of mistakes

Scharre, 2016 -- Vice President and Director of Studies at CNAS [Paul, Feb 2016, Center for New American Security, “Autonomous weapons and operational risk”, https://s3.us-east-1.amazonaws.com/files.cnas.org/documents/CNAS_Autonomous-weapons-operational-risk.pdf?mtime=20160906080515&focal=none, Acc 6/21/22, M. A.]

In a semi-autonomous system, after each task the machine stops and waits for human approval before continuing. Under this control type, the human operator has the ability to observe the machine’s actions in the environment and confirm that the behavior is appropriate before continuing.7 In supervised autonomous operation, in principle the human operator has the ability to intervene if necessary. In practice, there is likely to be some time delay between when a failure occurs and when the human is able to actually correct the behavior of the system. This could occur because of a time delay in communications or because it may take the human some period of time to understand that the system is performing inappropriately and determine the appropriate corrective action. For fully autonomous systems, the human operator lacks the ability to observe the autonomous system’s behavior and undertake corrective action in sufficient time if the system fails to perform appropriately. Presumably at some point in time the human operator will become aware of how the system performed. For example, a household thermostat is operating “fully autonomously” while one is away from the home. Once one returns home, one discovers whether the thermostat was performing as one intended or not.8 Thus a key element of risk in autonomous systems is the time between when a system begins failing (performing in a manner other than what the human operator intended) and when the human operator can undertake corrective action. Even in fully autonomous systems, presumably the system ceases operation at a certain point in time once the task is complete and the results of its actions can be observed. The damage potential of a system depends on its inherent hazard and the time from failure to corrective action When it comes to risk, we are concerned with an autonomous system’s damage potential. Damage potential is the amount of damage an autonomous system could do, if it failed to perform appropriately, before a human operator could take corrective action. Damage potential depends upon the inherent hazard of the system—the type of task being performed and the environment in which it is operating—as well as the control type. For supervised autonomous systems, the speed of the system’s operation and any potential time delays also are significant factors. A system that in principle has a human on the loop to intervene in the event of system failure might in practice still have a high damage potential if the system performs tasks much more rapidly than human operators can react.

Sign in to recut, update, or save this card.