TEVV is key to operator Trust in AI – it ensures Explainability – it allows humans to understand AI decisions
Haugh, 2018 - project leader for the Institute for Defense Analysis [Brian A., September “The Status of Test, Evaluation, Verification, and Validation (TEV&V) of Autonomous Systems,” https://www.ida.org/-/media/feature/publications/t/th/the-status-of-test-evaluation-verification-and-validation-of-autonomous-systems/p-9292.ashx, 6/23/22 MD]
B. Cognitive Instrumentation and Explainable AI 1. Summary The key distinguishing feature of autonomous systems is that they make decisions, when interpreting their environments and selecting courses of action. Test and evaluation of autonomy will depend critically on the ability to assess the quality of this decision making capability. In general, observed high-level system performance in limited test scenarios will not be sufficient to validate and verify autonomous decision-making in the ways necessary for successful development and deployment, especially for systems designed to team with humans. Instead, novel instrumentation approaches will be required to diagnose and characterize: • Adequacy of architectures and algorithms (including machine learning). • Adequacy and appropriateness of training data. • Effectiveness of operational concepts for HMT. 2. Why Explanations Are Needed Explanations of system behavior support at least four distinct goals for military systems: 1. Diagnosis: Knowing why the system is exhi biting undesired behavior is the first step toward fixing it. 2. Prediction: Being able to forecast how the system will behave in given circumstances is essential to effective employment of the system. 3. Bounding: Understanding the limits of dependable performance allows formulation of tactics/techniques/procedures for how the system is to be used and identification of where run-time monitoring of system state may be the only way to avoid undesirable behaviors. 3-3 4. Trust: If humans are teaming with autonomous machines, the overall performance of the human-machine system may depend on how well the humans feel they understand the reasons behind the machine’s behaviors. For traditional software employing procedural algorithms, the logic of the algorithm traditionally serves as the explanation. Describing the flow of control (e.g., using pseudocode or flowcharts) at various levels of description could “explain” the behavior of the system. As systems have grown in complexity, the ability of humans to understand the logic of the systems in a way that counts as an “explanation” has eroded. Machine learning breaks this paradigm completely by replacing comprehensible procedural logic with a trained ability to generate outputs from inputs in a manner that looks much more like an intuition or hunch than like reasoning. The relationship between input and output is fundamentally opaque; there is no procedural logic to be traced. Producing surrogate procedural descriptions that summarize how the system is reaching its conclusions in ways that can function as explanations for purposes of diagnosis, prediction, bounding, and trust requires additional work and access to the internal states (and possibly the training algorithm) of the machine-learning module. (Note that different explanatory frameworks will generally be needed for each of these goals—the explanation appropriate for diagnosis may be very different from the explanation required for bounding or trust.) The US Defense Advanced Research Projects Agency (DARPA) has recently launched a funded research program in “Explainable Artificial Intelligence” (XAI). The four goals mentioned above (diagnosis, prediction, bounding, and trust) are all represented in the Explainable AI program. “Explainability” in the XAI program involves multiple measures of effectiveness, depending on which of the goals is being pursued. These measures can include human satisfaction with the explanation, how helpful the explanation is in choosing how to interact with the autonomous system, how well the explanation supports diagnosis of unexpected behaviors, or how well the explanation predicts future system behavior. 3. Cognitive Instrumentation The phrase cognitive instrumentation refers to any measurement of internal system states of the software modules that provide the autonomous capabilities of a system. There is an obvious analogy with physical measurement of internal system state (e.g., temperature, pressure, voltage, torque) during development, test, and evaluation of hardware systems. As with those physical analogues, the measurements may be only for TEV&V purposes, or they may be incorporated into the design of the system as run-time monitors on behaviors that cannot be adequately assured through other means. One area of explainability research involves designing autonomous systems to selfreport their reasons for their behaviors. This is a promising approach, but it does not avoid 3-4 the need for instrumenting the internal states of the system. At a minimum, verification and validation of the self-reporting system will involve the same diagnosis, prediction, and trust issues as the mission behavior of the system, with an associated need to look inside the black box and make sense of what is happening there. At some point, all system development requires knowing the truth about what is happening inside the system. In the special case of HMT, where the machines have significant autonomous capabilities, cognitive instrumentation will include measurements of both the machines and the humans, as well as the interface between them. Measurement and characterization of trust by humans is already a rich area of academic research. Measurement and characterization of machine understanding of human intent is also receiving increasing attention. There are also active research programs exploring the design of HMT protocols. Some of these focus on the optimal allocation of work between humans and machines in various contexts. Others are concerned with optimal communications protocols—under what circumstances should humans or machines volunteer information (and which information), make suggestions, or ask for guidance? Without cognitive instrumentation, those optimization programs could easily devolve into guesswork and trial-and-error. Cognitive instrumentation is a necessary condition for Explainable AI; any valid explanation of how the AI is thinking or why it is behaving a certain way must be based on accurate measurements of its internal states. However, the measurements themselves are not the explanation. A complete state description will generally be so complex that it is not understandable, whereas too small a subset of measurements will generally not be sufficient to support explanation. Additional effort will be required to identify a minimal sufficient set of measurements that can be used as inputs to explanatory models that can be understood in human terms for diagnosis, prediction, bounding, and developing appropriate operator trust. There are many tricky steps between “the weights in layer 17 of the neural network are as follows” to “the system seems to be recognizing cats by their ears and coloration.” As noted above, different explanatory models will be needed for the different goals—an explanatory model supporting useful diagnosis for developers may be much more complex (and unwieldy) than a model that produces explanations of mission behavior for operators or commanders.