Adversarial testing will improve TEVV because it helps identify vulnerabilities
Haugh, 2018 - project leader for the Institute for Defense Analysis [Brian A., September “The Status of Test, Evaluation, Verification, and Validation (TEV&V) of Autonomous Systems,” https://www.ida.org/-/media/feature/publications/t/th/the-status-of-test-evaluation-verification-and-validation-of-autonomous-systems/p-9292.ashx, 6/23/22 MD]
C. Adversarial Testing As noted in Challenge 6, many defense applications will require very low probabilities of high-cost events occurring. For systems with extremely large and highly nonlinear state spaces, it will not be possible to provide that assurance statistically using traditional Design of Experiments techniques. Adversarial testing, especially when combined with machine learning and automated test design methods, provides a potentially more efficient means to identify and eliminate potential failure modes. In this approach, analytical techniques are used to identify test scenarios in which the system is most likely 3-5 to perform unacceptably and focuses testing in those portions of the state space to maximize diagnostic information and inform robust design. As a specific example, one could apply reinforcement learning to the selection of environmental factors as input to the test cases in a modeling and simulation driven test bed that included the AI software running the autonomous systems. The environment could be treated as a “thinking adversary” in an asymmetric game. The environment would learn optimal strategies to defeat the autonomous system, even as the autonomous system learned improved strategies to counter the environment. This would both maximize the chance of finding key vulnerabilities and weaknesses and help discover ways to mitigate or avoid those vulnerabilities. The vulnerabilities need not be isolated within the autonomous system itself—they might also arise from the proposed CONOPS, especially in the case of significant HMT. They could also be associated with bias or incompleteness in the training data used in supervised learning to develop the autonomous capabilities. Cognitive instrumentation would be needed to distinguish inadequate algorithms or models from inadequate CONOPS or flawed training. Finally, we note that this approach requires domain expertise as well. The operating environment and mission must be well enough understood so that the apparent “weaknesses” are not associated with impossible circumstances. If the weakness occurs only during heavy rainfall at temperatures below –40° C, it will be a weakness we can live with. More subtly, the domain expertise will need to be sufficient to estimate probability of occurrence of the conditions that exposed the weakness. As mentioned before, utilization in the defense sector will require attention to the likelihood of seriously adverse outcomes.