TEVV is key to effective autonomous systems because it will provide an internal model of how AI perceives its environment
Haugh, 2018 - project leader for the Institute for Defense Analysis [Brian A., September “The Status of Test, Evaluation, Verification, and Validation (TEV&V) of Autonomous Systems,” https://www.ida.org/-/media/feature/publications/t/th/the-status-of-test-evaluation-verification-and-validation-of-autonomous-systems/p-9292.ashx, 6/23/22 MD]
1. Instrumenting machine thinking To diagnose the causes of incorrect behavior or inadequate performance, it will be necessary determine whether the problem lies in the Perception, Reasoning, or (course of action) Selection functions of the autonomous system—even after it has been established that the problem is not in the sensor hardware or signal processing. It will also be necessary to distinguish coding errors from inadequate algorithms or bad training data. Without the ability to instrument and monitor internal states of the autonomy, diagnosing problems will be slow at best and impossible at worst. 2. Linking system performance to autonomous behaviors In complex collaborative activities, it can be very difficult to determine what enables (or hinders) success. For example, on a soccer or basketball team it can be very difficult to pinpoint which players (and which behaviors) are leading to wins and losses. To design and improve autonomous systems, it will be necessary to understand how the system’s various autonomous capabilities interact to enable (or hinder) mission execution. The requirements specification for autonomous behavior is often a problem here due to incomplete specifications based on an analogy with human behavior. 2. Linking system performance to autonomous behaviors 3. Comparing AI models to reality Autonomous systems represent reality through stylized internal models. Perception provides inputs for these models; Reasoning allows them to be expanded and corrected. The ability of an autonomous system to do its mission will depend on the degree to which the internal modeling of reality supports accurate Perception, valid Reasoning, and effective Selection. This will not generally be a function of how detailed the models are (“high resolution”) or even of how closely the models mirror reality (“high fidelity”)—it will be a function of whether the right kind of information is incorporated into the model and that the resolution and fidelity be enough to support the mission needs. TEV&V will necessarily include prototyping and experimentation to determine 2-2 what kind of internal model, using what kind of representation, is needed to achieve both performance and dependability.