Continuous TEVV is necessary for AI because initial tests will still have bugs in the testing programs themselves
Haugh, 2018 - project leader for the Institute for Defense Analysis [Brian A., September “The Status of Test, Evaluation, Verification, and Validation (TEV&V) of Autonomous Systems,” https://www.ida.org/-/media/feature/publications/t/th/the-status-of-test-evaluation-verification-and-validation-of-autonomous-systems/p-9292.ashx, 6/23/22 MD]
6. Elevated safety concerns and asymmetric hazard Traditionally, TEV&V personnel have relied on the training and common sense of equipment operators to provide many kinds of safety assurance, both in the field and on the test range. Autonomous systems potentially take many of the decisions underlying routine safety out of the hands (and minds) of operators, and depend instead on complex software that allows the system to “operate” itself. During Developmental Test and Evaluation and into Operational Test and Evaluation, it is likely that the software will still contain major bugs and that the algorithms and training data being used might not be the final choices. This creates a potential for various kinds of mischief—especially for weapon systems, highly-mobile systems, or other systems that could be dangerous in the hands of an unreliable operator. Similarly, the most striking successes of AI, machine learning, and autonomous systems to date have occurred in contexts where the cost of error is low. For autonomous military systems, this will not generally be the case—designating incorrect targets, responding to spurious cyber attacks, misidentifying 2-3 individuals or objects, crashing unmanned vehicles into obstacles, and other potential failure modes of autonomous systems are all potentially very costly. Apart from driverless car efforts, there are few active research programs today for applications with highly asymmetric hazard functions.