Free beta access. 5 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

No Solvency - TEVV does not pair well with formal methods of research

Haugh, 2018 - project leader for the Institute for Defense Analysis [Brian A., September “The Status of Test, Evaluation, Verification, and Validation (TEV&V) of Autonomous Systems,” https://www.ida.org/-/media/feature/publications/t/th/the-status-of-test-evaluation-verification-and-validation-of-autonomous-systems/p-9292.ashx, 6/23/22 MD]

3. Potential Approaches to Overcome These Challenges Academic and government researchers have begun work on a number of techniques and methodologies that might help to overcome these core challenges. Current efforts fall into four primary categories: formal methods, cognitive instrumentation, adversarial testing, and run-time monitoring. We discuss each of these below, then consider the question of how well they cover the full set of capability gaps implied by the challenge list. A. Formal Methods 1. Summary Formal methods in software development allow developers to specify certain properties that the software should have, produce the software, and verify that it does have those properties without needing to confirm that empirically by testing for them. Properties to be specified might be things like: Property 1: the weapon cannot fire while turret is still rotating Property 2: the course-of-action selector can never get into an infinite loop There are two approaches to formal methods for autonomous systems verification: (1) formal methods can be used after the fact as an analytic tool to verify some properties of existing software, or more importantly, (2) formal methods can be used as a design and development process that can assure much more about the behavior of the software to be developed. Formal methods of the second kind are most commonly used in the development of complex safety-critical or security-critical systems or for expensive one-time development efforts (e.g., deep space probes). Applying formal methods to complex AI and autonomous systems is a natural extension of this. 2. Limitation Although formal methods can be extremely useful, there are significant constraints on the current state of the art. These include the following: Scalability: There are currently fairly tight bounds on the size of development effort (or state space) that the techniques can be applied to. 3-2 Scope: Not all desired properties can be assured through formal methods, and there may be performance trades associated with achieving assurance. Rigidity: Any change to a system developed using formal methods risks invalidating the assurance proofs, unless the formal methods are reapplied to the new specification. Given that not all desired behaviors can be assured using formal methods, there are also open research questions concerning how to combine formal methods with empirical TEV&V techniques or run-time monitoring strategies.

Sign in to recut, update, or save this card.