Free beta access. 5 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

Brittle AI causes accidents and miscalculation – Machine learning breaks down when the real world does not act like their simulations

Flournoy, Haines et al 2020 - former Under Secretary of Defense for Policy and Director of National Intelligence [Michèle and Avril, Gabrielle Cheftiz Special Assistant to the Under Secretary of Defense for Policy October, “Building Trust through Testing Adapting DOD’s Test & Evaluation, Validation & Verification (TEVV) Enterprise for Machine Learning Systems, including Deep Learning Systems” https://cset. georgetown.edu/wp-content/uploads/Building-Trust-Through-Testing.pdf Acc 6/23/22 JZ]

ML/DL systems are not robust and it is difficult to characterize system performance. A fundamental challenge of ML/DL is its brittleness; it has trouble functioning correctly if the inputs or environmental conditions change. However, testing these systems in all possible scenarios and with all ranges of inputs is simply not feasible. It is nearly impossible to predict all the ways a system could break, or an adversary could manipulate or spoof it, which is partly why these systems may be especially vulnerable to adversarial attacks. As these systems become more sophisticated, as is the case with deep learning, their output becomes even less transparent, making it harder to determine the conditions under which they might fail and what steps could correct system behavior. Even when operating under the best conditions (within the same distribution of inputs or environmental conditions present during training), DL models generally don’t work to the reliability standards needed for safety-critical systems,10 and therefore may not be appropriate for these applications, at least given the current state of the technology. ML/DL systems are also particularly vulnerable to operational edge cases (both unintended and intended), which are cases that occur beyond the bounds of a system’s operational envelope or normal operating parameters. Because it is very difficult to characterize the actual performance envelop of these systems, it will be important to prioritize stress testing system performance with boundary conditions. Further, interactions within and between systems (including foreign autonomous systems) can induce unintended consequences and are even more complex to predict or understand. The potential for unintended engagement or escalation is even greater when U.S. and/or adversary systems have the sorts of advanced autonomy features that deep learning can enable, and their interaction cannot be studied or fully tested in advance of deployment.

Sign in to recut, update, or save this card.