Free beta access. 5 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

AI makes systems vulnerable to mistakes or cyber attacks because they are brittle and lack operator trust.

Nurkin and Konaev, 2022 - senior fellows at the Center for Strategy and Security at the Atlantic Council [Tate and Margarita, 5-25-2022, “Eye to eye in AI: Developing artificial intelligence for national security and defense” Atlantic Council https://www.atlanticcouncil.org/in-depth-research-reports/report/eye-to-eye-in-ai/ ARD]

There are, however, important limitations and drawbacks to AI systems—particularly in operational environments—in large part, because of their brittleness. These systems perform well in stable simulation and training settings, but they can struggle to function reliably or correctly if the data inputs change, or if they encounter uncertain or novel situations. ML systems are also particularly vulnerable to adversarial attacks aimed at the algorithms or data upon which the system relies. Even small changes to data sets or algorithms can cause the system to malfunction, reach wrong conclusions, or fail in other unpredictable ways.20 Another challenge is that AI/ML systems do not typically have the capacity to explain their own reasoning, or the processes by which they reach certain conclusions, provide recommendations, and take action, in a way that is evident or understandable to humans. Explainability—or what some have referred to as interpretability—is critical for building trust in human-AI teams, and is especially important as advances in AI enable greater autonomy in weapons, which raises serious ethical and legal concerns about human control, responsibility, and accountability for decisions related to the use of lethal force. A related set of challenges includes transparency, traceability, and integrity of the data sources, as well as the prevention or detection of adversary attacks on the algorithms of AI-based systems. Having visibility into who trains these systems, what data are used in training, and what goes into an algorithm’s recommendations can mitigate unwanted bias and ensure these systems are used appropriately, responsibly, and ethically. All these challenges are inherently linked to the question of trust explored later in the report.

Sign in to recut, update, or save this card.