Free beta access. 50 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

Developing trust in AI is impossible---black box problem means true integration is impossible

Erik Lin-Greenberg 20, postdoctoral fellow at the University of Pennsylvania’s Perry World House. Texas National Security Review, Vol 3, Iss 2. Spring. "Allies and Artificial Intelligence: Obstacles to Operations and Decision-Making" https://tnsr.org/2020/03/allies-and-artificial-intelligence-obstacles-to-operations-and-decision-making/ //pipk

AI can also strain alliance decision-making by fueling uncertainty about information and military actions. Unlike human analysts or military personnel who can be asked to explain and justify their findings or decisions, AI generally operates in a “black box.” 97 The neural networks that underpin many cutting-edge AI systems are opaque and offer little insight into how they arrive at their conclusions.98 These networks rely on deep learning, a process that passes information from large data sets through a hierarchy of digital nodes that analyze data inputs and make predictions using mathematical rules. As data flows through the neural network, the net makes internal adjustments to refine the quality of outputs. Researchers are often unable to explain how neural nets make these internal adjustments. Because of this lack of “explainability,” users of AI systems may have difficulty understanding failures and correcting errors.99

Policymakers have called for the development of more transparent AI systems, and researchers are working to develop explainable AI tools that peer inside the AI black box.100 Yet, many decision-makers remain uncomfortable with the uncertainty surrounding AI-enabled systems. The commander of the U.S. Air Force’s Air Combat Command, for instance, publicly explained that he was not yet willing to rely on AI programs to analyze the full-motion video collected by reconnaissance drones. He argued that although systems are improving, they are still unable to consistently provide accurate analysis.101 So long as the decisions and analysis of AI systems remain opaque, military commanders may be reluctant to trust AI-enabled systems. And if used, AI may contribute to the fog of war, rather than reduce it, making it difficult to make decisions using information delivered by AI technologies.

The operational implications associated with uncertainty and lack of trust in AI would likely be exacerbated in multinational alliance contexts. There is significant cross-national variation in trust in AI technologies, even among close allies. One 2018 survey, for instance, found that just 13 percent of respondents in Japan and 17 percent of respondents in South Korea trust artificial intelligence, compared to 25 percent of respondents in the United States. Similar disparities exist between the United States and many of its NATO allies. In Spain, 34 percent of respondents trust artificial intelligence, compared to 21 percent in Canada, 40 percent in Poland, and 43 percent in Turkey.102 Given this variation, policymakers and commanders from some states may be more reluctant to use AI-enabled systems or trust the information they deliver than leaders from other states during multinational operations.

Allied decision-makers will also face uncertainty when confronting a rival’s use of AI-enabled technologies. Leaders will be forced to wrestle with whether to respond to actions carried out by AI-enabled systems — like autonomous aircraft or ships — in the same way as actions carried out by traditionally manned assets. Existing doctrine and law are generally silent on these issues, providing no guidance on the appropriate response. States have drafted domestic policies to govern their own use of autonomous weapon systems, but these regulations and international law make no distinction between how states should react to a rival’s AI-enabled military actions versus “traditional” military actions.103 Yet, decision-makers may believe that a rival’s use of AI technologies demands different responses than those involving manned platforms.104 What happens if a rival claims that an attack carried out by an AI-enabled system was the result of a flawed algorithm? Should air defense forces respond differently to an adversary’s autonomous drones that penetrate friendly airspace than to a manned aircraft that does the same? Decision-makers may find themselves with little time to consider these complicated issues, particularly as AI technology accelerates the speed of a rival’s military operations.

Sign in to recut, update, or save this card.