Operator trust is crucial for military effectiveness – if users do not trust AI systems, they cannot adopt them.
Costello, 2020 - Chief of Staff for Test Wing Atlantic [Donald, "Certifying an Autonomous System to Complete Tasks Currently Reserved for Qualified Pilots” - ProQuest https://www.proquest.com/docview/2495035965/fulltextPDF/D926A79B68D249D7PQ/1?accountid=14667 accessed on 6-21-2022 TM]
Trust is vital for certification. When a military commander certifies a pilot as fully qualified they are bestowing their trust on that pilot. Following qualification, the pilot is expected to use his judgment to make decisions based on their experiences. When dealing with autonomy, trust is not inherent and certification is not business as usual. For commanders to trust that an autonomous system will perform as a pilot would will require methods and metrics different than what they are accustomed to. 37 The Defense Science Board identified trust as an integral requirement for the use of autonomous systems by the DoD. “The decision for DoD to deploy autonomous system must be based both on trust that they will perform effectively in their intended use and that such use will not result in high-regret, unintended consequences. Without such trust, autonomous systems will not be adopted except in extreme cases such as mission that cannot otherwise be performed. Further, inappropriate calibration of trust assessments – whether over-trust or under-trust during design, development, or operations will lead to misapplication of these systems. It is therefore important for DoD to focus on critical trust issues and the assurance of appropriate levels of trust [91].” Autonomy is a new concept for the DoD. When employing military systems in the field, someone is ultimately responsible for the actions of that system. This may include the actual military member employing the technology or the individual that certified it for use. Having a system that exhibits non-deterministic behavior inherent in autonomy is a new concept for certification officials. Trust needs to be built prior to the use of autonomy within the DoD.
Operator trust is essential because AI is a black box – operators cannot examine the machine logic for themselves.
Flournoy, Haines et al 2020 - former Under Secretary of Defense for Policy and Director of National Intelligence [Michèle and Avril, Gabrielle Cheftiz Special Assistant to the Under Secretary of Defense for Policy October, “Building Trust through Testing Adapting DOD’s Test & Evaluation, Validation & Verification (TEVV) Enterprise for Machine Learning Systems, including Deep Learning Systems” https://cset. georgetown.edu/wp-content/uploads/Building-Trust-Through-Testing.pdf Acc 6/23/22 JZ]
The black box challenge: unique features of DL traceability and interpretability. It is critical that all ML/DL systems are trustworthy, traceable, and transparent to the greatest extent possible. Deep learning presents challenges for each of these features. Unlike most previous types of computer systems, it may not be possible to trace why a deep learning system made the decision it did in a particular scenario. Not being able to determine what led to an error can obviously create significant challenges for TEVV. It can also undermine user confidence in any solution devised to address the problem identified. Challenges with interpretability in real-time will also hamper human-machine teaming—operators are more likely to trust a system and interact with it effectively if they understand roughly why it is taking certain actions or decisions. Further, the opacity of deep learning systems makes it difficult to identify or trace back certain kinds of adversary attacks, such as some forms of data poisoning. Some forms of attacks are not obvious to human intuition and, therefore, difficult to imagine and test against (i.e., a 3-D printed turtle that fooled Google’s image classifier into classifying it as a rifle.)12 The lack of interpretability, traceability, and explainability of DL systems has the potential to undermine trust and exacerbate challenges associated with developing, deploying, and governing ML/DL at scale.