Dozens of structural factors within the DoD make AI data-sharing with other countries impossible---storage, bandwidth, interoperability, incomplete documentation, murky ownership
Tarraf et al. 19. Dr. Danielle C. Tarraf is a Senior Information Scientist at the RAND Corporation. She was previously on the Electrical and Computer Engineering faculty at Johns Hopkins, and has been a visiting/summer faculty fellow at the Air Force Research Lab. She is the recipient of multiple awards, including an NSF CAREER award in 2010 and an AFOSR Young Investigator award in 2011. She received her Ph.D. from MIT in 2006; “The Department of Defense Posture for Artificial Intelligence”; 2019; RAND Corporation; https://www.rand.org/content/dam/rand/pubs/research_reports/RR4200/RR4229/RAND_RR4229.pdf //BY
Data The study team interviewed and collected input from many DoD stakeholders with responsibility for data and infrastructure. The team also interviewed DoD personnel who use these data and infrastructure for analytics or AI purposes. One striking aspect of our interviews was the general enthusiasm expressed for the JCF, with “wish lists” that included the development of uniform policies for sandboxing and application programming interfaces,32 the development of common algorithm libraries and repositories for open source projects, and reuse of code components and documentation of best practices (see section “Organization: At the OSD Level” in Appendix B). A notable development within DoD is the recent creation of the CDO role, with Michael Conlin in place as DoD’s first CDO as of August 2018. We anticipate that with time, the data posture of DoD will significantly evolve. Nonetheless, through our interviews, we identified several serious obstacles and impediments in regard to data at present. Data are not collected and stored at every opportunity. DoD’s software infrastructure exists in an environment in which storage space remains a scarce resource and many opportunities to record data are missed. Even if modern storage infrastructure capacities were acquired, additional barriers might prevent the mass collection and storage of 32 Sandboxing refers to providing an isolated environment for experimentation with software. Sandbox environments would ideally have some amount of test data and computing resources but are set up so that what happens in the sandbox is isolated from the production environment that supports actual real-world operations. 58 The Department of Defense Posture for Artificial Intelligence data. In particular, DoD still faces significant constraints on network bandwidth, which can hamper the ability to move data collected from sensors in the field to a location where they can be stored. Additionally, DoD’s suite of software was designed for and implemented in an era in which applications stored only data for which DoD had an immediate use, rather than speculatively storing data that could be mined for insights by professionals. This situation has resulted both in inadequate data storage and in storage of less-appropriate data; in addition, too much of the data collected have already been transformed from a raw, foundational form into an intermediate or aggregated result. These transformations, although an appropriate optimization at the time, strip away crucial context from information needed to train modern ML-based algorithms. Finally, outdated collection processes result in some data not being collected and digitized; recent RAND work on acquisition data within DoD illustrates this point.33 Access to existing data is limited. Several barriers within DoD present substantial obstacles to data-sharing today. First, personnel might view data as a means of retaining power or value in DoD or of protecting their work from extensive oversight, and therefore those personnel resist data-sharing. Additionally, many data owners resist sharing their data out of security concerns and the worry that another organization might suffer a security breach. Finally, the security clearance process and other bureaucratic procedures can introduce a significant lag before an individual will be allowed to access data. This problem presents a particular difficulty in recruiting new talent into DoD, and substantially lowers productivity, even for long-standing DoD personnel. Some of these issues, as they pertain to acquisition data, were highlighted in a recent RAND report.34 Informal networks can substantially reduce the delay in gaining access to data, but based on our interviews, these workarounds appear to be haphazard. Lack of interoperability in systems across DoD creates challenges. Interoperability of data collected by different systems, even within the same functional domain, remains a problem. Software applications within DoD have not typically been designed to work with other DoD applications—even applications in the same functional domain. Consequently, establishing relationships between data collected by one system and data collected by another can be virtually impossible. Even worse, it appears that DoD leadership is often presented with inconsistent values originating from different systems for the same data point, undermining leadership’s willingness to trust data from DoD systems or make decisions based on data at all (see section “Data” in Appendix B). The data that exist are not always understandable or traceable. DoD systems frequently lack the documentation or metadata required to provide context as to what particular data actually mean or how they were generated. For example, an Army database might store a numeric value for the number of tanks at a particular facility, but it might not explain whether this number indicates all functional tanks, all tanks assigned to a particular unit, or all tanks of any status present at the location. Other data values with less descriptive names will frequently be even more difficult to comprehend and use. The lack of any centralized tool for data service means that it is difficult for DoD personnel, even in leadership, to discover what data might be available to inform a question. Instead, discovering new data sources and interpreting them typically requires personal networks or other informal mechanisms. In sum, all indications are that DoD data are not currently being used to their full potential. Overall, the problems we noted here represent a formidable obstacle to implementing AI algorithms,35 even at the level of enterprise AI. Both Project Maven and the JAIC’s preventative maintenance prototype have found that issues around data quality and availability are a primary barrier to progress (see section “Data” in Appendix B). These issues, if left unresolved, will continue to hamper the development and deployment of AI throughout DoD. Additionally, we highlight the following friction point. There is ambiguity in data ownership where external vendors are involved. In theory, data that originate in DoD should be owned by DoD. In practice, when DoD data are analyzed or otherwise worked on by external vendors, the ownership of the data, albeit in a new modified form, becomes murky. This issue was highlighted in recent unpublished RAND work addressing weapon system intellectual property and data rights. This ambiguity of ownership leads to multiple problems beyond the specific ownership and potential loss of control of the data, including vendor lock to retain use of the data, the inability to aggregate data across multiple vendors, and the inability to use the data for additional internal purposes, among others. Although this problem is not unique to DoD, as some of our industry interviewees described (see section “Industry: Data” in Appendix C), it is one that DoD also needs to come to terms with.