Free beta access. 50 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

Creating new evaluation procedures to measure AI innovations prevents centralization of decision-making

Stefano Costalli 21, Associate Professor of Political Science in the Department of Political and Social Sciences, University of Florence, Italy and Research Fellow of the Michael Nicholson Centre for Conflict and Cooperation, University of Essex. “NATO Decision-Making in the Age of Big Data and Artificial Intelligence” Editors: Sonia Lucarelli; Alessandro Marrone; and Francesco Niccolò Moro. Sonia Lucarelli is Professor of International Relations and European Security at the University of Bologna, and member of the Board of Directors of the Istituto Affari Internazionali (IAI). Alessandro Marrone is Head of the Defence Programme of IAI and teaches at the Istituto Superiore di Stato Maggiore Interforze (ISSMI) of the Italian Ministry of Defence. Francesco N. Moro is Associate Professor of Political Science at the University of Bologna and Adjunct Professor of International Relations at the Johns Hopkins University Europe Campus. This publication is the result of the Conference “NATO Decision-making: promises and perils of the Big Data age”, organized by NATO Allied Command Transformation (ACT), the University of Bologna and Istituto Affari Internazionali (IAI) of Rome. https://www.iai.it/sites/default/files/978195445000.pdf //pipk

First of all, the WG defined the main features of Big Data, which set the terms to evaluate any possible organizational innovation and decision-making method. The first and most apparent characteristic of Big Data is the huge quantity of information available. The high-speed at which the data are generated and need to be processed is another defining factor that needs to be taken into account, in addition to the fact that data will typically be acquired from diverse sources and will have different formats. Moreover, the trustworthiness of the data has to be carefully evaluated, and finally any data can have different value in different phases of the decision-making process. All these features impose specific requirements on organizations that aim at using Big Data to reduce the uncertainty in which they operate. For instance, the huge volume of data compels organizations to acquire new data storage capabilities, while the high-speed demands new processing tools and the variable trustworthiness and value compel organizations to elaborate new methods of analysis. Most importantly, all these innovations must be implemented together, because otherwise the Big Data could not be fruitfully exploited and, on the contrary, could create new problems to the organizations, exposing them to dangerous short circuits.

Considering the available literature and exchanging views among the members of the WG, a general consensus emerged according to which any organization that seeks to exploit Big Data should have clear goals and a well-defined strategy to delineate and implement their specific objectives concerning these tools. This is essential to have a clear understanding of the domains and levels where the organization could profit most from the use of Big Data, which types of data are needed, and what investments will have to be made in order to achieve the organization’s goals. Founding the organization’s decision-making process on the use of Big Data will require important investments, and these have to be guided by a clear strategy. According to the emerging evidence, organizations that invest in Big Data analysis without a clear appraisal of the type of data needed and without carefully defined strategies, are bound to failure, wasting precious resources and possibly ending up in suboptimal situations.

Private companies have adopted diverse organizational structures to work effectively with Big Data, ranging from highly centralized to remarkably decentralized models and including hybrid solutions. However, developing forms of governance that allow the different units of the organization to share data and work together seems fundamental. Before the advent of Big Data, such an approach would have raised problems of information security, and the various units would have been advised to limit data circulation. Conversely, some members of the WG observed that in a Big Data environment, security should be recommended for the outcome of the analysis, not for the data source. Otherwise, organizations would risk obtaining “security through obscurity”. In private companies that use Big Data analysis, no attention is given to individual data, but rather to the overall amount of data, which is the real added value. In any case, while data ownership should not be centralized, the available evidence indicates that data governance should, and the Big Data Analytics (BDA) unit should be carefully located. It needs to be placed where it is most needed, easily accessible by core units, cross-functional and integrated. Relatedly, some members of the WG stressed that a key issue with Big Data is providing decision-makers with data that are truly relevant for their purposes, and not simply interesting.

A key requisite for all organizational innovations to occur and for Big Data analysis to be effective is the development and incorporation of a Big Data culture. Chief data officers and senior data-related leadership positions will acquire crucial importance in the analysis of information and in the actual decision-making process, but these positions require a special mix of talent and tools that are currently scarce in many large organizations, especially in the public sector. The organizations that are implementing big data analysis seem especially in need of ‘translators’ – professionals that can ensure effective communication between the Big Data analysis unit and other parts of the organization, where workers are not data scientist and may not be ready to work directly on complex models. However, organizations willing to use Big Data are also in need of real data scientists and analysts, because sophisticated techniques and data analysis tools eventually rely on talented humans who know how to manage the tools and interpret data. As a result, attracting new types of talented young workers and retaining them creating new career paths and opportunities will represent both an essential organizational innovation and an important challenge.

In fact, some members of the WG highlighted that it will not even be easy to find many workers with the appropriate knowledge and skills to perform the new tasks in old and complex organizations. It is possible to find computer scientists, but sometimes these individuals do not seem to fit well with large organizations whose main core business has not much to do with computer science. At the moment, it is even more difficult to find translators, since in principle these workers should be social scientists with an expertise in Big Data analysis, but most academic institutions are not ready to forge these profiles. For what concerns NATO and national armed forces, this educational task is not even performed by military academies, even though some experiments are emerging. The ideal profile would include technical awareness, quantitative analytical skills, broad vision, flexibility and open-mindedness – and this explains why it is not easy to produce it.

After having analyzed the main organizational innovations NATO will have to take into account in the age of Big Data, the WG tried to identify the main challenges. Most of these challenges are strictly linked to the organizational innovations mentioned above, representing the other side of the coin, as in the case of finding and retaining talented workers with new professional profiles.

The first challenge is embodied by an emerging tension between centralization and decentralization of the decision-making process of organizations that are introducing Big Data analysis in their work. As stressed by various members of the WG, a key issue in the use of Big Data for decision-making is that the data, once analyzed, need to reach critical decision-making levels, and not only (or simply) the top level. Nonetheless, it is apparently quite difficult to translate this principle into concrete organizational forms and procedures. Paradoxically, while Big Data should promote widespread responsibility and tactical awareness, at the moment advanced digitalization seems to be linked to clear centripetal forces in large organizations. So far, technologies such as remote-controlled arms systems and intelligence activities based on real-time data collection have produced strong incentives to micromanagement and to a re-centralization of decisionmaking in military organizations that have implemented these tools. As a matter of fact, these technologies are managed from the center of the command structure, and provide higher ranks with an increased possibility to control their subordinates as well as the expectation of increasing such control with the adoption of additional technologies. This powerful tendency risks to threaten initiative and reduce the flexibility of military organizations, decreasing their overall capacity to learn and adapt. While the full exploitation of Big Data and the optimization of their use would require the information to reach the most relevant levels, the centripetal tendency leads towards the de-responsibilization of the lower ranks on the ground and to a progressive loss of practice in choosing. This process is not only inefficient for the reasons mentioned above, but also potentially dangerous because Big Data do not fully eliminate uncertainty: in specific occasions, the data can be inaccurate or not available due to technical problems and, in these cases, the personnel on the ground has to be ready to adapt and react basing their decisions on their experience. Thus, various members of the WG recommended NATO to integrate Big Data in the organization’s decisionmaking, favoring diffused ownership and devising different tools for different branches of the organization, based on their specificities. The implementation of Big Data can and should be used to increase flexibility, situational awareness and action in accordance with the center’s intent but without centralizing all decisions and producing the illusion of complete control.

Some members of the WG suggested that it is crucial for NATO – and for all big organizations engaged in the use of Big Data – to create well-designed and reliable evaluation procedures to measure the effectiveness of organizational innovations as well as of the execution of the new decision-making processes. Identifying the initial failures is especially important, to learn from them and avoid structural problems. This issue is linked to other potential challenges highlighted during the discussions of the WG. For instance, some members pointed out that when adopting Big Data, certain member countries will probably do better than others, and this could cause problems of interoperability, but also learning opportunities. Finally, a well-designed evaluation system will also help avoid some deep risks that automated decision-making could represent for the values of NATO. In fact, some members of the WG stressed that what distinguishes democracies from authoritarian regimes or tech dystopias run by robots is that, ultimately, citizens make the rules that have to be implemented. Moving towards automated decision-making as a result of Big Data analysis in matters of war and peace, life and death, is extremely risky, and the implementation of Big Data cannot decrease human control and responsibilities. Some of NATO’s authoritarian adversaries are already facing smaller legal and normative obstacles in the use of Big Data, but NATO will have to carefully check the impact of next generation’s technologies on the democratic values and legal frameworks of its members.

Conclusion The availability of Big Data is promising to revolutionize the business of private companies, the work of public administration and the bases of decision-making in military organizations. The possibility to collect immense dataset on multiple dimensions on reality and update them constantly and in real time could greatly reduce the uncertainty that dominates many environments, including military affairs and actual combat operations.

However, organizations that aim at exploiting this possibility effectively will have to change their governance and their operational procedures substantially. For this reason, WG1 started its works reviewing the available knowledge on the organizational innovations required, seeking to identify useful lessons for NATO. Subsequently, WG1 tried to detect possible organizational challenges for NATO as a result of the Big Data revolution, suggesting ways to tackle them.

First of all, any organization willing to exploit Big Data must have clear goals and strategies to implement their objectives concerning Big Data. This is essential to produce a clear assessment of the areas and levels where the organization could profit most from the introduction of Big Data analysis. As a matter of fact, it is crucial to understand that, in order to be fully effective, the information collected through Big Data needs to reach the points where it is more relevant. Moreover, a key requisite for all organizational innovations to take place and produce results, is the development of a Big Data culture. The new positions will require special talents and expertise that are currently scarce in many military organizations. WG1 highlighted that it will not be easy to attract these talents and retain them in military organizations, if not by creating new career paths and opportunities.

As regards the main organizational challenges, the members of WG1 found that a major challenge is represented by an emerging tension between centralization and decentralization of the decision-making process in military organizations that are introducing Big Data analysis in their work. Somehow paradoxically, while Big Data should promote widespread responsibility and tactical awareness, at the moment advanced digitalization seems to be linked to clear centripetal forces in large organizations. This is not an inevitable path – flexibility and shared ownership of information are compatible with the introduction of Big Data, but big organizations have to be aware of this tension and set up evaluation systems to control the effectiveness of the innovations adopted.

Sign in to recut, update, or save this card.