Free beta access. 50 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

Data from cyber conflict modeling can be useful in research and development of new cyber defense technologies

Granasen et al 19 (November 2019, Granåsen, Magdalena, Swedish Defense Research Agency, Huskaj, Gazmend, Swedish Defense University, Varga, Stefan, Royal Institute of Technology, “Data Collection and Research in CDXs: Command and Control, Cyber Situational Awareness and Intelligence Perspectives on Cyber Defense,” 24th International Command and Control Research and Technology Symposium)

The results showed that the data collected for the case study of the Swedish blue team during LS19 is meaningful and of sufficient quality for further analysis within the three research themes. Some issues concerning the quality of data were raised in the results section. These and other issues are further discussed in the current section. Concerning data quality, it can be concluded that data loss in the collected data will not impair the ability to conduct the analyses needed in order to explore the research themes C2 , CSA and intelligence. Analysis of different data sources in conjunction will be necessary in order to make sense of the data. The language in chat logs was sparse, and although the researcher is acquainted with the terminology used in these logs, it does not make sense until correlated with other information such as observations or published threat reports. As the exercise was organized by the CCDCOE, the research team only had part control of the data. This is the reason why data on the progress of threat reporting, score and network overview (available services) was collected through screenshots. It will be a time-consuming task to manually transfer these screenshots to a format that can be further analyzed. Still, the research team assessed that a screenshot of the information was better than not having access to this information at all. Using a well-established CDX such as Locked Shields for research is cost-efficient in that the scenario and scenario injects (including a 70-person red team), set-up, gamenet (including support personnel), communication infrastructure, participant invitation, scoring system, exercise control personnel and a lot of the administration already are in place. Furthermore, Locked Shields has gained a reputation that attracts very proficient cyber security experts, while it is likely that these people would not feel equally attracted to participate in a pure research experiment. From a Swedish perspective, one of the greatest benefits of participation in LS19 was the ability to meet and work with people from other organizations. Research within an international CDX clearly entails lack of experimental control and reduced access to data for the researchers, as well as continuous adjustments to fit the data collection to the exercise schedule. However, it also entails highly motivated, skilled and competitive participants, and a far more complex exercise environment than a normal research project would be able to finance. The initial results from the surveys indicated that the exercise as a whole seemed to have progressed smoothly, with positive ratings of teamwork, information sharing and strategy, as well as cyber situational awareness. The high performance score further emphasizes the success of the team. It is therefore clear that the case studied is well suited for analysis of how a successful team develops its strategy and what functional structures for coordination in a cyber defence team may look like. The participation of the research team can also be viewed as a success story. The team was engaged from the very beginning of the preparatory phase of the exercise. In this way, the researchers were able to learn contextual factors, such as the team leadership view on issues such as strategy and coordination, which may have affected the final outcome of the exercise, that were not necessarily written down or documented in any other way. The researchers were given the opportunity to describe their initial planning, which included the aim to be as non-intrusive as possible. This means that the data collection did not include any "freezes" in the exercise in order to collect data, neither were there any plans to collect excessive unnecessary amounts of data (video recordings, key-logging or screenshots). The research team explained their measures to preserve the participants’ anonymity (if this was requested). The nature of the meetings, e.g., personal interactions between exercise participants and the research team contributed to mutual trust and confidence building. The cordial relationship between the researchers and the participants probably contributed to the high survey response rates and a benevolent attitude to answering the questions. Having three persons in the research team meant a high workload, and an appreciation for that every team activity would not be observed. It should be noted that there could be a limit as to how many researchers are feasible for data collection to reduce the 16 risk of collecting too much data, duplicates of data, or disturbing the observed team. The inclusion of CSA-related questions in the team’s ordinary reporting procedure provided a less intrusive method for collecting SA-related data than if a traditional probing measure such as SAGAT or SART had been used. There was no indication of that the inference of SA-related questions in the internal reporting template disturbed the sub-teams. The internal reporting template was designed and administered by the reporting team, and the sub-teams were not aware of that some of the questions in the internal reporting template were induced by the research team. The research team may have positively affected the SA and even the performance in the team, in that they directed the team’s attention towards issues useful to include in the threat reports as well as for the understanding of the threats. This is, however, a risk with all data collection. For instance, the presence of observers may affect the behavior of the team, and the questions in the start-up survey may direct the team’s focus towards the issues mentioned in the survey. For integrity reasons as well as risk of data overload, the research team made a deliberate decision to not to capture network activity. It is assessed that network activity logs would produce a lot of data of which only a small part would be useful. Subsequently, a great deal of analysis would be required, and there would probably not be a great deal of added value for the research themes. For the current paper, a pluralist methodology [31] was employed, where different types of data were collected in order to address the research themes from several perspectives. It needs to be acknowledged that some data will almost always suffer from some data loss (for instance surveys), while other data is more complete, but requiring quite a lot of analytic effort and needing to be combined with other data in order to interpret it (i.e. communication logs). Some data is based on subjective assessment (surveys, observations). However who is best suited to assess the workload, if not those who are experiencing it? The pluralist approach finds answers in combinations and comparisons of data. The dataset will suit the analysis within all three research themes, and the same data items may be combined and analyzed in different ways to correspond to research questions within different themes. 6. Summary and Conclusions The objective of this paper was to explore the possibilities to collect meaningful data for research on Command and Control (C2 ) Cyber Situational Awareness (CSA), and Intelligence in conjunction with an interorganizational cyber defense team during a cyber defense exercise. The purpose of the data collection was to explore the possibilities to conduct research within these research areas. A variety of carefully selected data collection methods specifically tailored to suit the needs for relevant data for our research fields of interest was employed. Our research has shown that a CDX indeed provided a good opportunity to collect pertinent data to analyze Command and Control, Cyber Situational Awareness and Intelligence related research questions. The trove of collected data will be used to write papers according to our research themes. The working titles of some future papers in the works are: i) Exploring the development of coordination structures and strategy in a cyber defense team, ii) Acquiring Cyber Situational Awareness in a cyber defense team, and iii) Intelligence in support of a defensive cyberspace operation. The results obtained in this paper is further expected to be useful for refining data collection methods for CDXs.

Sign in to recut, update, or save this card.