Assessing one’s own knowledge is difficult. Popular psychology warns us about phenomena such as the Dunning-Kruger effect,1 describing the phenomenon of those lacking knowledge or ability suffering a want of insight into the limit of their capacity. But even if one recognizes the possibility of having insufficient knowledge, the problem of identifying misconceptions and knowledge gaps or missing information remains. It would be most natural to seek advice and opinions from other experts. Methods for self-assessment are still valuable, however, for a variety of reasons, including the following:
Furthermore, it is generally preferable to sort out as many errors as possible before seeking external help.
The Norwegian Defence Research Establishment (FFI) provides analysis in support of both long-term strategic defense development and operational planning in the joint headquarters. We use a scenario- and capability-based approach to describe future requirements for military defense and to assess whether proposed defense plans correspond to relevant threats.2 We continually seek to improve our methods and revisit previous arguments and conclusions.
In this paper, we share our experiences from using a series of wargames as a vehicle for evaluating and developing our knowledge of the alliance aspect of Norwegian military operations – a subject particularly important to our analytical work. While it was clear to us that this dimension had not been given due attention in previous analyses, we found it difficult to know exactly what the shortcomings were. For all the reasons mentioned above, we found it critical to self-assess before involving experts.
Using wargames, we discovered areas where our knowledge of military systems and operations were insufficient, and exposed misconceptions and questionable conclusions that had previously been taken for granted. We are now in the process of expanding our scenario portfolio to cover previously neglected situations. We have also improved the way we present results from our analyses. In particular, the importance both of making basic premises explicit and having a sober estimate of the results we can draw from analytical wargames has become clearer to us. As a by-product, we have also significantly improved our ability to conduct dynamic wargames, which will be of benefit in later analyses.
This is not an attempt to advance the science of wargaming. Our aim for this paper is to inspire fellow practitioners to appreciate the potential of wargaming as a tool for finding out what they don’t know. The series of wargames was conducted, and this paper was written, before the Russian invasion of Ukraine in February 2022.
The activity described in this paper concerns our work providing analysis support to Norwegian military and political defense leadership. The principal source of difficulty in this work is its inherent uncertainty. What are the relevant threats within a reasonable time horizon? How will a conflict play out? Will future military systems perform as expected? By using a portfolio of scenarios as a testing-ground for possible force structures and defense concepts, we can explore this uncertainty. We use the scenarios for identifying capability needs for future defense and to provide context to other operational analyses.
Besides the national military and civil defense capability, the NATO membership and our bilateral relations to allies are cornerstones of Norwegian defense and security strategy.3 Optimal development of the Norwegian Armed Forces depends on insight into the nature of possible future allied operations in Norway and near Norway. However, our analyses have traditionally been based on scenarios that describe direct attacks on Norway, and have for the most part been concerned with the tasks Norwegian armed forces should perform without or before the involvement of major allied support. Revisiting the alliance aspect of defense analysis has therefore been important in the last few years.
We have also observed that our scenario analyses have had a tendency to be static in nature. This is especially true for scenarios that have been used for several years: over time analysts tend to take previous conclusions for granted. The activity presented in this paper is part of our effort to revitalize our ability to run more dynamic wargames, as an alternative to table-top map discussions. This exposes weak arguments and conclusions in our previous analyses and contributes to result validation.
With this series of wargames directed at studying possible allied courses of action we have achieved a dual objective. We have identified, and subsequently filled, numerous gaps in our knowledge about the alliance aspect of Norwegian defense, and we have significantly improved our wargaming technique.
Wargaming of different varieties is an important tool in our analysis. They provide qualitative and indicative answers, while we use other forms of modeling and simulation to obtain quantitative results. Such methods can be combined and used iteratively to form an Integrated Analysis and Experimentation Campaign Plan.4
Wargames are fundamentally acts of communication – structured conversations among experts. According to Pournelle (2017), there are three purposes for conducting wargames: knowledge creation, knowledge transfer, and entertainment. As noted in the British Ministry of Defence’s Wargaming Handbook (DCDC, 2017), while a wargame may profit practitioners in ways beyond those for which it was designed, it is recommended to point the wargame to one particular purpose.5 The main purpose of our wargames was to create knowledge of allied operations in the northern part of Europe and in the Arctic. Pournelle distinguishes between two categories of games suitable for creating knowledge. These are discovery games and analytical games. Discovery games are typically used for addressing unstructured problems, while analytical games are suitable for more structured problems (Pournelle, 2017). In our series, we applied both discovery and analytical wargames.
Although the term “wargame” is widely in use, there is no common accepted definition of the term. We can find different definitions in the literature; Perla (1990), for example, defines a wargame as “a warfare model or simulation that does not involve the operations of actual forces, in which the flow of events affects and is affected by decisions made during the course of those events by players representing the opposing sides.” The U.S. Naval War College (USNWC, 2020) identifies the basic component of wargaming as “people making decisions in a context of competition or conflict (with themselves, other people, or their environment)” while the Wargaming Handbook of the British Ministry of Defence (DCDC, 2017) describes wargaming as a decision-making technique using a “scenario-based warfare model in which the outcome and sequence of events affect, and are affected by, the decisions made by the players.”
Common to these definitions is the understanding that wargames are about conflict situations where different roles compete, making decisions about strategies and courses of action (CoA) to achieve objectives and to respond to decisions made by other players. Wargames are suitable for immersing participants into an environment described by a scenario where they are exposed to various unavoidable challenges (Perla, 1990; DCDC, 2017), and they are dynamic in nature, leading to a course or courses of events that depend on the decisions made by the involved actors.
Wargames can be categorized by how comprehensive and rigid they are. In this paper, we distinguish between three main types of wargames: seminar games, matrix games, and Kriegsspiel. This is in line with the wargame styles discussed by Pournelle (2017).
|Seminar games||Seminar games are moderated discussions between people in a common context/scenario. Typically:
• They have a free form.
• They are exploratory.
• They have open information.
Seminar games are particularly useful for supporting creativity and exploration. Adjudication is usually less rigid, but sufficient to drive the course of events forward. The adjudication can be performed by an umpire or by consensus among the players.
|Matrix games||Matrix games have a stronger roleplaying aspect than seminar games and generally use more formal adjudication rules. In a typical matrix game, the different roles will develop and present their preferred CoA and provide supporting arguments for it. The other roles will try to come up with counter arguments. Adjudication is performed by umpires that assess the pros and cons of the arguments, and decide on a probability for the CoA to be successful. The outcome is decided by rolling dice. Matrix games are suitable, for example, for exploring different courses of events within one scenario context.|
|Kriegsspiel||Kriegsspiel are more formal wargames with an increased level of detail and rigor compared to seminar and matrix games. They typically have a more realistic representation of the conflict with more detailed scenarios and representation of the environment.|
In order to decide on the most appropriate game format for an analysis, one should consider factors such as information requirements, the required level of detail, and the time and resources available for preparations and game execution. Different types of wargames are suitable for supporting different phases of the analysis process (Malerud & Fridheim, 2021). Less formal (discovery) games, such as seminar and matrix games, are particularly suitable for supporting the initial phase of an analysis – problem structuring and the exploration of different decision options, for example. More formal and structured wargames can be useful in later phases, if it is required to go into more details.
Before we initiated our game series, we considered the different types of wargames discussed above, and concluded that we needed to start out with Game 1 being a less formal game in order to improve our understanding of the topics to pursue within the context of allied operations in or near Norway, and to get ideas on how to design the following games. Thus Game 1 can be regarded as playtesting the real games. In our Games 2–4 we had a better understanding of our goals and analysis requirements, and had improved our wargaming technique. It was therefore natural to tighten the game format and conduct these last three games with a more clearly defined matrix-game-like setup. In the description below, we will indicate where the set-up of Game 1 differed from the later games.
In practice, we followed a wargaming process or cycle as described in the British Ministry of Defence’s Wargaming Handbook (DCDC, 2017) to make sure that the games were fit-for-purpose, and that new knowledge and lessons learned were used to refine the design of new games. In this process we also considered various constraints, such as participants, time available, and budget. This process is described in more details below.
As mentioned above, the overarching goal of our wargames was to explore how an allied operation could play out in Norway and the northern part of the Atlantic, and to identify misconceptions and lacks in knowledge. Our opinion was that we had insufficient knowledge of what allied operations in or near Norway could look like, and that in order to analyze Norwegian defense capabilities, we needed to improve our understanding of such operations. But exactly what is was that we did not understand, and even what areas and topics we should pursue, was unclear.
We therefore started out with a very loose Game 1, through which we aimed to identify issues that we could study in subsequent games and analyses. As our series of wargames proceeded, our increased knowledge and our improved technical ability to perform the wargames enabled us to direct the attention more towards details and realism in the actors’ courses of action. Games 2–4 were thus more directly concerned with studying possible courses of action for both Red and Blue side in relevant scenarios.
Before we started to develop the game series, we identified several limitations and requirements.
Each role in the game was played by a small group. The White cell6 consisted of a facilitator and one or two note takers. This group also acted as umpires.
Before the game started, the players were briefed on:
As a starting point of Games 2–4, the groups were tasked with making a simplified operational design7 and defining their preferred end-state given their objectives. This was collected by the White cell and used in analysis. Each game comprised three to five rounds within a timeframe of 7–8 hours.
In Game 1, the groups were simply asked to decide a set of actions and moves, which were orally presented in plenary sessions and discussed among all participants, including the White cell. After each plenary session, the White cell adjudicated conflicts according to their subjective opinion and thoughts about how the course of events should proceed.
In each round of Games 2–4, the players were tasked with writing down their actions or moves and explaining why it would work, in line with matrix game methodology. The groups were allowed to interact during this breakout session using “diplomacy.” The actions and moves were emailed to the White cell and then presented in a plenary session in a predefined order. Arguments for and against the success of the moves were presented and discussed in the plenary session. Moves from the actors that led to combat needed adjudication, and were written down for assessment by the White cell at the end of the plenary session. The White cell umpires defined the possible outcomes and their probability. The outcome was then adjudicated by rolling dice. After each plenary session, a large situational map was updated and presented to the groups by the facilitator.
As the purpose of the wargames was to explore allied operations in or near Norway, we needed roles or actors representing Norway, Norway’s allies, and Russia, as a relevant example of a possible future threat to Norway.
The roles played were Norway, the United States, NATO (excluding Norway and the United States), strategic-level Russia, the Russian Northern Fleet, and, combined, Sweden and Finland. In Game 1, Russia was represented with only one role, while Sweden and Finland were not represented (see below). The Northern Fleet was an independent role in the sense that they were not forced to follow orders from strategic-level Russia. We used two Russian roles in an attempt to balance the opposing sides. See Figure 1.
During the game, the input from the players was collected, the adjudication was documented, and resulting CoAs were discussed. As a part of the wrap-up, lessons learned were collected and knowledge gaps identified (see below). Each game was documented in a separate report.
In our games we used variations of a scenario in which the conflict originated somewhere else in the world and escalated horizontally to the High North. In Game 1, the scenario and the goals of the roles were loosely described. The players started out in a situation where Russia was seeking to protect their second-strike capability – the ballistic nuclear submarines located at Kola – and had attacked Norwegian territory in order to create more depth in their defensive position. The game was difficult to play because of the loosely defined scenario and the lack of information about basic premises. We learned more about our shortfalls in designing wargames than about allied operations. We will discuss the importance of properly defining basic premises in games like these below. The main purpose of Game 1 was to identify issues and topics for further investigation; in this sense, the game was successful.
In Games 2–4 we used a scenario where Russia and NATO were engaged in combat action in the Black Sea region. Russia was seeking to reduce Turkish pressure on Syria and the conflict could possibly escalate horizontally to the Baltic region and the high north. The games had well-defined goals and orders of battle for the involved roles and basic premises regarding use of nuclear weapons, the stances of Sweden and Finland, and other parameters.
In these games, it was a recurring observation that the moves made by the players rarely triggered military combat action. Since our objective was concerned with military courses of action and capability needs in an allied conflict in the high north, this was a problem. Players on both sides of the conflict mobilized and moved forces closer to the potential conflict area in the North, but several players tried to avoid full-scale war in the region.
In order to trigger a conflict, the White cell interfered by introducing submarine incidents in different ways. In two of the games, “rogue” submarines from one party were discovered near the other party’s home bases, which escalated the conflict as desired by the moderators. In Game 4, the scenario was slightly changed from the start with a missing submarine and revised goals in an attempt to trigger a conflict. Somewhat surprisingly, this turned the game into a crisis management game and the supposed enemies cooperated in order to find the submarine. In parallel, both sides mobilized their forces to prepare for a possible escalation.
In the games, Norwegian land territory was only affected to a very small degree. The exception were air and naval bases used for projecting force to other areas, and which were attacked with long-range precision weapons. On the other hand, islands in the area were attractive targets and the land forces of the two main parties took control over Iceland and Svalbard respectively in two different games. Regardless, most of the activity was at sea, both below and above the surface, with maritime and air forces. In addition to this, both parties emphasized attack in the cyber domain and used sabotage on logistics and infrastructure in the opponent’s rear areas. While we got some ideas about possible courses of action, we still have a long way to go for quantifying capability needs. We have, however, identified, and later filled, several knowledge gaps, and are much closer to involving the right subject matter experts and closing in on our ultimate goal of improving our analyses of alliance-related capability needs.
It is also important to be aware that we played only a few samples from a very large variety of courses of action, and that we can obviously not use our observations to predict how a real scenario would unfold (see below). Note that this kind of prediction is not the purpose of our scenario analyses; our ultimate goal is to determine capability requirements that are robust to variation of scenarios.
It was important to communicate the purpose and the goals of the game clearly, and we struggled with getting the players in line with our intent. This could have been because several participants did not have experience with scenario analysis of military operations, but had previous experience from crisis management games. Several players attempted to avoid escalation and the use of military force, although this was an explicitly stated intention from the White cell.
Regarding this problem, it did not help that we played with a mix of the operational and strategic level. An advantage of this mix was that the players who were mainly thinking on the strategic level had military operational tools to use. This gave the players a useful insight into both levels. The problem was that the playing participants were tasked to achieve objectives of a political nature while being expected to employ a considerable, detailed, order of battle. The contradiction between objectives and means probably made it more difficult for the White cell to achieve their intentions. Many players were not comfortable with the level of detail provided by the order of battle. To some extent, the detailed order of battle also drew the attention to a detailed tactical level, and players could have thought more about the overall CoA. Since the overall intention was to learn about military courses of action, the players’ objectives could have been given in military operational terms rather than on the political level.
Over the course of the wargame series, we learned a lot about how to introduce the scenario in order to make the players properly involved in the setting. However, even in Game 4, the last in the series, events took unexpected turns, as the White cell’s introduction of what was intended to be a conflict trigger turned the game into cooperative search-and-rescue game. We have no formula for how the scenario might be briefed so that the players are aligned with the White cell’s intentions other than to be aware of these difficulties.
In our support to the Norwegian long term defense-planning process, we use a portfolio of scenarios developed through morphological analysis.8 Through the kind of wargames we have described above, we can improve our methods – from creating a better understanding of the parameters and combinations defining the scenario portfolio to identifying courses of action and capability needs in the scenarios. We discovered both possible new combinations of parameters and missing scenarios within already defined combinations of parameters.
For example, in the type of conflict between great powers that we studied in our series of wargames, Norway and the Norwegian defense will probably have little impact on the big picture. Nevertheless, such scenarios can have important implications for the development of Norwegian forces. Our experience from the games is that we need to develop more scenarios in which Norway gets involved in this kind of conflict, not triggered by any attack on Norwegian territory or Norwegian forces, between our alliance and an opponent. We have now developed techniques to include scenarios of this kind into our scenario portfolio.
Another important issue we discovered was that the timelines in the scenarios are difficult to determine. How long will it take before the alliance arrives? Where will they arrive? And with what forces? Different answers could lead to very different conclusions with regard to capability needs for the Norwegian armed forces. This can be explored with sensitivity analysis varying some of the parameters regarding time, geography, and forces over the scenarios, new or existing.
In Game 1 we wanted to identify aspects of allied operations that we had to address in our methodology development and analysis work. The underlying questions were as simple as “What is it that we don’t know about allied operations?” This led us to conduct the game with a very loose “see-what-happens” approach which led to discussions about the ways in which parameters will affect or influence an allied operation in the north. There were many hidden conditions in the scenario that we expected to be clear or unimportant, but that turned out to be ambiguous and critically important. The players were not able to determine the best courses of action, since there were undefined strategic factors. The major outcome was therefore a much better understanding of the importance of basic premises.
One example of a missing basic premise in Game 1 was the role and attitude of Sweden and Finland. Access to Swedish and Finnish territory and airspace would increase allied options significantly, but this would depend on Swedish and Finnish cooperation. There were no players representing these nations, and we had not defined their positions in the scenario. This oversight may have been due to an implicit assumption by the White cell that these nations would side with Norway and its allies. However, in a real-world future situation, their stance would depend on their fear of serious involvement in a conflict that they could avoid, and how they would value a good relationship with Norway and Norway’s allies.
In the aftermath of Game 1, we therefore established a list of strategic factors that we needed to make explicit in the following games. Among these factors were defined goals for each participant, technological factors, public opinion, and events outside the local game theater (in particular whether the players needed to balance their military effort with conflicts in other theaters).
The problem of non-explicit basic premises is much more general than the context of wargaming. When decision-makers evaluate the available information about their options, they rely on the clarity and transparency of the presentation. It is the presenter’s duty to know and convey what assumptions the conclusions and recommendations rest on. This demonstrates their validity. It is the decision-maker’s duty to judge whether the options are sufficiently robust to change in underlying assumptions. This is a risk judgement.
A full record of all underlying assumptions is not feasible. There will be assumptions that are so certain that it does not make sense to chart them. There will also be less-certain assumptions, where failure will not be consequential enough to warrant the effort needed. We must identify assumptions that are consequential and significantly probable to fail.
We can never know whether there remain important assumptions that we have not yet made explicit. But mindful attention towards this issue should improve the depth and applicability of the analysis.
We have thus seen that our experience from Game 1 taught us to be more careful to avoid implicit premises in our analyses. We believe that this is an important lesson for any participant in the development of the armed forces, and that officers graduating from defense education programs should habitually meet any analysis with concern for the underlying assumptions. Furthermore, we believe that the wargaming format, where the players are forced to choose their actions based on the available information, exposes and highlights the importance of this concern.
As discussed in the introduction, it is difficult to assess one’s own knowledge, both individually and as a group. As much as one might believe that one understands allied military operations, misconceptions and lack of information will surface when one’s knowledge is put to the test. Discovering these knowledge gaps in a safe wargaming environment is preferable to discovering them in a sharp planning situation at the joint headquarters, or when decisions based on faulty recommendations produce bad outcomes.
A significant example from our wargaming series is that the participants, most of whom were civilian analysts, did not really understand operations involving aircraft carriers and carrier groups. Our analysts normally study Norwegian forces, and to them, the capabilities of aircraft carriers are somewhat exotic. However, as our games showed, allied deployment of such capabilities to our areas as part of a local scenario is a possibility that we need to take into account if we want to understand future allied operations that Norwegian forces could become involved in.
Another example of a knowledge gap that became apparent during the wargames was the nature of operations to recapture land from an enemy when this would involve an amphibious assault. In addition, multiple unknowns of a technical nature affected our wargames. Examples include operational ranges for different types of aircraft, and effects of missile attacks on airbases defended with ground-based air defense systems.
Such knowledge gaps come in two varieties: On the one hand, there are those that lead us to seek more information from external sources. They are related to the question “How does X work?” X in this case could be a military system (an aircraft carrier group, for example) or it could be a general operational concept such as amphibious operations. On the other hand, there are knowledge gaps that lead us to further analysis in the form of things such as modeling or simulation of a more detailed vignette-type sub-scenario.
In our series of wargames, we used observations from each game to establish lists of topics for further investigation until the next game. We solved some of these issues by consulting literature and creating simple fact sheets. For some of the conceptual topics, we held briefs prior to the next game, either by a designated fact checker in our group, or by an invited expert. Furthermore, we have identified tactical vignettes that we plan to study in more detail in order to improve our capability-based analyses.
Understanding the limits of one’s own insight is important. For military officers, a sharp planning situation will probably contain elements that differ from previous operations and exercises. We have seen that our wargames became a vehicle for identifying gaps in our knowledge that could otherwise have gone unnoticed. This led to knowledge development giving us a firmer footing in our work. One can, of course, never fill every knowledge gap – but wargaming as a method for evaluating one’s own knowledge should be of interest to officers.
Scenario-based analysis is by definition associated with uncertainty. There can be enormous variations in how a situation can unfold from some defined initial conditions. Wargames like the ones we discuss here will only probe into one of these continuations at a time. It follows that the outcome of a wargame only represents a sample result. More rigorous results, if at all obtainable, call for stochastic simulations and careful modeling.
The validity of information based on wargames may be overestimated unless it is clearly understood to what degree the courses of action played represent what is possible. There is a significant danger of over-generalization from a wargame; the observed outcome can depend heavily on arbitrary choices by the players, on random adjudication, or on the personality and preferences of the players. We did observe in our games that certain players were much more willing to conduct offensive pre-emptive operations, while others were much more careful not to escalate. Since teams were shuffled between games, this had significant impact on the course of events.
As wargames can reveal risks that one should avoid in a real situation, and other vulnerabilities, they may therefore be better for finding out how things should not happen than how they would happen.
Analysts delivering decision support must be aware of these issues – but it is also necessary that decision-makers recognize them. With first-hand experience from such wargames, one is probably less likely to believe that wargames can provide definitive answers as to how a real situation will unfold.
Analyses supporting decision-making in defense matters tend to be very complex. Such work necessarily involves many people, none of whom will have intimate knowledge of the full subject matter. First, participants must acquire a deep understanding of their own area of responsibility. Then they need to understand that of their colleagues well enough to trust the results and understand its implication for their own work.
One virtue of wargaming in defense analysis is that is an engaging way to get people involved. So it is important to note that while subject matter expertise, in particular that held by military officers, is crucial in FFI’s work, it may be difficult to elicit such knowledge efficiently. Subject matter experts (SMEs) and analysts must share a common frame of reference in order to understand each other. Wargaming provides an arena for focused discussion where the appropriate context and rules are forced.
Our wargames have provided an opportunity to explore scenarios together as a group comprising operational analysts, military officers, security policy experts, and technology experts. The wargaming format forced a common frame of reference for discussions, and helped us understand the whole analytic puzzle, and not just our own pieces.
Scenario-based analyses are frequently subject to skepticism among those who did not take part in the analytic process. Wargaming provides an arena for involving stakeholders and result users, helping them trust and understand the analyses resulting from the games. Having been an active participant, they will better understand the impact and limitations of the results.
Our series of wargames has taught us much about ways in which we can improve our analysis support for long-term defense planning and operational planning in the joint headquarters. We have used the wargames to discover areas where our knowledge of military systems and operations is insufficient and to expose misconceptions and conclusions previously been taken for granted. We have also been able to improve our scenario portfolio’s coverage of the military threat environment. Furthermore, we now better understand how to describe some of the factors that determine the validity of results from scenario-based analyses.
While we have yet to revise future capability requirements, we are now in a much better position to invite external experts and stakeholders to participate in wargames contributing to an analysis campaign leading to revised requirements. As such, this whole internal series can be regarded as a way to playtest more consequential games.
Wargaming is not just an excellent way to engage knowledgeable people and to explore possibilities in scenarios. It is a good tool for understanding risks and weaknesses in plans. But it should be noted that matrix games of the format that we have used are not necessarily suited for prediction or for obtaining firm information about how a scenario will unfold: there are simply too many degrees of freedom. It is important for both analysts and decision-makers relying on analysis support that these limitations be understood.
In particular, we believe that officers serving in positions where results from scenario-based analyses are used need to be trained to recognize whether underlying assumptions and the validity of results are adequately described. First-hand experience with wargames can give a better understanding of this importance.
Wargames provide an opportunity to learn through experimentation, exploring possibilities, discovering gaps and limitations in knowledge, challenging current opinions, and understanding uncertainty.
1See, for example, Dunning (2011). We do not claim any expertise in psychology and acknowledge that we have only read the abstract of this paper for the purpose of making an introductory comment.
2The methodology is described in Vatne et al. (2020).
4See, for example, Labbé (2006).
6The use of the term “White cell” is not consistent internationally. We use the term to identify game control, in line with practice in the United States. This differs from, say, practice in the United Kingdom, where White denotes “national and supranational political organisations and diplomats, humanitarians, international organisations and non-governmental organisations,” as described in the UK MOD Wargaming handbook; see DCDC (2017).
8Morphological analysis is a technique for structuring a large set of complex possibilities by identifying relevant parameters and possible values for each parameter. Combinations of values are assessed for pairwise consistency, leading to a parametric classification of possibilities. See Johansen (2018).
We would like to thank the reviewers for their helpful comments on earlier drafts of this paper.
The work presented here has been funded through FFI projects 1552 and 1596.
The authors have no competing interests to declare.
Dunning, D. (2011). The Dunning-Kruger effect: On being ignorant of one’s own ignorance. Advances in Experimental Social Psychology, 44, 247–296. DOI: https://doi.org/10.1016/B978-0-12-385522-0.00005-6
Forsvarsdepartementet. (2020). Prop. 14 S (2020–2021). Evne til forsvar – vilje til beredskap. Langtidsplan for forsvarssektoren. https://www.regjeringen.no/no/dokumenter/prop.-14-s-20202021/id2770783/?ch=1
Johansen, I. (2018). Scenario modelling with morphological analysis. Technological Forecasting and Social Change, 126, 116–125. DOI: https://doi.org/10.1016/j.techfore.2017.05.016
Malerud, S., & Fridheim, H. (2021). Expert judgements in security analysis – The pros and cons of analytical wargaming. Proceedings of the 31st European Safety and Reliability Conference. DOI: https://doi.org/10.3850/978-981-18-2016-8_547-cd
McHugh, F. (1966). Fundamentals of war gaming. U.S. Naval War College. DOI: https://doi.org/10.21236/AD0686108
U.S. Naval War College. (2020). About wargaming. What is a war game? https://usnwc.edu/Research-and-Wargaming/Wargaming/About-Wargaming
Vatne, D. F., Køber, P. K., Guttelvik, M. S., Arnfinnsson, B., & Rise, Ø. R. (2020). Norwegian long-term defence analysis – a scenario – and capability-based approach. FFI-rapport 20/02367. Forsvarets Forskningsinstitutt. https://publications.ffi.no/nb/item/asset/dspace:6869/20-02367.pdf