Behind the Scenes of Scenario-Based Training: Understanding Scenario Design and Requirements in High-Risk and Uncertain Environments

Noori et al. Behind the Scenes of Scenario-Based Training
WiPe Paper – Prevention and Preparation
Proceedings of the 14th ISCRAM Conference – Albi, France, May 2017
Tina Comes, Frédérick Bénaben, Chihab Hanachi, Matthieu Lauras, Aurélie Montarnal, eds.
Behind the Scenes of Scenario-Based
Training: Understanding Scenario
Design and Requirements in High-Risk
and Uncertain Environments
Nadia Saad Noori
University of Agder
nadiasn@uia.no
Yan Wang
Delft University of Technology
Y.Wang-16@tudelft.nl
Tina Comes
University of Agder
tina.comes@uia.no
Philipp Schwarz
P.Schwarz@tudelft.nl
Heide Lukosch
H.K.Lukosch@tudelft.nl
ABSTRACT
Simulation exercises as a training tool for enhancing preparedness for emergency response are widely adopted in
disaster management. This paper addresses current scenario design processes, proposes an alternative approach
for simulation exercises and introduces a conceptual design of an adaptive scenario generator. Our work is based
on a systematic literature review and observations made during TRIPLEX-2016 exercise in Farsund, Norway.
The planning process and scenario selection of simulation exercises impact directly the effectiveness of intra- and
interorganizational cooperation. However, collective learning goals are rarely addressed and most simulations are
focused on institution-specific learning goals. Current scenario design processes are often inflexible and begin
from scratch for each exercise. In our approach, we address both individual and collective learning goals and the
demand to develop scenarios on different layers of organizational learning. Further, we propose a scenario
generator that partly automates the scenario selection and adaptively responds to the exercise evolvement.
Keywords
Humanitarian simulation exercise, scenario design process, collective learning, interorganizational coordination.
INTRODUCTION
Requirements of scenario development are domain specific; whether it is disaster management (Meesters and Van
de Walle, 2014), water management (Dong, 2013), or military training (Hartog, 2009). In addition, the planning
of scenario-based training requires intensive domain knowledge and specific learning aims. It often lacks the
agility to be re-used in creating new scenarios, or to be scalable in different environment (Luo, Yin, Cai, Lees, &
Othman, 2014, Pharmer & Milham, 2016).
In the field of disaster management and humanitarian aid, response operations involve different levels of
cooperation and coordination among numerous international and national organizations and authorities. Success
and failure of missions is determined by many factors: the organizations’ specialties and capacities, resources
availability, and field conditions create an extremely dynamic and challenging environment for field coordination
(Comfort & Kapucu, 2006). Coordination networks among humanitarian organizations (Noori & Weber, 2016)
and coordination clusters among organizations engaged in emergency response operations (Noori, Wolbers,
Boersma, Cardona, 2016) are in place to deal with the complexity of operational dynamics in disaster response.
The network-based view provides a holistic and task oriented scope of the operational environment and identifies
core elements of disaster and emergency operations in unstable environments.
948

Scenario-based methods present a valuable tool to improve the performance of decision-making in the field of
disaster and emergency management. There is a wide variety of scenario approaches related to planning and
policy-making (Amer, Daim & Jetter, 2013) that is mostly related to forecasting the future by constructing
different possibilities within certain contexts (Kosow and Gaßner, 2008). Developing paths that enable the
achievement of organizational or societal aims largely drives those goal-oriented approaches. However, the high
levels of risk and uncertainty in response operations and the dynamic and volatile environment make the task of
planning and selecting scenarios more difficult (Comes, Hiete, Wijngaards & Schultmann, 2011). Scenarios
selection for exercises and simulations in the field of disaster response need to be purposeful and relevant in
addressing specific learning aims (Comes, Wijngaards, Maule, Allen & Schultmann, 2012). Purposeful scenarios
respect time and resource constraints, correspond to unfolding dynamics of events in real-time, and contain
context and elements affecting the process of decision-making (Comes, Wijngaards & Van de Walle, 2015).
In many countries, civil defense and organizations involved in disaster and emergency response follow the formal
and highly structured Incident Command Systems (ICS) in their training exercises and actual response operations
(Boersma, Comfort, Groenendaal & Wolbers, 2014). ICS consists of a number of mandates and protocols
describing communication and coordination between response organizations (Bigley & Roberts, 2001). However,
studies based on network analysis found that members of response networks self-organize following tasks needed
during response operations and do not necessarily follow the rigid ICS structure (Topper, C. M., & Carley, K. M.
1999; Noori, Wolbers, Boersma & Cardona, 2016). The emerging task-based structures proved flexibility and
resilience in responding to uncertainty and dynamic changes during both natural and humanitarian disasters (Noori
& Weber, 2016). Following ICS in their exercises and lack of scenario adaptability to dynamic condition in real-
time; made it difficult for humanitarian and civil defense organizations to exploit the benefits of coordination-
network behaviors to increase resilience and agility.
This paper focuses on scenario-based exercises that target humanitarian aid operations as parts of a larger effort
aiming at the improvement of the security and the efficiency of humanitarian aid work. The work is based on
observations made during TRIPLEX-2016 exercise, one of the largest exercises for humanitarian organizations.
It is organized every three years by the International Humanitarian Partnership (IHP) with a focus on exercising
cooperation and coordination among various organizations. It involves joint response planning by humanitarian
organizations, civil protection authorities, and other entities.
This paper is structured as follows. The background section examines the importance of and requirements for
scenario-based training, scenario design and requirements in disaster and emergency management context. The
section of TRIPLEX-2016 case study provides details of the exercise itself, its scenario design process and
implementation. In the following section, a framework for generating scenarios in unstable contexts is introduced.
It is developed based on the literature and the observations made during TRIPLEX-2016 Finally, in the discussion
and conclusion we reflect on the insights gained and outline room for future research.
RESEARCH METHOD AND DATA COLLECTION
The research strategy followed in this paper was twofold. First, a systematic literature review was conducted
(Creswell, 2013) using combinations of the following key words: simulation exercise, scenario development,
scenario design, and injects, collaborative response, interorganizational coordination, and humanitarian aid. The
literature review objective was to compile literature about scenario development and simulation exercises mainly
in disaster management and humanitarian aid and other related fields like healthcare and military. Second, a team
of researchers joined the TRIPLEX-2016 exercise in Norway. During the preparation and execution phases, we
conducted participant observations as a social science research method based on gathering data from personal
experience made by observation and participating in activities and discussions (Musante & DeWalt, 2010).
949

BACKGROUND
Scenario-Based Training in Disaster and Emergency Management
Scenario-based training and simulation exercises are powerful tools that enable organizations to (1) train
executing relevant procedures and processes; and (2) practice decision-making in complex and dynamic
environments (Lateef, 2010; Comes, Wijngaards & Van de Walle, 2015). Exercises provide insights into the
organization’s, team and individual performance. On the organizational level, exercises enable assessing the
implementation of concepts such as inter-organizational coordination, reveal shortcoming and gaps in planning,
and improve organizational coordination. On team level, trainings can support the elimination of ambiguity by
clarifying roles and responsibilities. Finally, for individuals, it can help to strengthen the confidence in operational
roles and helps to gain recognition and support of official (Pharmer & Milham, 2016).
Simulation exercises can emulate ambiguous and equivocal situations that humanitarians face in their operations
in the areas of armed conflicts and the aftermath of natural disasters. Besides, simulation-based learning can be
applied in designing a learning experience aimed at practicing other objectives that are hard to address otherwise
(Hartog, 2009; Nikolai, 2014). A simulated environment can represent the central characteristics of a dynamic
and complex system in order to understand and experiment within this system (Duke, 1980). Participants take
over certain roles within this system (Duke & Geurts, 2004). The scenario in this sense is the setting of the
simulated environment to constrain behaviors and to simulate the complex system (Greenblat & Duke, 1975). The
scenario or the simulated environment plays an important role as supporting means for (1) the experience, (2) the
effectiveness and difficulty of the simulation, and (3) the feeling of flow or immersion of participants’ experiences
when doing the exercise (Lukosch, van Ruijven & Verbraeck, 2012).
For example, it is widely recognized that no single organization can undertake all aspects of disaster response
alone but instead responses require to pool and allocate resources from many organizations, including affected
government entities, UN agencies, international and domestic NGOs, and affected civilian populations (Kapucu,
2009; Kapucu, Arslan, Collins, 2010; Butts, Acton, Marcum, 2012; Noori & Weber, 2016). Figure 1 shows is an
example of a response network of a humanitarian aid mission for refugees in a conflict zone. Despite the
complexity in cross-organizational coordination, organizations are self-organized in the form of task-based
clusters (marked with dotted circles in Figure 1): national relief sector, international relief sector, and logistics.
Therefore, scenarios simulating such complex environments require joint exercises as an effective way to build
trust among actors, and deepen the knowledge and confidence in coordination concepts such as the cluster system.
Further joint exercise approach supports and enhance mutual understanding of roles and mandates (Banuls,
Turoff, Hiltz, 2013).
Figure 1. Coordination complexity for a humanitarian aid mission in conflict zones.
950

Scenario Development for simulation exercises
In essence, scenarios are stories on how the future could potentially evolve. Scenario development techniques aim
to compile a comprehensive set of plausible clearly defined situations that can be tested (Wright & Goodwin
2009; Schoemaker, 1993). The multitude of scenarios relates to the uncertainty about the behavior of the system
and the complexity and diversity of crisis response operations. Nevertheless, covering all possible scenarios is
infeasible, as this would require an infinite number of scenarios or situations to be played (Comes et al. 2011).
Hence, using scenarios entails explicit or implicit scenario management – meaning to balance scenario
construction with limited time and resources.
Scenarios in simulation exercises aim to be challenging and test the response of professionals. The scenario design
process fundamentally determines the effectiveness of the training. However, planning simulation exercises are
known to be a complex and resource intensive process. Different approaches have been proposed to conduct the
scenario design. Dong (2013) emphasizes that the scenario design procedure should be iterative and encompasses
at least the following three steps. First, the main focal questions, driving forces and sources of uncertainty should
be clearly written down. Second, the storyline should be formulated and the scenario logic be constructed. Finally,
the scenarios should be linked to the goals and issues of interest.
Although appearing in a different wording, the use case modeling (Bittner and Spence, 2003) can also be
considered as a scenario design process. For every use case, the roles of involved actors, their interactions and
associations with each other and the environment are specified. Uses -cases are normally defined in a workshop
setting and jointly it is decided about the relevance of the use case and the expected behavior in certain situations.
The US Department of Homeland Security Exercise and Evaluation Program (HSEEP) describes an inventory of
all potential injects named Master Scenario Event List (MSEL) as key element in the successful planning of an
exercise. The MSEL is the exercise blueprint, consisting of a structured listing of the injects, each of which contain
a brief description of events, expected actions, intended players and estimated time of inject (Teclemariam &
Yang, 2014). It is designed such that each inject is linked back to one or more exercise objectives and provide
options for adaptive exercise play. Importantly, injects should be grounded in reality, so that players feel
motivated to execute the policy or procedure being tested. Renger (2009) propose the following steps for writing
the MESEL:
1. Delineate size and location of exercise
2. Formulate objectives of exercise
3. Define exercise events and add an initiation inject for each event
4. Write reminder inject (trigger response when inject is not taken up)
5. Brainstorm redirect injects (anticipate other plausible actions that may still resolve situation)
6. Include non-objective injects (optional and only used to fill time gaps)
7. Estimate times for each inject
Scenario-Based Training in Dynamic, Complex, Uncertain and High-Risk Environment
Simulation exercises and the described methods for scenario design have been adopted in a wide range of
application fields from health care, civil protection, humanitarian, and the military. Despite the differences, all
those fields share common work settings, which are described as dynamic, complex, uncertain and high-risk
environments. In this paper we specifically study the process of planning and designing scenario-based exercises
for disaster and emergency response operations.
Acknowledging the complexity and dynamics of the settings, it is important that the scenarios can adapt in real-
time to the unfolding exercise. Enabled by modern monitoring and tracking technology, we understand a scenario
no longer as a static playbook, but as a dynamic network of events (injects) and response by the players to ensure
that training objectives will be achieved. In this sense, we propose to use approaches of scenario selection and
relevance assessment (Comes et al., 2012) as a way to develop scenario paths in real-time.
Learning objectives are manifold; therefore we only focus on two common key issues:
Firstly, both scholars and professionals have referred to improving the level of collaboration among the
participating teams as central. In complex operations in high-risk environments, inter- and intra- organizational
coordination is a factor that can determine success or failure. Inter-organizational coordinating requires an
orchestration of efforts during planning and execution phases of disaster management plan (Comfort & Kapucu,
2006; Kapucu et al., 2010). In disaster management, the standard for this is the “Incident Command System”
951

(ICS) (Boersma, Comfort, Groenendaal & Wolbers, 2014). ICS consists of a number of mandates and protocols
describing communication and coordination between response organizations (Bigley & Roberts, 2001). Also in
the training for crisis response ICS is used standard operating procedure for managing and coordinating activities.
Both civil and military actors use it as formalized institutions, to facilitate collaboration among different actors in
a response group. Despite the careful planning and the efforts made to create a holistic framework for facilitating
coordination between the different stakeholders during response operations, reality shows a different picture. The
coordination dynamics in response operations are not as rigid as defined by the ICS, instead they emerge as loosely
coupled network formations named coordination-clusters. The coordination-clusters are a representation of
executed tasks during response operations (Butts et al., 2010). For an illustrative case for this observation we refer
for example to Noori et al, 2016 which studied with a network approach how the coordination cluster formatted
in the disaster response to the Elbe river flood 2002 in Germany.
Secondly, another widely recognized issue is that many scenario based trainings provide an specific template for
expected behavior and do not sufficiently consider designing scenarios which train to deal in an unusual situation
where following specified procedures is not enough. Commonly, the scenario development is suspended before
the beginning of the exercise. Consequently, there is little room for adaptive behavior and out-of-the box thinking
(Reason, 1997). However, in today’s world, characterized by unprecedented complexity and uncertainty, this is
indispensable. Risks are hidden and opportunities and constraints may rapidly evolve, which brings high pressure
to the current crisis management and response system. Hence, there is the need for alternative training methods
that support organizations and individuals to interpret ambiguous situations and find innovative solutions to
unexpected situations To our best knowledge, there is currently no approach to dynamically generate scenarios
that focus specifically on this learning goals (Comes, Bertsch & French, 2013). Therefore, a dynamic approach
to scenario design is needed that:
1. Focuses on the learning aims within and across organizations
2. Aims at revealing vulnerabilities inherent in the technical systems, processes and policies;
3. Recognizes the complexity and interconnectedness of today’s socio-technical systems;
4. Is a continuous process that iteratively recognizes and adjusts to new information;
CASE STUDY: THE TRIPLEX-2016 EXERCISE
To complement the literature review, next through a case study we demonstrate properties and challenges of the
scenario design processes. The selected case is not meant to be generally representative for the scenario-based
training in the field of disaster response and emergency management but be an illustrative example.
Triplex is a 4-day humanitarian and civil protection exercise, triennial organized by the international humanitarian
partnership (IHP), a network of seven European governmental emergency management agencies. As a team of
researcher we joined the most recent simulation exercise taking place in Lista, Norway between 25 - 29 September
2016. In Triplex-2016 over 36 organizations working in the field of humanitarian aid and civil protection jointly
trained the response to a simulated large-scale disaster. The main objectives of the TRIPLEX-2016 exercise were
to: (1) simulate a training environment for humanitarian actors to exercise coordination, information management,
assessment and other mission- related issues, and (2) train and exercise cooperation capacities and coordination
mechanisms (e.g. UNDAC/cluster coordination) between humanitarian and civil protection actors. Participating
agencies in the exercise included humanitarian United Nations (UN) agencies, iNGOs, and governmental civil
protection agencies. According to the organizational team the objective of the exercise was to provide a safe
learning and training environment, for exercising cooperation in humanitarian operations between civil protection,
humanitarian organizations and other actors.
TRIPLEX-2016 exercise covered the following phases of response operations: (1) Arrival of the United Nations
Disaster Assessment and Coordination (UNDAC) team and establishment of a UN Reception/Departure Centre,
(2) arrival of international response teams at the airport and customs and immigration procedures, (3)
establishment of the UN On-Site Operations Coordination Centre (OSOCC) registration and assignment of
international response teams, (4) establishing inter-cluster coordination mechanisms in the key areas Food, Water
and Sanitation (WASH), Logistics, and Health, (5) perform (joint) rescue operations and Multi Cluster Initial
Rapid Assessment (MIRA), and (6) stakeholder management including host government and media.
For the process of evaluating the performance of the participants and the quality of the exercise itself, the
organizers of TRIPLEX 2016 established a number of predefined roles in the exercise as set out below:
Exercise Control (EXCON): Responsible for managing and monitoring the whole exercise play, including the
continual update of the inject list.
952

Response cell: Intermediary between participating organizations on the ground and EXCON core group.
Responsible for permanently communicating all developments related to the ongoing and planned injects and
events.
Controller: "Eyes and Ears" of EXCON in the field. Usually assigned to a specific participating organizations.
Concerned with safeguarding that the exercise runs as planned.
Role players: Comprise of amateur actors (prevailing UN university students) that for each inject play a role and
story according to a brief script. Exercise participants (practitioners) interact with amateur actors. For some injects
professionals were playing their own role (e.g. local authorities of host government). A few inject were concerned
only with internal processes and therefore did not require any role players but for most injects the interaction
between role players and participants was essential.
Exercise evaluators: Exercise evaluators were usually a senior member of participating teams. They briefed the
role players and assessed team's performance during each inject. For this the evaluator formulated expectations
on the appropriate response to inject and observed whether they were matched. In addition, exercise evaluators
facilitated the internal evaluation discussion.
Scenario Design and Planning Process at TRIPLEX 2016
The TRIPLEX 2016 planning process involved all participating organizations, and took more than a year,
including three planning conferences and various workshops. Scenario planning was led by a core group and
organized by the Norwegian Civil Protection Authority (DSB). The general outline of the disaster events were
predetermined by the setting and context of the exercise at the sea in southern Norway. The basic scenario or
backdrop assumed that a hurricane and heavy rainfall had hit the country causing major flooding and heavy
damage in infrastructure and displacing hundred thousands of people.
The exercise itself focused on the initial relief phase and the days immediately after the disaster. In this period,
international help started arriving in the affected nation and after setting up camps and equipment, rapid need
assessments were undertaken and rescue operations performed.
After the initial deployment, the exercise was inject-driven. A role-play simulation in which amateur role-players
improvised the interaction with responders following minimal scripts of pre-defined incidents created the
experiential learning environment. Moreover, Norwegian local authorities were actively participating in the
exercise by playing oneself. These injects and events were planned based on questionnaires for specific injects or
situations that the individual organizations wanted to simulate and play. For instance, in some areas
telecommunication, electricity and water supply were unavailable and people were trapped by floodwater and
desperately waiting for rescue.
Scenario Implementation at TRIPLEX 2016
The exercise objectives were set out as to test and train the deployment and relief mechanisms following the
UNDAC and ERCC guidelines and specifically exercise collaboration across organizations. We had many good
and insightful conversations with practitioners during the exercise and after the exercise had completed conducted
five interviews. The interviews were guided by interview guidelines. The official interviews and informal talks
revealed that many practitioners perceived the exercise as meaningful. One interviewee stressed that the exercise
purpose was not to use or test new training materials or innovative tools. The goal was to create an environment
to facilitate collaboration within a larger community and to network with peers where an opportunity arises for
meeting and learning about other humanitarian actors.
Interviewees also highlighted the importance of exercises like TRIPLEX 2016 in the learning and training process
of newcomers. The TRIPLEX 2016 provided a sandbox and a learning opportunity for young professionals where
they can observe and participate in response operations.
Impact of TRIPLEX-2016 Scenario Design Process on Exercise Execution
In addition to the data collected through the interview, the researchers team recorded several observations during
the TRIPLEX 2016 execution. One of the main observations was related to the interorganizational coordination
characteristics during the exercise. Unlike the patterns observed in real-world response operations, during the
exercise, the coordination dynamics between participants appeared to follow different patterns. Participating
organizations were occupied in applying their own methods and utilizing own technology or resources during the
exercise. Although during the planning phase the goal was to improve the collaborative learning regarding
953

communications and a unified situation-room. The intra-organizational coordination dynamics were more visible
than the inter-organizational ones. In the TRIPLEX 2016 instructions, participants were instructed:
“
 During the exercise, participants should act according to their own, regular functions as if it was a real-
life situation, i.e. a real emergency deployment.
 Participants should use the regular chain of command and communication channels that their respective
organizations would use during an emergency deployment.
“
The instructions might contribute to the pattern observed during the exercise. Therefore, the choice of the scenario
design and the language employed had an impact on teams’ behavior.
Repetition in scenarios selection from previous years (i.e. TRIPLEX 2013 or other training events) was another
observation made by the team. Moreover, due to the absence of an automated process execute the scenarios, the
participants had knowledge of the scenarios in advance and knew the injects associated with them. This might
have had a negative impact on the performance because, in reality, disaster events develop following highly
unexpected patterns. So the surprise factor and the instability conditions were lost during the simulation and the
exercise’s capacity to increase preparedness was diminished.
Finally the evaluation process was performed in the teams within each organization in isolation from the others.
It is important to gage the organization’s performance during such events, but more important is to gage the levels
of dependencies on external organizations because it impacts on the performance teams too.
SCENARIO GENERATION FRAMEWORK FOR UNSTABLE ENVIRONMENTS
Generally, the function of scenario development can be differentiated into process-oriented and product-oriented
(van Notten 2006). Process-oriented: facilitates learning, communication and improving observational skills and
entangled the information overload to the essential. Learn from the past and reflect on uncertainties in the future.
Product-oriented: Testing policy options by performing practice runs of possible future situations. Our approach
goes a step beyond that by developing adaptive scenarios that change and adapt dynamically depending on the
actions of the players.
From a design perspective, scenario developed should be comprehensive (scope of the entire exercise
environment), agile (modular components that support adaptability and scalability), valid (quality metrics should
be designed for scenario validation), and effective (task performance measurements).
Having an automated and adaptive scenario generator would provide an alternative to the existing approach in
designing scenario-based exercise simulations. However, one of the challenges in dynamic settings is the
unlimited number of scenarios that can be created. Therefore, there is a need to transform a limitless number of
scenarios into a smaller set of manageable, and select the representative and most relevant injects to create a
scenario tree tailored to the exercise at hand (see Figure 2). Each step of selecting a specific inject is thus
understood as reducing the number of possible scenarios to a limited subset, analogous to the approaches used in
branch & bound algorithms. Prioritization of injects is performed by calculating the potential learning of the
possible scenario paths related to the selection and selecting the inject that maximizes the potential learning at
each step, while ensuring that constraints (time, resources, skills) are met.
954

Figure 2: Scenario development as continuous real-time scenario selection to maximize learning potential
In Figure 3, we present a proposed framework to automate the process of scenario development. We distinguish
between the scenario catalogue, which provides potential contexts and inserts, and the simulated scenarios, which
are adapted in real-time to the actual behavior during the exercise. The heuristic data is stored using a scenario
catalogue that contains elements like contexts, actors, injects and cues.
955

Simulator
TrainingDataandLearningmodule
Feature
Extraction
Random
Scenario
Generator
Scenariogenerationmodule
Desiredtraining
objectives
Extractedtrainee
Features
ScenarioGenerator
Scenario
Designer
Trainer
Trainee
AIPlayer
Scenariocontentdesignmodule
Scenario
Catalogue
Training
models
Event-Based
Neuralnetworks
Figure3.Proposedframeworkofanadaptivescenariosgenerator.
956

The conceptual design in Figure 3 shows the core elements of the framework for a scenario generator that can
adapt the process of generating scenarios based on the changes in the environment. The basic design idea: create
a ‘scenario pool’ (see Figure 2) with modular components from the following elements:
● Actor’s features (local governments, UN agencies, iNGOs, NGOs, local communities, attackers, etc.)
● Training models and aims (improve decision making, avoid errors, etc.)
● Scenario features (contexts, threats, resources (assets, technologies, etc.))
● Cues (continuous and cross-operations related to training aims, trigger cue and response cue)
These elements will be further specified regarding contents, relationships and attributes like possibilities, level of
impacts and training objective or goals.
The framework contains a scenario contents design module that provides both manual and automated capabilities
of contents’ authoring. The module comprise a scenario template consists of the elements mentioned earlier (i.e.
actors, context, assets, cues, and injects etc). An automated training module would provide possible scenario
elements for an adaptive scenario. Input from the scenario catalogue and heuristic data collected from
participants’ performance are used to train an event-based neural network system (Pedersen, Togelius,
Yannakakis, 2010; Sahoo., Xu & Jagannathan, 2015). The participants’ performance during exercises needs to be
evaluated and logged (Thomas, et al., 2012), where it would be used to as in input for adapting and selecting the
scenarios generated. The final module is the simulator, which aggregates contents from the scenario generation
and training modules to produce new scenario path (or events sequence) that reflects changes in the environment
and participants’ response.
DISCUSSION AND CONCLUSION
The scenario-based exercise methods are gaining a great popularity in recent years due to technological
advancements which made simulations and virtual environments creation much easier than before. However, the
scenario-based training in the field of disaster management is still lagging because of the complex nature of
conditions associated with a disaster event (Comes et al., 2011, Luo et al., 2014). The high levels of uncertainty
and risk factors related to disaster events are not easy to simulate in real-time or to cover the limitless number of
existing scenarios (Comes et al., 2015). The lag in creating scenarios satisfying requirements such as
effectiveness, agility, comprehensiveness and adaptability leads to negative effects on the outcome of the exercise
and the organization's’ performance in the field (. As we saw from the coordination patterns observed during the
TRIPLEX-2016 due to the language used and scenario selection and other factors. Instead of observing the
emergence of task-based coordination clusters, organizations exhibited a silo effect like behavior when responded
to injects during the exercise.
We reflect upon the issues observed during the exercise and proposed an automated approach in planning and
generating scenario in real-time to help create simulations that reflect real-live situations. The proposed
framework will help to improve the outcomes and strengthen the use of scenario-based training in disaster and
emergency response. Using scenarios will help forecast possible risks and find proper ways to mitigate or respond
to those conditions. Future work has to be conducted to test and validate our framework. Simulations aiming at
humanitarian aid workers will be developed and observed for this purpose.
ACKNOWLEDGMENTS
The authors gratefully acknowledge their anonymous interview participants and the organizers of TRIPLEX -
2016 for all the efforts put into making this event a success and enabling our research by inviting us to participate.
This research has partly been funded by the European Union’s Horizon 2020 research and innovation program
under grant agreement No 700510.
957

WiPe– Prevention and Preparation
REFERENCES
Amer, M., Daim, T. U., & Jetter, A. (2013). A review of scenario planning. Futures, 46, 23-40.
Banuls, V. A., Turoff, M., & Hiltz, S. R. (2013) Collaborative scenario modeling in emergency management
through cross-impact, Technological Forecasting and Social Change, 80, 9, 1756-1774.
Bishop, P., Hines, A., & Collins, T. (2007) The current state of scenario development: an overview of techniques,
Foresight, 9, 1, 5-25.
Bittner, K., & Spencer, I. (2003) Use case modeling: The Addison-Wesley object technology series, Addison-
Wesley Longman Publishing Co Inc.
Boersma, F.K., Comfort, L.K., Groenendaal, J., & Wolbers, J.(2014) Editorial: Incident Command Systems: A
Dynamic Tension among Goals, Rules and Practice, Journal of Contingencies and Crisis Management, 22, 1,
1-4.
Butts, C.T., Acton, R.M., Marcum, C. (2012) Interorganizational collaboration in the Hurricane Katrina response,
Journal of Social Structure, 13, 1-36.
Creswell, John W. (2013) Research Design. Qualitative, Quantitative, and Mixed Method Approaches (4th ed.),
Thousand Oaks, California: SAGE Publications.
Comes, T., Hiete, M., Wijngaards, N., & Schultmann, F. (2011) Decision maps: A framework for multi-criteria
decision support under severe uncertainty, Decision Support Systems, 52, 1, 108-118.
Comes, T., Wijngaards, N., Maule, J., Allen, D., & Schultmann, F. (2012) Scenario reliability assessment to
support decision makers in situations of severe uncertainty. IEEE International Multi-Disciplinary Conference
on Cognitive Methods in Situation Awareness and Decision Support, 30-37.
Comes, T., Wijngaards, N., & Van de Walle, B. (2015) Exploring the future: Runtime scenario selection for
complex and time-bound decisions, Technological Forecasting and Social Change, 97, 29-46.
Comfort, L. K., & Kapucu, N. (2006) Inter-organizational coordination in extreme events: The World Trade
Center attacks, September 11, 2001. Natural hazards, 39, 2, 309-327
Dong, C., Schoups, G., & van de Giesen, N. (2013) Scenario development for water resource planning and
management: a review, Technological forecasting and Social change, 80, 4, 749-761.
Duke, R. (1980) A paradigm for game design, Simulation and games, 11, 3, 364–77.
Duke, R. and Geurts, J. (2004) Policy games for strategic management. Rozenberg Publishers, Amsterdam.
Greenblat, C. S. and Duke, R. (1975) Gaming-simulation: Rationale, design, and applications, Sage Publications,
New York.
Hartog, C. C. (2009) Scenario design for serious gaming, Master thesis, Delft University of Technology.
Kapucu, N. (2009) Interorganizational Coordination in Complex Environments of Disasters: The Evolution of
Intergovernmental Disaster Response Systems, Journal of Homeland Security and Emergency Management,
6, 1, 47-51.
Kapucu, N., Arslan, T., & Collins, M. L. (2010) Examining Intergovernmental and Interorganizational Response
to Catastrophic Disasters: Toward a Network-Centered Approach, Administration & Society, 42, 2, 222–247.
Kosow, H., Gaßner R. (2008) Methods of Future and Scenario Analysis, Research Project “Development Policy:
Questions for the Future”, Bonn, Germany.
Lukosch, H.K., van Ruijven, T., & Verbraeck, A. (2012) The other City – How to design Authentic Urban
Environments for Serious Games, Proceedings of the International Conference on Information Systems for
Crisis Response and Management (ISCRAM), Vancouver, Canada, 1-5.
Luo, L., Yin, H., Cai, W., Lees, M., & Othman, N. B. (2014) Towards a data-driven approach to scenario
generation for serious games, Computer Animation and Virtual Worlds, 25, 3-4, 393-402.
Luo, L., Yin, H., Cai, W., Zhong, J., & Lees, M. (2016) Design and Evaluation of a Data-driven Scenario
Generation Framework for Game-based Training, IEEE Transactions on Computational Intelligence and AI
in Game, 99.
Meesters, K., & Van de Walle, B. (2014) Serious Gaming for User Centered Innovation and Adoption of Disaster
Response Information Systems. International Journal of Information Systems for Crisis Response and
Management (IJISCRAM), 6, 2, 1-15.
Niehaus, J. M., Li, B., & Riedl, M. O. (2011) Automated Scenario Adaptation in Support of Intelligent Tutoring
958

WiPe– Prevention and Preparation
Systems, FLAIRS, 24, 531-536.
Nikolai, C. M. (2014) SimEOC: A Virtual Emergency Operations Center (vEOC) Simulator for Training and
Research (Doctoral dissertation)
Noori, N. S., Wolbers, J., Boersma, K., & Cardona, X. V. (2016) A dynamic perspective of emerging coordination
clusters in crisis response networks, Proceedings 13th International Conference on Information Systems for
Crisis Response and Management (ISCRAM). Rio de Janeiro, Brazil.
Noori, N. S., & Weber, C. (2016). Dynamics of coordination-clusters in long-term rehabilitation, Journal of
Humanitarian Logistics and Supply Chain Management, 6, 3, 296 - 328.
Pedersen, C., Togelius, J., & Yannakakis, G. N. (2010) Modeling player experience for content creation. IEEE
Transactions on Computational Intelligence and AI in Games, 2, 1, 54-67.
Pharmer, J. A., & Milham, L. M. (2016) Adaptive training for command-and-control teams: challenges in
scenario-based training, Theoretical Issues in Ergonomics Science, 17, 2, 169-182.
Renger, R., Wakelee, J., Bradshaw, J., Hites, L., (2009) Steps in writing an effective master scenario event list,
Journal of emergency management, 7, 6, 51-60.
Sahoo, A., Xu, H., & Jagannathan, S. (2015) Event-based neural network approximation and control of uncertain
nonlinear continuous-time systems, American Control Conference (ACC), 1567-1572.
Thomas, P., Labat, J. M., Muratet, M., & Yessad, A. (2012) How to evaluate competencies in game-based learning
systems automatically? International Conference on Intelligent Tutoring Systems, 168-173.
Teclemariam, N. & Yang, L. (2014) Chapter 6: SUMMIT: An In-Depth Case Study of an Emerging Training and
Exercise Technology. Designing Crisis Management Training and Exercises for Strategic Leaders. Eds Stern,
E. Swedish National Security College. Elanders Sverige AB, Stockholm, 59-70.
Topper, C. M., & Carley, K. M. (1999). A structural perspective on the emergence of network organizations.
Journal of Mathematical Sociology, 24, 1, 67-96
Van Notten, P. W., Rotmans, J., Van Asselt, M. B., & Rothman, D. S. (2003) An updated scenario typology,
Futures, 35, 5, 423-443.
959

Behind the Scenes of Scenario-Based Training: Understanding Scenario Design and Requirements in High-Risk and Uncertain Environments

More Related Content

Similar to Behind the Scenes of Scenario-Based Training: Understanding Scenario Design and Requirements in High-Risk and Uncertain Environments (20)

More from COMRADES project (19)

Recently uploaded (20)

Behind the Scenes of Scenario-Based Training: Understanding Scenario Design and Requirements in High-Risk and Uncertain Environments