The Science of Cyber Security Experimentation: The DETER Project

The Science of Cyber Security Experimentation:
The DETER Project
Terry Benzel
USC Information Sciences Institute
4676 Admiralty Way #1001
Marina del Rey, CA 90292
+1-310-822-1511
tbenzel@isi.edu

ABSTRACT cyber-security experimental science.
Since 2004, the DETER Cyber-security Project has worked to Our vision for the next iteration of DeterLab is a facility that is
create an evolving infrastructure – facilities, tools, and processes – used as a scientific instrument, where researchers create
to provide a national resource for experimentation in cyber knowledge and understanding through observation, modeling, and
security. Building on our insights into requirements for cyber measurement of computer and cyber security threats and defenses.
science and on lessons learned through 8 years of operation, we We are actively engaged in research and development to improve
have made several transformative advances towards creating the the scope and usability of this instrument, throughout the life
next generation of DeterLab. These advances in experiment cycle of an experiment – definition, execution, and interpretation.
design and research methodology are yielding progressive An additional and fundamental goal is that the scientific facility
improvements not only in experiment scale, complexity, diversity, will be to support diverse research communities, and enable users
and repeatability, but also in the ability of researchers to leverage to contribute to and leverage a common body of tools and
prior experimental efforts of other researchers in the DeterLab knowledge. That leverage will enable researchers to build on one
user community. This paper describes the advances resulting in a another's experimental work, with reproducible experiments and
new experimentation science and a transformed facility for cyber- repeatable experimental procedures.
security research development and evaluation.
We believe that this evolution of DeterLab will enable shifting the
science of cyber security experimentation towards rigorous
Categories and Subject Descriptors design, construction, execution, and interpretation of experiments.
D.4.6 Security and Protection D.4.8 Performance Measurements - Such a shift is required to advance the scale, pace, and power of
Modeling and prediction, Monitors. cyber-security research, and to expand the research community
and accelerate their work of developing the innovative,
scientifically tested, and demonstrably effective new cyber-
General Terms defenses required to meet challenges ranging from personal digital
Algorithms, Management, Measurement, Performance, Design, security to national-scale critical infrastructure protection.
Economics, Reliability, Experimentation, Security, Verification.
In pursuit of this goal, the DETER project's research program has
been influenced by several lessons learned from our experiences
Keywords in evolving DeterLab, and from supporting and interacting with
Cyber-security, testbed, experimental research many researchers using DeterLab. Those lessons have driven
several new areas of work on developing cyber-security science
1. INTRODUCTION: THE DETER methods and technology, that is, the tools, techniques,
PROJECT methodology, resources, and facilities that are needed to provide
DETER is a research project that is advancing cyber security DeterLab researchers with major advances in their scientific and
research practices, by extending the methods, technology, and experimental capabilities: repeatable, flexible, and variable
infrastructure required for scientific development of cyber-defense experimentation, discovery, observation, and testing.
technology [1], [2]. Our research results are put into practice by This paper provides: some background motivation and history for
operating and evolving DeterLab, transforming it from a basic the DETER project; a review of our key advances in creating a
hardware-focused network security testbed within the early phase science of experimentation, and lessons learned from these
of the project, through the shared experiment and testing lab of the advances; a description of the current DETER research program
middle phase of DeterLab use, towards a new kind of facility for and roadmap; and a prognosis for the use in DeterLab of the
resulting innovations in cyber-security research practices and
infrastructure, as we progress toward our vision of a new
Permission to make digital or hard copies of all or part of this work for environment and community for science based cyber security
personal or classroom use is granted without fee provided that copies are experimentation and test.
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, or 2. BACKGROUND: MOTIVATION AND
republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee.
HISTORY
ACSAC’11, Dec. 5-9, 2011, Orlando, Florida, USA. The DETER project’s creation grew out of a set of related
Copyright 2011 ACM 978-1-4503-0672-0/11/12…$10.00. observations made within the computer and network security

research community, funding organizations, and security product existing tools, and the creation of a collaborative community of
companies: researchers.
• Security technology and development was largely re- The testbed became operational in March 2004. The first DETER
active in nature. Community Workshop was held in November 2004, with working
groups of researchers who published refereed publications on
• Security technology development was slower in pace
work performed in the DETER testbed covering, e.g., DDOS
than the evolution of existing threats and the emergence
defense [5], worm dynamics [6], worm defense [7], and detection
of new threats.
of routing infrastructure attacks [8]. The ensuing years saw
• Successfully and widely deployed security technology maturation of the testbed through use and expansion, and growth
(host security, communication security, network of the research community with a greatly increased breadth of
boundary control) could be tested with common activity. Both DETER researchers and community collaborators
equipment at small scale. worked on research topics in the technology for supporting and
• Emerging threats, not addressed by deployed security enabling cyber-security research work: experiment automation,
technology, operate at Internet scale (worms, DDOS); benchmarking, scaling via hypervisor usage, malware
requiring radically new classes of defense, and hence containment, and our initial work on federation [9], now a central
component of DeterLab technology.
radically new evaluation strategies for these defenses,
that focus on scale and aggregate behavior. In the second phase of DETER, 2007-9, the results of this
“research on research” – our exploration of novel technologies
• New security approaches (e.g., behavioral anomaly
and methodologies for cyber-security research – were put into
analysis) also need large scale and highly varied testing.
practice in the testbed, which was also expanded in capacity. The
• Security innovators lack the facilities to test new result was the evolution from the DETER testbed to DeterLab, a
security technology in test environments with scale and shared virtual lab composed of the underlying testbed resources,
fidelity to the real deployment environment, and technology for using and managing the resources as test fixtures,
typically construct their own test environment with little and a growing variety of tools and services for experiment
or no leverage from the testing work of other support, including the full-support release of the federation
innovators. capability, and the first-generation experimenters’ workbench,
A consequence of these observations was that promising new SEER [10].
security technologies, often from innovators with limited testing With the technological maturity achieved in this phase, and the
resources, fared poorly when tested by applied security experience gained from supporting over 1000 researcher team
practitioners in real deployment environments. [3] In such cases, members, the stage was set for DETER project activities to focus
technology transfer was problematic because of significantly increasingly on research and development in the areas of cyber-
lower effectiveness outside the innovator’s limited test facility. In security experimentation methodology, infrastructure, tools,
many cases, commercial organizations did not find it cost resource expansion, utilization innovations, and other new
effective to engage in further development to increase methods of using DeterLab for scientific experimentation.
effectiveness.
The experience during this phase included several lessons learned
With this background in 2001-2002, one of the several factors of that provided important guidance for the current DETER research
cyber-security deficiency seemed to be addressable: the lack of program. The balance of this paper describes five significant
testing facilities with significantly greater resources and flexibility lessons learned, and outlines the current research program
than the limited test environments of most innovators, and greater developed from the combination of our vision of an advanced
fidelity to real deployment environments. One DARPA-sponsored scientific instrument, and lessons learned from the developing
report [4] called for and stated requirements for a national cyber- community.
defense technology test facility. One result of that report was the
impetus for funders at NSF and DHS to define and fund the 3. LESSONS LEARNED: CHALLENGES
project that was the first phase of DETER. FOR SCIENCE BASED
The initial focus of DETER was to build such a national testbed, EXPERIMENTATION
enabling cyber security innovators to test new technology at larger Here we describe several lessons learned from early experience
scale, with more complex test fixtures, assembled to be more with DETER. These lessons derive from observing norms of
representative of real deployment environments. The first-phase researcher activity that emerged in the 2nd phase of DeterLab use.
DETER project (led by USC/ISI, UC Berkeley, and Sparta, Inc.) Those norms included some basic terms and a typical workflow.
was funded by two NSF programs – Experimental Infrastructure Researchers working in DeterLab are called “users” or
Network (EIN) and Network Research Testbeds (NRT) – and by “experimenters”. A team of users working together is called a
DHS. At the same time EIN and NRT co-funded the EMIST “project.” The focus of activity of a project is a construct called an
project, composed of researchers from Penn State, McAfee “experiment” – a term that applies whether the construct is used in
Research, ICSI, Purdue, Sparta, Inc., SRI International, and UC the strict sense as part of activity to prove or disprove a
Davis. EMIST researchers were to use the DETER testbed, help hypothesis, or the construct is used for broader purposes such as
build knowledge about researcher’s needs based on experience observing malware behavior, measuring the effect of counter-
working in the testbed, and build experience with existing testing measures, or other forms of testing or observation that could
tools used in the testbed. Together these efforts led to the success contribute to hypothesis formation or demonstration of
of the first phase of DETER, with the assembly of the network effectiveness of a counter-measure.
and physical resources, development of controls and user
interfaces for experimenters, assessment and integration of The concept of an “experiment” is broad. In DeterLab’s
vernacular the term “experiment” is used at minimum, to describe

the experimental apparatus or environment that users have must be network-configured with network interface devices, IP
constructed from computing and network resources, hosts, address, DNS and default route settings, etc.
software, test fixtures, measurement tools, and other fixtures or In addition to the level of effort required, we also had a concern
components of a system that experimenters operate. Beyond this, about methodology – in most cases, each experimenter had to do
“experiment” is also frequently used to describe the experimental almost all the above activities, with very little leverage of
procedures and experimenter activity to interact with an apparatus previous experimenters’ work, other than perhaps using published
in operation, and to review results of operation – that is, the entire network definition files as a model or template. Our own work
“experimental protocol” for a particular research activity. [13] to build a reference case of a DDOS experiment, designed for
A large portion of our learning consists of observations and user re-use and extension by others, was instructive and useful, but
feedback about the process of designing, building, and carrying also served to highlight the large amount of required detail that
out an experiment. Experimenters typically work in a high-level was not central to some researchers’ work, but was nevertheless
workflow that consists of: initial design and construction of an pre-requisite to a functioning experiment.
experimental apparatus; exploratory operation of the apparatus In other words, the experiment definition methodology lacked
during iterative construction; initial full test runs of operating the abstraction and re-use. Acceleration of the pace of cyber-security
apparatus; reviewing and analyzing results of test runs; iterative research was blocked by the necessity of each experimenter
re-working of the apparatus, towards a fully satisfactory apparatus needing to specify a great deal of structure, much of which was
that repeatably operates as expected, and generates output such as not critical to their needs, and without recourse to others’ work.
log data, network traces, etc.; analysis of output data to create Our lesson was that the construction part of the experiment
experimental results. We refer to this workflow as the lifecycle needed considerable additional automation, new
“experiment lifecycle.” Our goal is to change this lifecycle from a methodology, and supporting features for abstraction, data hiding,
manual human intensive ad hoc set of processes to a highly and re-use. As our research on higher-level experimental
automated set of processes tied semantically to an experimenter’s infrastructure support turned to “Experiment Lifecycle
model and exploration motivations. Management” (ELM), we added the objective that a new
experiment should be able to “stand on the shoulders of previous
3.1 Lesson Learned: Experiment experiments, rather than standing on their feet”.
Construction Considered Difficult
DETER’s original tools for construction of experimental 3.2 Lesson Learned: Diverse and Flexible
apparatus were inherited from Emulab [11], the base technology Federation Considered Valuable
of the original DETER testbed. These tools provided sophisticated DeterLab’s federation capability was originally conceived out of
but low-level capabilities for managing the physical computing our goal to expand the possible scale of an experiment by
and network resources of the testbed, and using them to create enabling an experiment to use not only DeterLab’s physical
emulated networks within which an experimenter’s activity took computing and network resources, but also those of other testbed
place. For our original set of EMIST researchers, this toolset was facilities, such as such as Emulab [14], PlanetLab [15], and GENI.
useful, because they were experienced researchers who valued the [16] The DETER team set up federation arrangements and
“expert mode” in which every detail of a test network could be functionality with such testbeds, both as part of our own research
specified. In addition, many of these early users were familiar in on federation, and also to benefit experimenters seeking larger
concept or in experience with other network testbeds. scale and/or more fidelity by introducing wide-area network
However, we quickly confirmed that the “expert mode only” topology into an experiment.
approach was limiting for many of our researchers, some of whom However, once the first-generation federation capability was used
were less concerned with network-centric security research, and by DeterLab researchers, we learned of additional benefits sought
more oriented toward security research that did not depend by them, beyond DETER-managed federation with other testbed
critically on an exactly specified network environment. Novice facilities. One type of additional benefit was the inclusion in an
DeterLab experimenters with modest research experience faced a experiment of unusual or unique resources available in specialized
steep curve to learn how to create an emulated network of low facilities, for example: SCADA systems and embedded
complexity, but useful for testing. For very experienced controllers, in national labs; supercomputing facilities in high-
cybersecurity researchers starting work in DeterLab, there was performance computing facilities; and rare, new, or very large
also a steep curve to learn how to create an emulated network of scale networking gear available in labs set up for testing them,
moderate complexity and realism sufficient for their work. such as the University of Wisconsin WAIL [17] facility.
One reason for the complexity of network definition lay in the In addition to this “specialized” integration of computing and
typical first step, preparing a file containing expressions in a network resources outside DeterLab, some researchers also sought
specification language to define each individual network node. federate with their own facilities and/or those of their
From “toy network” examples [12] one can extrapolate the level collaborators. Some additional degree of scale-up could be
of detail required to specify non-trivial network topologies. In achieved by joining those more ordinary resources “down the
addition to the topology, each node may need to be assigned hall” from the researcher with the larger scale resources in
certain properties so that, e.g., some nodes may serve as DeterLab.
background traffic generators or traffic delay nodes. Following
topology definition and attribute specification, there are further These types of desired federation were beyond the scope of the
steps: nodes acting as hosts need to be loaded with a boot image original federation model of linkage between an external network
of OS and application software, often followed by the addition of testbed and the testbed underlying DeterLab. To support these
other software, such as experiment-specific software, or packages types of federation – and others not yet conceived – we began to
for monitoring and logging host or network activity. All nodes address issues of resource usage policy and access control, and
enriched the federation mechanisms to support them, beyond the
originally conceived testbed-to-testbed federation.

3.3 Lesson Learned: Experiment Isolation The short-term approach involves the use of an ad hoc,
experiment-specific tunnel node as one node in an experiment
Considered Limiting apparatus, to permit other nodes to interact with outside systems.
During this same period, we noted that changes in the threat The tunnel node is specially prepared by DeterLab network
landscape required stark changes to the DETER facility’s initial operations, to implement network controls on outside access, but
conception of experimental environment. For example, the initial permit the interaction that the experimenter desired and that
intended methodology for malware experimentation in DETER DeterLab management would permit. Naturally this approach is
was to observe and capture malware in the wild, and then to run hardly scalable, requiring the use of scarce operations staff time,
the captured malware in a simulated network in the testbed, which and relying on informally stated requirements for outside
was fully isolated from the public network by a number of interaction.
extremely rigid segregation measures [18].
Recognizing the limits of this approach, we also began work on a
This approach quickly proved limiting for cases where the methodology for more flexibly managing CIA, by transforming a
software-in-the-wild has non-deterministic behavior; because in risky experiment into a safe experiment. Our work builds on a
this case the behavior of a copy in the testbed may have low single, simple fundamental observation:
fidelity to behavior in the wild. Another limitation is a timing
issue: for emerging threats, the time required for accurate capture • If the behavior of an experiment is completely un
from the wild may introduce delays in the experimenter’s ability constrained, the behavior of the host testbed must be
to test. completely constraining, because it can assume nothing
about the experiment.
As a result, we began to explore a methodology for “controlled
Internet access” from DeterLab, in order to explicitly support and • However, if the behavior of the experiment is
control valuable and effective, but also potentially risky, constrained in some known and well-chosen way or
experiments – that is, experiments that pose a risk to the outside ways, the behavior of the testbed can be less
world, or are at risk from the outside, in addition to the inherent constraining, because the combination of experiment
risk of using malware in a testbed or test lab. Some examples of and testbed constraints together can provide the required
experimentation to be enabled by risky experiment management: overall assurance of good behavior.
• Place in DeterLab some targets for malware in the wild, We refer to this approach as Risky Experiment Management
and observe in a controlled environment the methods of (REM) T1-T2 because it combines two sets of constraints, derived
attack; more expedient than trying to capture the from the above observation, to limit the overall risk of the
malware and accurately replicate its execution in the test experiment. We call the first sort of constraints “experiment
environment. Researchers at CMU and UC Berkeley constraints” or “T1 constraints”; these are constraints naturally
were some of the first to use the new controlled internet exhibited or explicitly imposed on the experiment. We call the
access in order to attract drive-by downloads. The second class of constraints “testbed constraints” or “T2
scenario was: a node in DETER visits some Web page, constraints”; these are constraints imposed by the testbed itself.
gets infected by malware and that malware instructs it to We often refer to overall concept as the “T1/T2 model.”
go visit other Web pages in unpredictable manner. Then Implementation of the REM-T1/T2 approach [20] will require
they were able to use the infected nodes and behavior to tools for formal definition of the experimenter’s requirements –
analyze the malware. [19] defining the T1 transformation – and methods and automation for
• Place in DeterLab some peer computing elements to defining the additional constraints that define the T2
join in collaborative computing in the wild, for example transformation. These advances will be required for risky
real anonymity services and infrastructure, the operation experiments to be defined, controlled, and permitted with more
of which is dependent on small-time changes in assurance than experiment-specific tunnel nodes. Section 4.4
behavior that are non-deterministic; more expedient provides further information on future efforts on REM-T1-T2.
than replicating a privacy network at scale in a lab, and
have the simulated behavior have high fidelity to real-
3.4 Lesson Learned: A Requirement for Scale
world behavior. and Fidelity Flexibility
In the initial DETER testbed, the basic approach to scalability
• Place in DeterLab some nodes to serve as bots in
rested on scaling up the amount of available hardware, and on a
botnets, to observe bot/botmaster behavior; more
fixed set of approaches to fidelity using individual nodes. By
expedient than trying to replicate botmaster behavior
“fidelity” we mean that in a testbed there is a set of nodes that
with the same software and the same human operator
exist to model the behavior of nodes in the real world, and the
behavior as real botmasters.
testbed nodes’ behavior in the simulated environment is behavior
The common theme – whether or not malware is used in DeterLab that is similar to real nodes in real environments. The fixed set of
– is that some software of interest has properties that depend on approaches to fidelity, in this classic approach, is a range from a
test fixtures or test procedures – specific software, or specific single testbed node acting like a single real node, with high
behavior, or human factors – that are difficult to replicate at high fidelity; to a single testbed node standing for a large number of
fidelity. Partly in response to experimenter requests, and partly internal network end-nodes.
from our desire to expand DeterLab’s capabilities to
In this approach, the maximum size of an experiment is
accommodate this common theme, we began work on both short-
essentially bounded by the number of hardware nodes. As we
term expedient approaches to controlled internet access and
built out DeterLab, not only did we want to increase the scaling of
longer-term approaches to flexibly manage this sort of
the hardware, but we also recognized that many of our users’
interactions.
experiments did not require high fidelity in every part of the

experimental apparatus. We recognized a class of “multi- over that obtainable with more simplistic use of physical
resolution” experiments [21] in which: resources.
• some parts of an apparatus require high-resolution Other lessons learned were about the dynamic aspect of running
nodes with high fidelity; an experiment. Early in the 2nd phase, we recognized the need for
a workbench that experimenters could use to operate an
• some other parts require a lower degree of resolution
experimental apparatus, feeding it input data and events,
and can represent real computing at a larger scale;
observing its operation, and adjusting the fixtures for collecting
• there is a “scale of scaling” with points that range from experimental data. The first-phase workbench, SEER [10], met
high fidelity and linear scaling, to low fidelity and high that need to some extent. However, adoption of SEER also
scalability; brought into focus a growing need for DeterLab experimenters: an
• different points on the scale will be enabled by different approach to the “big data” problem. As DeterLab facilities have
mechanisms for emulation and simulation. matured with scale and power and data capture capability, and as
observation of the behavior of a running experiment drove
As a result of this observation, we began to explore methods to improvements in data collection, the result was, for many
incorporate a number of representation methods that together experiments, a much larger set of output data to be analyzed from
provide a full spectrum of scale-fidelity tradeoffs for experimental each experiment run.
system components. The following is a partial list of examples:
Further, not only the size of data grew, but also the structure and
• a single hardware node running a single experiment complexity of the datasets increased. In addition to log analysis
node, either natively, or via a conventional Virtual tools to help deal with raw data size, there was a need for other
Machine Manager (VMM) supporting a single guest methods – and automated support for them – to analyze data in
OS; terms of the intended semantics of the experiment run, and
• a single hardware node running several virtualized ultimately to proceed from data analysis to actual experimental
experiment nodes, each a full-blown conventional results: proving or disproving a hypothesis, or stating knowledge
Virtual Machine (VM) on a conventional VMM; of malware behavior, or use of metrics for effectiveness of
countermeasures.
• a single node running a large number of lightweight
VMs on a VMM designed for scaling the number of In other words, experimenters need both tools and methodologies
experiment-nodes with limited functionality; for mining experimental data to discover experiment results. This
lesson learned served to underscore the importance of our
• representation of individual experiment nodes as threads research work on narrowing this large “semantic gap” as part of
of execution in a large-scale thread management our research efforts on Experiment Lifecycle Management.
environment;
• large-scale software-based network simulation [22].
4. CURRENT DETER RESEARCH
Further, we recognized that these methods would be more useful
PROGRAM
to experimenters if all methods were part of a unified framework Our current research program includes, but is not limited to,
for the construction of composable experiment apparatus, using activities related to the above lessons learned. Current research is
both some common building blocks and methods of composition in some cases an outgrowth of work performed as part of our
with abstraction and re-use. Our approach to such a framework is agenda to enrich the testbed with progressive enhancements that
to base on it on an abstract fundamental building block called a resulted in what we now call DeterLab. During the 2nd phase in
“container” which represents experimental elements at the same which we were learning from DeterLab users, our own
enhancement efforts included:
level of abstraction, and is the basic unit of composition for
constructing an experimental apparatus. The container-based • First generation of federation capabilities [9];
methodology is a key part of pursuing some important goals:
• Risky experiment management and abilities to include
• leverage DeterLab’s physical resources more flexibly to outside communication [16];
create larger scale experiments;
• The first-generation “experimenter workbench” for
• enable experimenters to model complex systems with managing an experiment in process, viewing its activity
high resolution and fidelity for the things that matter and results data [10].
most to them, and abstract out the less important
In some cases, there was real synchronicity between our
elements;
objectives and the needs of DeterLab experimenters. As described
• reduce the experimenter’s workload of experiment above, the first generation of federation capability arose from our
apparatus construction, enabling larger scale apparatus desire to reach greater scale by using resources in other testbeds
with lower levels of effort. that we could link to; in addition, we learned that experimenters
wished to link into their experiments some outside resources of
3.5 Lesson Learned: From Experimental their own, and/or specialized resources that they had access to. As
Apparatus to Experimental Data to a result, our research agenda (for federation with access control)
was enriched with new use cases and additional requirements.
Experimental Results
The previous four lessons learned were largely related to the static 4.1 Experiment Lifecycle Management
aspects of setting up an experimental apparatus: basic construction Experiment lifecycle management is an outgrowth of work on our
of an apparatus; use of fixtures for federation; use of fixtures to first generation workbench, SEER. Indeed, many of SEER’s
enable limited communication outside the testbed; and use of capabilities, including experiment monitoring and visualization,
fixtures that support orders-of-magnitude experiment scale-up are carried over into the next generation workbench, the

Experiment Lifecycle Manager (ELM), albeit in a new usage apparatus. In ELM, all these types of experiment resources,
paradigm. elements, fixtures, and artifacts do of course need to be managed
One critical aspect of ELM focuses on the general concept of as individual objects, as building blocks for components of an
objects that an experimenter uses. DeterLab has grown to include experiment. More importantly, we’re working on construction
a large number and variety of objects available to experiments. methods that include both the basic building blocks, and also
With that growth has come the challenges of giving experimenters structures of them that prior experimenters have contributed. In
the tools need to effectively manage their working set, and other words, with ELM, experiments can be highly modular, and
(critically) to effectively share with other experimenters. The explicitly structured for re-use as shown in Figure 1.
objects used by an experimenter include scientific, physical, Although the detail-oriented “expert mode” is still available, we
communication, and computational resources used in an expect most researchers to use the newer facilities for defining an
experiment. Also included are models, designs, procedures, experiment’s components abstractly, with requirements,
programs, and data. Storage, presentation, archival, browsing, and constraints, and invariants, rather than specify directly and in
searching are basic ELM functions for managing an experiment’s every detail. For example, an earlier experiment may already have
components – and allowing other researchers to access them – far defined an apparatus that simulates a handful of large enterprise
beyond the original testbed approach of shell login and filesystem networks connected over the public network, a number of ISP
access. We are building this basic management framework on the networks, and home computers. This apparatus, though conceived
Eclipse [23] platform, in order to leverage and build upon the for use in worm spread, may nevertheless be described with meta-
many integrated development environment (IDE) capabilities of data that enables a later researcher to identify it as a suitable
Eclipse. starting point for their work. The later researcher should be able to
use the archived design, and state some new requirements and
constraints relevant to their work, or specify some properties of
specific experiment fixtures for input generation or monitoring.
Without having to know other detail beyond their requirements,
experimenters can describe an experiment apparatus entirely
independent of its realization on computing and network
resources.
Thus far, the description of ELM is analogous to an IDE with
source code repositories, modules, libraries, facilities for
combining them, with shared storage, versioning, and change
control – all valuable advances from the early DETER testbed.
However, ELM also provides other critical facilities analogous to
an IDE:
Figure 1: Screenshot of an experimenter using ELM to view a • Mechanisms for “realizing” an abstract, modular
catalog of experiment components, and select and view a experiment definition by allocating and configuring real
network topology displayed visually network and computing elements.
New levels of abstraction in experiment definition are also a key • Tools for interpreting experimental data to yield
component of ELM. In the original testbed approach, information that expresses experimental results in terms
experimenters had to specify in detail a number of different types of the experiment’s model and the abstractions that
of resources: helped define the apparatus.
• Computational elements such as physical or virtual Following sections describe some of our work on advances in
hosts, and the complete “network plumbing” realizing and running experiments at scale, and on model-based
configuration of each. experimentation that enables semantic analysis of results. That
work is directly reflected into the ELM methodologies and tools
• Elements of a network operating environment, including mentioned above.
network topology, router and switch nodes and their
configurations. 4.2 Containers: Scale-up and Flexible Fidelity
• Network nodes that perform traffic shaping to simulate Our continuing work on scalability is based on the observations
real world network conditions, delays, throughput (summarized in Section 3.4) about trade-offs between the fidelity
limits, etc. or realism of a computational element in DeterLab, and the scale
of network and computing resources required to realize a
In addition, experimenters had to specify in detail a number computational element. However, re-usability is also an important
experiment elements running within the network and goal for the ease of use of DeterLab tools for constructing an
computational resources: host operating systems, guest operating experimental apparatus. By adding new types of computational
systems for VMs, application software, and logging and other element (conventional VMs, QEMU lightweight VMs, processes
infrastructure software typical of real systems. Further, on conventional OSs, QEMU processes, individual threads of
experimenters had to deploy on these systems a number of execution), each of which can be used to model a node in a
experimental fixtures such as traffic generators, tools for running simulated network, we added both flexibility and complexity to
experimental procedures and collecting result data, and often the methods of constructing an apparatus.
malware to be observed and cyber-defenses to be tested.
To manage complexity and increase ease of construction, we are
Perhaps most significantly, each experimenter tended to do their developing an apparatus framework centered on an abstraction
own apparatus construction largely from the ground up, with that we call a "container" [21]. In our new construction
limited leverage of others’ work in defining experimental

methodology, a container is the fundamental building block. A components of them, and for other experimenters to use these
single container may support one or multiple components archived items as building block for the construction of a new
(elements) of an experimental apparatus, and implements an apparatus definition. The definitions can then be used with core
abstraction layer that hides the details of the inner components, DeterLab embedder capabilities for realizing the definition in a
when that container is itself placed inside another container. real apparatus composed of DeterLab network and computing
Figure 2 shows a simple container that contains no other resources. Again, there is work to do building the workbench
containers, containing only 2 concrete computing elements, such technology for containers being one of the objects that can be
as a VM or thread. Abstraction is provided by the container's archived and reused. Likewise, there is work to do with the
communication mechanism, which both connects the contained embedder, to automate realization with little input from
elements with one another, and also presents an entry/exit point experimenters, while also giving experimenters visibility on
for communication into the container; the communication embedding so that they can vary some of resource utilization or
mechanism advertises to other containers the properties of its vary the fidelity/scale tradeoff points for a specific apparatus.
container.
4.3 Model Based Experimentation
As described in Section 3.5, DeterLab experimenters have a “big
data” problem that will only grow as DeterLab magnifies the scale
available to experimenters, and the breadth of tools for collecting
experimental data. Our approach to this problem has been to
completely re-conceive the methodology for how cyber-security
experiments are defined in order to yield data to be analyzed for
information that comprises experimental results.
The basis for this approach is no more or less than adopting basic
ideas from other experimental sciences that are more mature than
experimental cyber-security is at present. The conceptual starting
point of an experiment is a real-world situation that displays an
interesting problem that is inconvenient to investigate in situ in
the real world. Instead, we define a conceptual model of the
situation, and begin to define laboratory activity that allows us to
construct in the lab a physical (or chemical, or biological, or
informatic) model of the real-world situation. Part of the function
of this model is to serve as a design for an experimental apparatus
Figure 2: A simple container of two basic computing resources that the experimenter will observe or modify in order to make
Abstraction is a central point for continuing to expand the inferences from lab observations to the real world where
scalability options in DeterLab. Other researchers frequently analogous modifications may create analogous results.
create new techniques for scalable computing, or new methods of This common methodological framework is, however, somewhat
using virtualization or simulation, or performing a form of unusual for some areas of computer science, where much research
lightweight computation. Our goal going forward is to incorporate is in fact in situ – modify an actual computational or
promising work in this area, defining a new abstract type of basic communication system to observe whether it better meets the
computing element, creating a standard interface for a researcher’s goal for speed, efficiency, data access rates, power
containerized component based on each new technology, and utilization, etc. Cyber-security research, however, is very
expressing its tradeoffs explicitly, for use in construction tools. definitely model-based. The real world has a host of large-scale
Thus far, our containers work has been successful for scalability systems with complex structures with vulnerabilities and potential
for multi-resolution experiments, and has illustrated the mitigations. Where experimental modification of real systems (for
requirements for integrating container technology into the ELM example, induced cyber attack) is not feasible, we use a lab
workbench. In addition to the expansion of available basic environment to create a model, an apparatus to approximate the
computing elements described above, there are several areas of model, experimental procedures to observe or induce changes in
ongoing work. the apparatus, and so on.
Apparatus construction: We demonstrated feasible levels of However, the early stage use of the DETER testbed was not
scalability and complexity by creating several experiment significantly model-based. Figure 3 is somewhat whimsical but
apparatus, the largest containing over 600 components that were accurate view of experimentation in which modeling is entirely
then realized on 8 physical computers and is capable of mental, with no externally visible relation between the model and
representing 50,000 computers at a coarse granularity in a specific an ad hoc constructed apparatus, or its operation. The lab
scenario. The low end of this spectrum would be a modest sized procedures are in fact quite valuable to the researcher, but ad hoc,
mixed-resolution experiment. The high end would be a large and difficult to document or to be repeated by others. Nowhere is
experiment in which all but a few elements had low fidelity. the ad hoc nature more evident in the research’s unique ability to
However, construction involved manually matching each element pore over network traces to find variations in worm behavior that
in a desired network topology with a specific container. Clearly, may be attributed to worm propagation countermeasures.
there is promising work to do in automating this process, and
integrating containers into the apparatus construction mechanism
of our new workbench, ELM.
Re-usability and embedding: ELM provides the ability of
experimenters to archive experimental apparatus definitions, or

Realization of Worm Experiment

Command &
Victim Victim Control (VMs)
(Physical Host)
Command &
Control

Network
(VMs)
Network

Ad Hoc Process: difficult to repeat
Figure 4: The iterative aspect of the experiment lifecycle
Figure 3: An informal experiment model leading to an ad-hoc As in any scientific discipline, often the greatest challenge lies in
experiment apparatus creating an appropriate representation of the object for study,
The DETER research program includes work to assist researchers representative across the measurement dimensions that matter,
in defining several model-related structures that become part of while carefully documenting the simplifying assumptions and
the methodology for building experimental apparatus, defining abstractions that are made in the process. While the most general
experimental procedures, and analyzing experimental data for case of this problem is very hard, we are working to extend
expected (or unexpected) patterns or changes predicted by the DeterLab’s experimenter support back into the early reasoning
model or a hypothesis derived from it. One purpose of modeling process of experiment design. We approach this through a set of
and semantic definition techniques is to define specific Model Based Scenario development techniques, in which
measurements and expectations to be sought for in the results of experiments are constructed from a basis in general models that
an experiment’s operation in the lab. capture the behavior of different dimensions of cyber security
experiments.
This model-based approach requires new cyber-security
methodology and new lab technology, integrated into the already-
described experiment lifecycle facilities, but oriented to defining
semantics for an experiment and its results, validating an
experimental apparatus, and extracting understanding from
results. Several types of tools under investigation can potential
benefit:
• semantic mechanisms to capture the intent of the
experimenter;
• support for monitoring this intent and distributed
execution with breakpoints;
abstraction and modeling techniques for experiment design,
realization, visualization, and analysis.
The use of these tools is inherently iterative, shown in Figure 4.
An experimenter defines a model, an apparatus to implement it,
procedures to operate it; then runs the experiment by operating the
apparatus, executing software or manual steps to perform
experimental procedures; the resulting data is interpreted to
extract information, which can then be used to iterate on the Figure 5: Development of experimental knowledge from
experimental apparatus, measurement scheme, or procedures. experiment data
To date, most experimentation presumed the existence of some Using the workbench and tools that we are investigating, an
form of model of the system under test that the experimenter uses experimenter may be able to refine the models into apparatus
to map his experiment objectives onto an apparatus in an templates or experiment-procedure definition or recipes, which
experimental facility such as DeterLab. While this has often been can be used to plan data analysis. The analysis would not be
true for the low-abstraction-level tasks of defining network bottom up or ad hoc pattern based, but rather following a
topologies and traffic generator configurations, the lack of rigor in knowledge discovery procedure (shown in the middle of Figure 5)
the initial steps often undercut the efficacy of the entire that is derived from the model and its various components and
experimental process by providing little guidance or expectation formalized assumptions such as expected behavioral invariants or
for the resulting experimental data, and no ability for knowledge- functional constraints. In other words, we are working towards a
based iteration. shift in methodology where new tools assist experimenters in
rigorous construction, execution, and interpretation of
semantically validated experiment design and execution.

During the specification phase of an experiment, invariants multiple sites and will enable the use of specialized resources to
associated with the model will be used to verify that the participate in large-scale experiments.
experiment being developed is internally consistent. During We view current research efforts such as the Smart Grid and other
execution, invariants will be used to ensure that the intended emerging cyber physical domains as new use cases for examining
experimental semantics are realized (validity management). and validating the evolving features and capabilities of the
Finally, invariants will be invoked as part of the interpretation and DeterLab that we are developing as part of the DETER project
visualization of results – providing methods for the experimenter research program.
to tackle large amounts of data in order to determine whether an
experiment run produced expected results or potentially 4.4 Additional Directions
interesting unexpected results. The three previous sections have outlined some of the key areas of
A simple example is the development of a model state space for our current research work, some of which was guided by lessons
execution of (and potential attack on) a communication protocol. learned, in addition to the results of our own research. The
A variety of data (packet dumps, web server logs, auth logs) can research program also includes areas for future work, in which we
be normalized for input into analysis and visualization tools that will extend the results of earlier topics by applying to them some
assist the experimenter in mapping from actual events to expected of the advances that we are making in our current research work.
behaviors. Figure 6 shows a conceptual view of the model state Risky experiment management is one area of prior work that also
space, with various possible paths through it; a path to the occupies a place in the roadmap. To put into practice the
“success” node would be expected results of experiment execution management approach described in Section 3.3, we will need to
(visible in detail in event logs), while other paths indicate a (a) develop DeterLab facilities for an experimenter to develop and
violation of an assumption about correct behavior, which may be refine specifications of their experiment’s requirements for
detectable sign of an attack or malfunction (accompanied by a Controlled Internet Access, and (b) develop automation tools to
particular reason for the violation, attributable to event logs). create an experiment-specific gateway node. The automation tools
will need to both implement the experimenter’s requirements, and
also implement DeterLab’s constraints defined in the T1/T2
approached described in Section 3.3.
For risky experiment management, this future elaboration will
depend on the results of two areas of current research activity. The
modeling and specification work (described in Section 4.3) will
provide key elements of the experimenter facility to define
constraints and invariants on the experiment’s communication via
controlled internet access. The container work (described in
Section 4.2) will enable DETER project research staff to create
reusable building blocks for gateway implementation, each with
advertisements that will assist the automation tools in constructing
a container to serve as a gateway node that implements the
required controls for controlled internet access as needed by the
particular experiment.
A second part of the research roadmap is elaboration of prior
work on federation, to support a new form of DeterLab
experimentation. A multi-party experiment is one in which the
experimental apparatus is built from sub-components that are
Figure 6: An example of a semantic model for an experiment
federated to create the whole, and each federant has complete
Model based experimentation takes on an increasing importance information only about their own sub-component, with only
when designing experiments that span both cyber and physical partial information about other sub-components. This form of
elements. The physical components are likely based in some set experiment can be used to model several different kinds of cyber-
of models (real world, empirical, or theoretical). In order to defense situations: adversarial situations (e.g. red-team/blue-team
capture the interactions and relations between the cyber and exercises); realistic forensic or defense scenarios (e.g., attack
physical, it will be necessary to compose models. Recent work in target with limited information about attacker); or partial
Secure Smart Grid Architectures [24] argues that: collaboration situations in which separate organizations
“An analysis of the cyber-physical security of a smart collaborate on defense without granting full visibility to
grid architecture must focus on the impact of faults and collaborators.
interactions that cross domains rather than the localized Support for multi-party experimentation will depend on the
response that might be seen in traditional penetration current full-production federation capability in DeterLab, and the
testing. This requires a capability to model large scale results of several areas of current DETER research: modeling and
response to cyber-attack, as well as to perform modeling specification work (described in Section 4.3) to state constraints
or simulation of the physical components of a system.” and invariants on activities of each party; and the container work
The Smart Grid security team at USC-ISI and JPL are currently (described in Section 4.2), which is essential to scale out each
using DeterLab to develop a secure smart grid architecture. They party’s sub-apparatus to realistic proportions needed for the types
ave created a taxonomy of cyber and physical threats and are of multi-party experiments currently envisioned.
exploring federation of DeterLab with other labs and testbeds that
provide physical simulation and emulation tools for modeling the
systemic response of the grid. Such experiments will span

4.5 Integrating the Pieces: Towards a New the scientifically demonstrated effectiveness of innovations as
they move from the lab into practical use.
Experimental Cybersecurity Research
Paradigm 5. SUMMARY
The above areas of current research and future research roadmap The DETER project is continuing with its research program, much
provide the foundation for our program towards new science of which has been described in brief in this paper. Our research
based experimental cybersecurity. Our focus is work is extending target continues to be the technology and methodology needed for
DeterLab new capabilities resulting from work in these areas, as the practice of cyber-security research to become an experimental
well as integrating the new and existing capabilities. The science, with rigorous experiment design, construction, execution,
integration is critical, including functional integration with the interpretation, sharing, and repeatability of experiments. We
new ELM workbench; but more important is integration into a believe that such a transformation of cyber-security research is
new methodology for the experiment lifecycle. Five of several needed to expand and accelerate the research community’s efforts.
possible lifecycle phases are illustrated in Figure 7: Expansion and acceleration are becoming increasingly important,
as inter-connected cyber systems continue to expand into nearly
every area of human infrastructure, from critical infrastructure for
communication, finance, transportation, and power, to networked
computing systems that are intimately integrated into our lives:
our vehicles, our handheld wireless personal computing and
communications (“smart phones”), regulation of our homes’
utility use (“smart meters”), remote control and regulation of our
homes’ fixtures (“smart grid”) including safety-critical appliances,
as well as our healthcare, and even our bodies, with networked
controllers for implanted medical devices.
From national and international security to personal safety, the
number and variety of targets continues to increase along with an
accelerating expansion of the total attack surface available to
adversaries who have an increasingly powerful portfolio of tools
and customers. When the DETER project started, cyber-security
Figure 7: New cyber-security research methodologies technology development and technology transfer were often
These are resource intensive, and often lacking in proactive approaches for
asset protection to be sufficient to increase the level of cyber
• a new methodology for specifying experiments, defense of critical assets. In the past 8 years, we have built some
including model-based specification, and elements of significant infrastructure for the development and test of new
previous experiment descriptions; cyber-defenses. As we look ahead to the coming years, we expect
• new tools to completely flesh out the structure of an that transformation of research tools and methods will contribute
experiment, with only the essential elements and to the much needed expansion and acceleration of research, which
abstractions; can lead to an accelerated pace of deployment of scientifically
tested and effective cyber-defenses.
• new technology for realizing the conceptual structure of
an experiment, by embedding it in a subset of Continuing maturation of the DeterLab facility is necessary, but
DeterLab’s real and virtual resources for computation so is the accelerated growth of a cyber-security research and test
and networking; community that can rapidly leverage one another’s work. In
addition to the research and methods described in this paper,
• new facilities and new controls that enable larger scale further development of the cyber-security experiment science
and more flexible use of federated systems and domain- community is a critical shared responsibility of the larger cyber-
specific resources – especially domain-specific security community.
resources that are available via federation; and
• across all of these areas, new mechanisms and facilities 6. ACKNOWLEDGEMENTS
to share experiment building blocks among This paper builds on efforts at ISI over the past 8 years and would
experimenters, who can accelerate their experiment- not have been possible without the contributions of the entire
creation work using the results and knowledge gained DETER team. Special recognition is given to: John Wroclawski
by previous work in DeterLab. for setting an ambitious research program; Ted Faber, Mike Ryan,
Alefiya Hussain and Jelena Mirkovic for delivering new
As we gain experience with this integration, we expect that we capabilities under that program; to Bob Braden for careful crafting
and DeterLab experimenters will develop cyber-security of proposals, capturing all of the prior work and proposed new
experimentation methodologies that can help to accelerate the work; and to John Sebes amanuensis extraordinaire.
pace of cyber-security innovation, and also dramatically improve

This research was sponsored by the US Department of Homeland Security
and the Space and Naval Warfare Systems Center, San Diego (contract
number N66001-10-C-2018), and the US National Science Foundation
(contract number CNS-0831491). Any opinions, findings and conclusions
or recommendations expressed in this material are those of the author(s)
and do not necessarily reflect the views of the Department of Homeland
Security or the Space and Naval Warfare Systems Center, San Diego.

7. REFERENCES of the DETER Community Workshop on Cyber Security
Experimentation and Test, June 2006
[1] Current Developments in DETER Cybersecurity Testbed
Technology T. Benzel, B. Braden, T. Faber, J. Mirkovic, S. Schwab, [14] An Integrated Experimental Environment for Distributed Systems
K. Sollins and J. Wroclawski. In Proceedings of the Cybersecurity and Networks, by White, Lepreau, Stoller, Ricci, Guruprasad,
Applications & Technology Conference For Homeland Security Newbold, Hibler, Barb, and Joglekar, appeared at OSDI 2002,
(CATCH 2009), March 2009 December 2002
[2] The DETER Project - Advancing the Science of Cyber Security [15] Operating System Support for Planetary-Scale Network Services. A.
Experimentation and Test. Terry Benzel, Jelena Mirkovic, et al. Bavier, M. Bowman, B. Chun, D. Culler, S. Karlin, S. Muir, L.
IEEE HST 2010 Conf, Boston, MA, November 2010 Peterson, T. Roscoe, T. Spalink, and M. Wawrzoniak. (NSDI '04),
May 2004
[3] Vulnerability Detection Systems: Think Cyborg, Not Robot, S.
Heelan. In IEEE Security and Privacy, special issue “The Science of [16] J. Wroclawski, J. Mirkovic, T. Faber and S. Schwab, “A Two-
Security,” Vol. 9, Issue 3, May/June 2011. Constraint Approach to Risky Cybersecurity Experiment
Management,” Invited paper at the Sarnoff Symposium, April 2008.
[4] Justification and Requirements for a National DDoS Defense
Technology Evaluation Facility, W. Hardaker, D. Kindred, R. [17] Barford, Paul; Landweber, Larry. Bench-style Network Research in
Ostrenga, D. Sterne, R. Thomas, Network Associates Laboratories an Internet Instance Laboratory, In Proceedings of SPIE ITCom,
Report, July 26, 2002. Boston, MA, August, 2002
[5] Cyber defense technology networking and evaluation, R. Bajcsy, T. [18] A Plan for Malware Containment in the DETER Testbed Ron
Benzel, M. Bishop, B. Braden, C. Brodley, S. Fahmy, S. Floyd, W. Ostrenga and Stephen Schwab, Robert Braden. In Proceedings of the
Hardaker, A. Joseph, G. Kesidis, K. Levitt, B. Lindell, P. Liu, D. DETER Community Workshop on Cyber Security Experimentation
Miller, R. Mundy, C. Neuman, R. Ostrenga, V. Paxson, P. Porras, C. and Test, August 2007
Rosenberg, J. D. Tygar, S. Sastry, D. Sterne, S. F. Wu. [19] T BitBlaze: A New Approach to Computer Security via Binary
In Communications of the ACM, Special issue on “Emerging Analysis. Dawn Song, David Brumley, Heng Yin, Juan Caballero,
Technologies for Homeland Security,” Vol. 47, Issue 3, pp 58-61, Ivan Jager, Min Gyung Kang, Zhenkai Liang, James Newsome,
March 2004. Pongsin Poosankam, and Prateek Saxena. In Proceedings of the 4th
[6] Preliminary results using scale-down to explore worm International Conference on Information Systems Security, Keynote
dynamics, Nicholas Weaver, Ihab Hamadeh, George Kesidis and Invited Paper. December 2008
Vern Paxson. In Proceedings of the 2004 ACM workshop on Rapid [20] J. Wroclawski, J. Mirkovic, T. Faber and S. Schwab, “A Two-
Malcode, pp. 65-72, 2004. Constraint Approach to Risky Cybersecurity Experiment
[7] A hybrid quarantine defense, P. Porras, L. Biesemeister, K. Levitt, J. Management,” Invited paper at the Sarnoff Symposium, April 2008.
Rowe, K. Skinner, A. Ting, In Proceedings of ACM WORM, [21] Ted Faber, Mike Ryan, and John Wroclawski “Building Apparatus
Washington, DC, Oct. 29, 2004. for Multi-resolution Networking Experiments Using Containers”, in
[8] Combining visual and automated data mining for near-real-time submission
anomaly detection and analysis in BGP, S.T. Teoh, K. Zhang, S.-M. [22] "Scalability and Accuracy in a Large-Scale Network Emulator,"
Tseng, K.-L. Ma and S. F. Wu, In Proceedings of ACM Amin Vahdat, Ken Yocum, Kevin Walsh, Priya Mahadevan, Dejan
VizSEC/CMSEC-04, Washington, DC, Oct. 29, 2004. Kostic, Jeff Chase, and David Becker. Proceedings of 5th
[9] T. Faber and J. Wroclawski, “A Federated Experiment Environment Symposium on Operating Systems Design and Implementation
for Emulab-based Testbeds,” in Proceedings of Tridentcom, 2009. (OSDI), December 2002
[10] Stephen Schwab, Brett Wilson, Calvin Ko, and Alefiya Hussain, [23] TSilva, Vladimir (11 March 2009). Practical Eclipse Rich Client
“SEER: A Security Experimentation EnviRonment for DETER,” in Platform Projects (1st ed.). Apress. p. 352. ISBN 1430218274.
Proceedings of the DETER Community Workshop on Cyber [24] Clifford Neuman and Kymie Tan, “Mediating Cyber and Physical
Security Experimentation and Test, August 2007. Threat Propagation in Secure Smart Grid Architectures”, To Appear
[11] Emulab Testbed Web page, http://www.emulab.net in the Proceedings of the 2nd International Conference on Smart
Grid Communications (IEEE SmartGridComm), October 2011,
[12] https://trac.deterlab.net/wiki/Topologies
Brussels
[13] DDoS Experiment Methodology, Alefiya Hussain, Stephen Schwab,
Roshan Thomas, Sonia Fahmy and Jelena Mirkovic. In Proceedings

The Science of Cyber Security Experimentation: The DETER Project

More Related Content

What's hot (15)

Similar to The Science of Cyber Security Experimentation: The DETER Project (20)

Recently uploaded (20)

The Science of Cyber Security Experimentation: The DETER Project