Week 7 - Model Experiementation ibn simulation modelling

Silas Maiyo (c) 2022
Model Experimentation –
Simulation Results
Based on Stewart Robinson (2004). Simulation: The Practice of Model
Development and Use
March 2022

Introduction
• Discussed:- Model experiments, how to design a simulation
experiment;
• To examine:- Issues surrounding the model
experimentation and simulation results

Simulation Model Experiments
• A simulation experiment is a test or a series of tests in which meaningful changes are
made to the input variables of a simulation model so that we may observe and
identify the reasons for changes in the performance measures.
• The number of experiments in a simulation study is greater than or equal to the
number of questions being asked about the model
• (e.g., Is there a significant difference between the mean delay in communication
networks A and B?,
• Which network has the least delay: A, B, or C? How will a new routing algorithm
affect the performance of network B?). Design of a simulation experiment involves
answering the question: what data need to be obtained, in what form, and how
much?
• The following steps illustrate the process of designing a simulation experiment.

Aims of Model Experimentation
•To obtain a better understanding of the real world system
that is being modeled and look for ways to improve that
system.
• ➔The experimentation should be carried out correctly;
• ➔The results that are obtained from the model should be
accurate;
• ➔The search for a better understanding and improvements
should be performed efficiently and effectively.

The Nature of Simulation Models
• Terminating and non-terminating simulations
Terminating simulation
=>One in which there is a natural end point that determines the length of a
run.
The end point can be where model reaches an empty condition e.g:
• A bank that closes at the end of a day;
• The end of the busy lunch period in a supermarket;
• The completion of a trace of input data, e.g. the completion
of a production schedule

• Non-terminating simulation
Þ One that does not have a natural end point.
Example
A model of a production facility that aims to determine its throughput
capability.

The Nature of Simulation Output
• Transient output
ÞIs one for which the distribution of the output constantly changes.
ÞTransient outputs arise from terminating simulations.
Example
The number of customers served each hour in a bank. This is
different day by day.

• Steady-state output
ÞThis one in which a steady state is reached when the output varies according
to some fixed distribution (the steady-state distribution).
Example
• A production facility;
• Throughput varies daily due to breakdowns, changeovers and other
interruptions;
• In the long run, the throughput capability (the mean throughput level)
remains constant;
• In a steady state the level the variability about the mean remains constant.

• Steady-state output
ÞInitial transient
• This is a period in simulation when the model starts at a low level and gradually builds
up to its steady-state level.
• It occurs at the start of a run (initial) and because the distribution of the output is
constantly changing (transient).
ÞInitialization bias
This the bias on the output due to inclusion of the initial transient data
that is unrealistic.
Initial data should be ignored until the steady-state is reached.

• Steady-state cycle output
ÞThese are cycles that arise due to participation of several components such
as where there are two shifts.
ÞHandle it by lengthening the observation interval in the time-series to the
length of the longest cycle.
Example
Instead of recording hourly throughput or throughput by shift in the
production the data can be recorded daily.

How to design a simulation experiment
1. Select appropriate experimental design
2. Establish experimental conditions for runs
3. Perform simulation runs – according to step 1-2 above

Select appropriate experimental design
• Select a performance measure, a few input variables that are likely to
influence it, and the levels of each input variable.
• When the number of possible configurations (product of the number
of input variables and the levels of each input variable) is large and
the simulation model is complex, common second-order design
classes including central composite, Box-Behnken, and fullfactorial
should be considered.
• Document the experimental design.

Establish experimental conditions for runs
• Address the question of obtaining accurate information and the most information from
each run.
• Determine if the system is stationary (performance measure does not change over time) or
dynamic (performance measure changes over time).
• Generally, in stationary systems, steady-state behavior of the response variable is of
interest.
• Ascertain whether a terminating or a nonterminating simulation run is appropriate. Select
the run length.
• Select appropriate starting conditions (e.g., empty and idle, five customers in queue at time
0).
• Select the length of the warm-up period, if required.
• Decide the number of independent runs - each run uses a different random number stream
and the same starting conditions - by considering output data sample size.
• Identify output data most likely to be correlated

Perform simulation runs
• Perform simulation runs according to steps 1-2 above
• Sample size must be large enough to provide the required confidence
in performance measure estimates.

Issues in Obtaining Accurate Simulation
Results
• Inaccurate data → inaccurate prediction of the real system
performance.
• The main aim of simulation output analysis
Obtain an accurate estimate of the average performance.
• Two options for ensuring the accuracy of the estimates
➔ Remove any initialization bias;
➔ Generate, enough output data from the simulation so that an
accurate estimate of performance is made.

Obtaining sufficient output data: long runs
• Get enough output data from the simulation by: -
Performing a single long run with the model, the only option for a non-
terminating simulation;
Performing multiple replications, the only option for terminating
simulations.
ÞReplication
Þ This is a run of a simulation model that uses specified sets of random
numbers, which in turn cause a specific sequence of random events;
Þ To get a new replication: use a different random number set;
Þ What to do: perform multiple replications and take mean value of the
results.

Dealing with Initialization Bias: Warm-up and Initial
Conditions
Determining the warm-up period
ÞThe warm up period should be long enough to ensure that
the model is in a realistic condition.
ÞFor a non-terminating simulation go past the initial
transient period so that the model output is in a steady state.
ÞThere is need to have the model in a realistic condition.

Þ The methods include:
ÞGraphical methods: involve the visual inspection of time-series of the
output data;
ÞHeuristics approaches: apply simple rules with few underlying
assumptions
ÞStatistical methods: rely upon the principles of statistics for determining
the warm up period;
ÞInitialization bias tests: identify whether there is any initialization bias
in the data;
ÞHybrid methods: these involve a combination of graphical or heuristic
methods with an initialization bias test.

Þ Note that;
ÞEach of the methods above has limitations;
ÞThere is no single one method that can be recommended for
all circumstances;
Þ They either overestimate or underestimate the length of the
initial transient;
ÞThey also rely on very restrictive assumptions and using highly
complex statistical procedures.

Time-series inspection
• This is a simple method for identifying the warm-up period;
ÞLimitation:- for a single run, with a very noisy data it can be hard to spot any
initialization bias;
ÞAlternatively:- run a series of replications use the mean averages of the
replications to plot a time-series;
ÞAt least five replications should be performed, more may be required for
very noisy data;
ÞMore replications make the time-series to be smoothed as outliers are
subsumed into the calculation of the mean for each period.

Week 7 - Model Experiementation ibn simulation modelling

More Related Content

Similar to Week 7 - Model Experiementation ibn simulation modelling (20)

Recently uploaded (20)

Week 7 - Model Experiementation ibn simulation modelling