The Art and Science Behind Successful Risk Workshops

// The Art and Science Behind
Successful Risk Workshops

Dr. Dan Patterson, PMP
CEO & President, Acumen

July 2009

+1 512 291 6261 // info@projectacumen.com

www.projectacumen.com

Table of Contents

Introduction ..................................................................................................................... 3

The Risk Assessment Process .......................................................................................... 3

Step 1: Schedule Review .................................................................................................. 4

Step 2: Risk Identification ............................................................................................... 4
Risk Events, Risk factors or Uncertainty Ranges? ........................................................ 5
Avoidance of Analysis Paralysis ................................................................................... 5
Table 1 – Uncertainty Factors ............................................................................... 5
Black Swan Risks ......................................................................................................... 6
Avoidance of Double Dipping ....................................................................................... 6
“What is the appropriate level at which to risk load a schedule?” ............................... 6
Loudest Doesn’t Always Win ......................................................................................... 6
It’s OK to Admit We Have Risk in Our Project ................................................................ 7

Step 3: Model Development and Analysis ........................................................................ 7
Avoidance of Central Limit Theorem ............................................................................. 7
Integrating Uncertainty and Risk Events ...................................................................... 7

Step 4: Interpretation of Risk Results .............................................................................. 8
The Danger of Reporting Confidence Levels ................................................................. 8
Range, Range, Range .................................................................................................. 9
How to Report Contingency .......................................................................................... 9

Conclusion ....................................................................................................................... 9

Checklist for a Successful Risk Workshop ....................................................................... 9

Additional Information .................................................................................................. 10

www.projectacumen.com 2

Introduction
Despite the project risk software tools available today, the process of conducting a risk
workshop and subsequent risk model development continues to be a major headache
for many projects. The process is often, at best, a necessary evil only executed to
follow company or project protocol.

With this perception, getting a project team engaged and on board with the risk
assessment process often can be as challenging as building the risk model itself.

So, why the big challenge? After living and breathing project risk management for over
a decade, I have come to the conclusion that the reason is this: when conducting a risk
workshop, too many projects rush towards the end goal without really understanding
how to get there or in many cases, not fully understanding the benefits as to why they
are even doing a risk workshop in the first case.

This paper discusses how a well-structured balance of risk process combined with
sound workshop facilitation can provide more value to a project’s bottom line than most
typically ever realize. Imagine a silver bullet that enabled you to objectively determine
accurate project costs; requirements for, and allocation of, contingency; strategic
insight into which projects should be considered for organization portfolio inclusion;
and, as a project manager most importantly, a true indicator as to how realistic the plan
is, against which you are being held accountable for during execution. Sound too good
to be true? Read on…

The Risk Assessment Process
If the objective of a risk assessment is to determine risk exposure, required contingency
and confidence as to whether the plan is adequately realistic, then the process needs to
critique not only the risks impacting the project but also the plan itself.

Think of risk on a project as the ‘anti-plan’. We can go to great lengths to develop the
perfect schedule or cost estimate but then during execution if either the scope changes
(uncertainty) or unforeseen events occur (risk events), then the plan is immediately out
of date and not reflective of what needs to be done for a successful on time, on budget
completion. With this, the taming of two moving parts should be considered:

1) All events, uncertainties and risk factors that may impact our ‘best laid plan’


2) The plan itself – consider the extreme case of a project with no risk: if the plan is
inaccurate or incomplete then there is still a high chance the project will still not
finish on time/to budget.

To give ourselves the best chance of developing a sound model, consider the following
four steps that will give us a framework for our risk assessment.

Step 1: Schedule Review
Step 2: Risk Identification
Step 3: Model Development and Analysis
Step 4: Interpretation of the results

Step 1: Schedule Review
Building a risk model against a sound schedule basis and/or cost estimate is paramount
to successful risk assessment. Apply risk and uncertainty to a structurally weak schedule
and the results will be as weak as the plan itself.
Some practitioners and tools today offer checks for quality of schedules. However,
simply running a set of checks isn’t sufficient. Accepting scheduling best practices such
as the absence of open ends, out of sequence updates, lags on links etc, there is a
further layer of intelligence that needs to be applied. Running computed schedule checks
is certainly a good first step but then more importantly, working with the planning team to
understand the presence of these so called errors and justification as to the absence of
perhaps certain areas of detail, ensures that the schedule basis is as accurate and
realistic as possible.

Based on facilitation of countless project risk assessments, Acumen has developed a
series of thresholds that act as quality and performance indicators enabling accurate
determination of the suitability of a project plan for risk assessment.

This early assessment of the plan also ensures team buy-in on the risk assessment.
Without this initial prep work and collaboration with those involved in the development of
the plan, the workshop itself often gets misconceived as a witch hunt from which a
defensive team then offers little insight into the true risk drivers within their project.

Step 2: Risk Identification
The pitfalls to avoid during risk identification warrant a thesis unto themselves: having a
sound risk identification process is key to the successful assessment of project risk
exposure.


Risk Events, Risk factors or Uncertainty Ranges?
Over recent years, sources and types of risk have been reasonably well documented.
Project teams are generally well educated as to the types of different risk ranging from
risk events and uncertainty factors to weather events, rig availability and other risk
sources. For this very reason, when the team is brought together in a risk workshop, it
is critical to truly identify how these risk types not only interact and potentially drive
each other but also how big an impact they truly have on the project. Approximating
them all under the umbrella of a ‘risk range’ will lead to tenuous results at best.

Avoidance of Analysis Paralysis
Estimate uncertainty is largely a result of incomplete or varying scope. Typically,
uncertainty is modeled using the likes of three point estimates (for example +/- 10%
range around the deterministic duration). However, in a risk workshop that may have
anywhere up to twenty highly opinionated team members, completing a walkthrough of
a detailed schedule gaining consensus can be extremely challenging. To overcome this,
consider using uncertainty factors. Rather than expecting a workshop team to quantify
minimum and maximum ranges to the nth degree, adopt a factor-based system as
shown in table 1 below.

Risk Factor Best Case (optimistic) Worst Case (pessimistic)

Very Conservative 75% 100%

Conservative 85% 105%

Realistic 95% 105%

Aggressive 95% 115%

Very Aggressive 100% 125%

Table 1 – Uncertainty Factors

Again, based on experiences from countless risk workshops, I have found teams are
more engaged and objective about risk ranging when thinking in terms of degree of
realism/aggressiveness. Map this back to an agreed upon quantitative score using a
mapping such as table 1 and the result is an extremely consistent and manageable
means of risk loading uncertainty during a risk workshop.

This technique also ensures that team members don’t inadvertently include risk events
in their uncertainty ranges.


Black Swan Risks
There has been a lot of noise made about black swan, extreme or so called ‘strategic
risks’ in recent publications. Yet in so many risk workshops, we still today, have to guide
the team away from slipping into the World War III type risks that result in ridiculously
skewed results. Impact of risk is of course a key consideration, but is it valid to include a
major risk event in a risk model that only has a 1 in 500 chance of happening during the
life of the project? Separate out risks that may impact the organization from those of the
project.

Avoidance of Double Dipping
A similar pitfall to avoid is that of ‘double dipping’. Risk ‘double dipping’ occurs when the
impact of a risk event is included in the general uncertainty range (often applied as a
min, most likely, max range of uncertainty) as well as being defined as an impact from a
risk event.

“What is the appropriate level at which to risk load a schedule?”
This a question posed frequently. The answer is simple: we can only report down to the
level at which we risk loaded a schedule. If we want to be able to determine
contingency spread at a level 3 within a WBS, then the schedule needs to be risk
loaded to at least this level. The flipside to this is that if we are working on a 10,000
activity schedule, it is simply not practical to risk load that many activities in a risk
workshop.

Again, through an evolved process, Acumen has developed a highly effective means of
risk loading ‘enough but not too many’ activities to ensure a balance of sufficient detail
and manageability for the team. This balance typically results in the order of magnitude
of two hundred risk factors/ranges being identified that then get applied to the various
disciplines/WBS groupings/sub sections within the project as a whole.
One of the benefits of this approach is that of continued maintainability of the risk
model beyond the initial risk workshop.

Loudest Doesn’t Always Win
Successful risk workshop facilitators have one key objective: to determine consensus
from the workshop team on realistic risk input values that will then be used to develop
the risk model. Risk workshops benefit from not having traditional project or
organizational hierarchical protocol. A domain expert should have as much say as the
PM when it comes to risk identification. In a recent workshop I worked with a highly
passionate and somewhat argumentative team, so I implemented a ‘wooden spoon’


system as to ‘who had the floor’. As the facilitator, I simply passed the spoon to the
team member wanting to voice an opinion. This forced the team to present well thought
out and structured arguments surrounding risk values and the net result was a
controlled, consensus driven set of highly accurate risk inputs.

It’s OK to Admit We Have Risk in Our Project
All too often, project teams are optimistic in their assessment of risk exposure. A risk
workshop should not be a destructive project moaning session. Instead, it should be
viewed by, and presented to, the team as a means of gaining consensus on project
confidence and subsequently what is needed to increase that confidence level.

Step 3: Model Development and Analysis
Once we have obtained the risk and schedule inputs (the building blocks of the risk
model), we then need to build a sound model against which to analyze. Over the years,
I have seen many approaches to risk model development yet in reality, very few truly
hold water. Due to the logistical challenges of risk loading very large schedules, one
approach is to create a manual rollup summary of the project and risk load this. This
approach is extremely dangerous: the task of re-creating equivalent logic on a summary
schedule that truly represents the inner workings of the detailed basis schedule is
extremely difficult and at best is an approximation.
Equally, shortcuts such as focusing on and risk loading of, the critical or near critical
activities can give a skewed perception and can obscure risk hot spots or areas of the
project highly sensitive to risk.

Avoidance of Central Limit Theorem
A proven approach is to identify and risk load both risk events and estimate uncertainty
at a suitable reporting summary level within the schedule, with the detailed activities
inheriting these summary ranges/risk events. A major pitfall to avoid here is that of
Central Limit Theorem (CRT): a well-known phenomenon caused by hierarchical, equal
and opposite data cancelling each other out. In schedules, this has the effect of
summary activities not reflecting the true range of risk that is held within the child
activities. This can easily be overcome through the use of risk correlation but must be
considered when building the model.

Integrating Uncertainty and Risk Events
One of the most overlooked inputs in risk model development is that of risk event
integration. Risk loading estimate uncertainty and running a Monte Carlo simulation is
commonplace but, in reality the impact of such uncertainty diminishes into insignificance


when the larger impact of risk events are considered. In light of this, it is paramount that
risk events get properly integrated into the uncertainty risk model prior to running a risk
analysis. Linked to this, is that of the analysis of mitigation. Too often the impact of
mitigation is subjectively assessed without analytically determining the cost/benefit of
planned mitigation as part of the integrated risk model.

In short, the risk model should contain the following inputs:

• Framework for risk model build
Schedule
• Sound logic driving the risk model

Estimate • Ranges representing scope uncertainty
Uncertainty
• Risk events not included in ranges

• Probability, cost, schedule impact
Risk Events
• Risks mapped to activities/WBS

Mitigation • Risk responses to risk events
Plans
• Consideration of cost/time overhead

These four inputs form the basis of the risk model that can be analyzed using a Monte
Carlo simulation.

Often, mystery surrounds the mechanics of a Monte Carlo simulation and yet in reality,
it is extremely straightforward. The simulation is nothing more than a normal schedule
time analysis (CPM technique) with each iteration taking into account the risk and
uncertainty variables that the team has identified. Run enough of these simulations and
a well-defined pattern will emerge in the form of risk results.

Step 4: Interpretation of Risk Results
The Danger of Reporting Confidence Levels
One of the biggest misperceptions about risk reporting is to focus on the confidence
level derived from a risk histogram. Why is this an issue? It can be a highly misleading
metric. The confidence level metric describes the chance or probability of achieving the
project completion date. After many years of risk analyzing projects of known low risk, I
began to question why so many of these turned up very low ‘confidence levels’. Further
investigation into this has revealed a phenomena known as merge bias. In layman’s

terms, merge bias occurs due to multiple parallel paths ultimately having to converge
on an activity or project completion milestone. Even with minor risk and uncertainty, the
chance of each of these predecessor paths completing on time and not having an
impact on the node point is very low. In fact, it is a compound probability effect.
Extrapolate this throughout the project in its entirely and you soon realize that the
confidence level metric needs to be used in caution.

Range, Range, Range
The range of uncertainty gives context to the confidence level. In addition to reporting
the confidence level, report on the range of uncertainty and even better, report this
range as a percentage of remaining duration. Reporting to the board that you have a
15% range of uncertainty on your project is more useful than reporting that you only
have a 3% of hitting the completion date. A small confidence level doesn’t necessarily
indicate a high-risk project…

How to Report Contingency
Too often contingency values are thrown around in absence of a confidence level. A
contingency of 30 days is meaningless unless we can understand the confidence level
at which this is being reported. The exact same project may have 90 days required
contingency if the required confidence level is 90% instead of say 50%. The ‘take
home’ here is to always report and request contingency amounts within the context of a
specified contingency level.

Conclusion
Like most systems, a risk analysis is only as good as the basis upon which the model is
built. Throw erroneous risk ranges at a schedule and the results will be equally
erroneous. Instead, by pursuing a well structured, third party facilitated risk workshop,
you ensure accurate inputs and model development.

Equally important is the intelligent interpretation of results. Risk exposure can be a
difficult topic to report and so it is key that the likes of risk range and required
contingencies are reported in a meaningful way within context of the given project.

Checklist for a Successful Risk Workshop
Listed on the next page is a checklist of guidelines recommended when considering
conducting a risk workshop. It is these factors that have enabled Acumen to have
continued world-class success with risk assessment workshops.


Risk Assessment Checklist

þ Tune the team into why you are having a risk workshop
þ Conduct pre-workshop discovery sessions
þ Employ a 3rd party workshop facilitator for truly independent assessment
þ Perform a well structured workshop – know the project schedule prior
þ Adopt risk factoring during the workshop – avoid statistics at all costs
þ Don’t overlook risk registers and risk events – they carry large impact
þ Brainstorm mitigation plans during the workshop
þ Keep the workshop group to a manageable size
þ Leave team hierarchies outside of the workshop – everyone has their say
þ Understand model limitations such as merge bias and central limit
þ Report intelligently: be wary of confidence level reporting in isolation
þ Hold periodic workshops in alignment with the likes of project/gate reviews

Additional Information
If you would like further information about Acumen’s risk assessment workshops,
please visit our website at http://projectacumen.com or contact
info@projectacumen.com.


The Art and Science Behind Successful Risk Workshops

More Related Content

What's hot (20)

Similar to The Art and Science Behind Successful Risk Workshops (20)

More from Acumen (20)

Recently uploaded (20)

The Art and Science Behind Successful Risk Workshops