Advanced Performance Measurement Workshop Develop Measures That Drive Performance By David Wilsey

Advanced Performance
Advanced Performance
Measures Workshop:
Measures Workshop: Develop
Develop MeasuresDrive
Measures That That
Drive Performance
Performance
David Wilsey, BSMP, SMP
Vice President of Education & Technology
Balanced Scorecard Institute
March 2012

1
©1997-2012 Balanced Scorecard Institute, a Strategy Management Group company. All Rights Reserved. Do not copy without permission.

Purpose and Agenda
Presentation Purpose: Understand how to develop
meaningful performance measures that drive
breakthrough strategic performance.

Agenda:
• Driving Breakthrough Strategic Performance and
Strategic Alignment with performance information
• A Practical Step-by-Step Approach to developing
meaningful performance measures
– Exercise: Identifying Intended Results
– Exercise: Developing Candidate Measures

• Performance Analysis Steps
• Problem-Solving Q&A (time permitting)

2

Why Do We Need Effective
Strategic Performance
Management & Measurement
Now More Than Ever?

3

We Live in Uncertain Economic Times

Source: Kauffman Economic Outlook, A Quarterly Survey of Leading Economics
Bloggers, Tim Kane, First Quarter, 2012.

4

We Live in a World of Routine
Government Cutbacks

• U.S. Federal debt plus
unfunded liabilities is now
over $100 Trillion
• Projected 2012 State
budget shortfalls is over
$153 Billion
• 2012 Municipal Budget In this environment,
shortfalls will exceed $53 programs and services
that cannot be defended
Billion will be cut!

5

We Live in a World of
Information Overload

6

We Work in Siloes

“This is not in my Job Description.
I draw lines; I don’t remove trees”

Source: John Phatshwe, BSMP, Botswana National Productivity Centre.
7

We Often Measure the
Wrong Things

Sabermetrics: the use of statistical analysis to
analyze baseball records and make determinations
about future player performance based on past
performance and correlations.

Teams which appear to value the concepts of
sabermetrics are often said to be playing
"MoneyBall".

8

“Sabermetrics” Transformed
Baseball Scouting Into a Science

Traditional Scout Evaluations
“Sabermetrics” Measures
Based on “Tool Set”

Speed On-Base Percentage

Arm Strength OPS = On-Base + Slugging

Runs Created = (Hits +
Hitting for Average
Walks) * (Total Bases)

WHIP = Walks + Hits per
Hitting for Power
inning pitched

9

“MoneyBall” is a Case Study on How
to Transform an Organization With
Performance Information

• There is value in
understanding the “drivers”
of overall (financial and
customer) success
• Make “fixing gaps” in driver
performance a priority
• Focus energy (and
resources) on the things
that matter most
• Measure what matters

To accomplish this, you have to think
differently about your organization!
10

Why Integrate Strategic Planning &
Performance Management?
A Framework Brings Discipline

• Get everyone using the
same language
• 'Connect-the-dots'
between activities and
strategy & vision
• Systematically align vision,
strategy, program, budget
& people
• Align resources with
strategic objectives

 11

Why Integrate Strategic Planning &
Performance Management?
Priority & Focus Improves Efficiency & Effectiveness

• Measure only what matters
• Focus on the right initiatives
and projects and choose the
right products and services
• Process improvements
guided by strategy
• Employees focus on results

12

Align the Organization to Mission,
Vision & Strategy

• Strategy Budget
• Operational Budget
• Capital Budget
Vision
Mission
• Customer Values &
Needs Strategy Budget &
• Political Priorities Actions
• Organizational Values
• Programs, Products,
• Stakeholder
Considerations • Strategic Goals / Services
Themes • Projects, Activities &
• Policies & Governance Tasks
• Strategic Results
Strategic • Rewards, Recognition
& Incentives
Objectives
• Schedule, Scope,
Resources, Risk

• Strategic Objectives/Map
• Measures & Targets
• Strategic Initiatives

 13

Why Integrate Strategic Planning
& Performance Management?
Strategic Alignment

Mission: Air Botswana is
committed to being the airline of
first choice through the delivery
of high quality service
14

Building & Implementing a Balanced
Scorecard: Nine Steps To Success™

15

Topics From the Nine Step Framework
to Be Addressed Today

• Understand Strategic Context:
– Steps 1-4, culminating in Strategic Objectives
• Develop Performance Measures:
– Step 5.1: Describe the Intended Results
– Step 5.2: Describe the
Measures
– Step 5.3: Describe the
Desired Level of
Performance
• Analyze Performance
Information
– Step 7: Performance
Analysis

16

The Goal is to Develop Meaningful
Performance Measures

• Meaningful means strategic
– Focused on desired results
– Derived from the Objectives from
a Strategy Map or Strategic Plan
• Meaningful means
relevant to the user
– It tells the user what he or she
needs to know to make better
decisions
– Accepted as valid (measures what
it intends)

 17

Example: Airline Balanced Scorecard
Mission: Dedication to the highest quality of Customer Service delivered with a sense of
warmth, friendliness, individual pride, and Company Spirit.
Vision: Continue building on our unique position -- the only short haul, low-fare, high-
frequency, point-to-point carrier in America.
Simplified Strategy Map Objectives Measures Targets Initiatives
Financial • Increase Profits • Market Value • 25% per year • Optimize routes
Increase • Lower Costs • Plane Lease Cost • 20% per year • Standardize
Profits • Increase Revenue • Seat Revenue • 5% per year planes

Increase
Lower Costs
Revenue

Customer • More Customers • # of Customers • 5% change • Quality
More • Increase On-time • FAA On time • First in the Management
Customers Flights Arrival Rating industry • Customer
Increase On- • Lower Prices • Customer • 98% Loyalty
Time Flights Ranking Satisfaction Program

Lower Prices

Internal Improve • Improve • On Ground Time • <25 Minutes • Cycle time
Process Turnaround Turnaround • On-Time • 93% Optimization
Time Departure Program
Time
Learning & Align • Align Ground • % Ground Crew • 70% (100% by • Stock
Growth Ground Crew Stockholders year 6) Ownership Plan
• % Ground Crew • 70% • Ground Crew
Crew
Trained Training
Adapted from work by Harvard University / Balanced Scorecard Collaborative
 18

Identify Strategic Objectives Before
Developing Performance Measures
Strategic Result
(Outcomes)
Strategic Objective:
Key strategy component;
continuous improvement
activity that must be Financial
performed; the “building
blocks” that make up a Customer
strategy.
Internal
Processes
Strategic
Strategic Objectives are Objectives
the “DNA” that break Organization
Capacity
high level strategy into
smaller pieces

19

Strategic Objectives are “Mid-Altitude”

Strategic
Altitude Vision & Mission type language: Problem: Too High (Vague)
30,000 ft. •“World – class” Risk: not actionable; will not
•“Provider of choice” give actionable focus and
•“High-performance” direction

Objective language:
25,000 ft.
•“Improve Customer Service”
Strategic •“Improve Knowledge & Skills”
Objective •“Increase Revenue”
•“Improve Service Delivery”
15,000 ft.

Project, Program or Service Language: Problem: Too Low (project
“Train staff” specific)
Ground “Build a new system” Risk: too narrowly defined;
Level “Deliver services” strictly operational

 20

Strategic Objectives Describe the Intended
Results of Continuous Improvement
Good Examples:
• Describe the result that we
intend to accomplish Increase
Economic Reduce Waste
• Should be “verb-object” Abundance

form, normally using a verb Improve Improve Access
that implies continuous Product Quality to Services
improvement:
– Improve Improve Brand Improve
Customer
Awareness Satisfaction
– Increase
– Reduce
Not-So-Good Examples:
– Decrease
Better: Improve
– Strengthen Train All
Employees Knowledge & Skills

– Possible: Optimize (can be Better: Improve
confusing) Write a understanding of our
Strategic Plan strategic direction

 21

Performance Measurement
Development Model
Step 5.1: Step 5.2: Step 5.3:
Describe the Describe the Describe the Desired Performance Level
Results Measure(s) and Interpretation
Choose & define
the appropriate
Yes “direct” measure(s)

Describe the Can the Should
Intended Intended Result measures be Develop targets
No
Result of the be measured combined & thresholds for
Objective directly? into a single each measure
“index”?

No
Yes
Choose & Develop a
Define measureable
define the “composite
components that
appropriate index”
thoroughly describe
“indirect”
the Intended Result
measures

To Step 7:
Documentation: Intended Result, Measurement Definition & Desired Performance Levels Performance
Analysis


22

Step 5.1: Describe the Intended
Result of Each Strategic Objective

• For each Objective, describe the objective's intended
result explicitly, in a few sentences
– Use action verbs
– Avoid words that create ambiguity
– Identify changes
in state that
accompany the
result

Establish agreement
across your team on the
desired result!

 23

Measurement Development: First
“De-Weasel” the Intended Results
• Strategy is hard to measure because it is often written using “weasel”
words, or vague ideals that are hard to interpret, for example:
• We employ “best practices”
• Performance is “Optimized” or “Maximized” or “World Class”
• Employees are more “Efficient,” “Effective” or “Productive”
• “Intended Outcome/Result” in the
Objective Commentary
should avoid this practice
– Use plain language: Not “Improve
Thought Leadership,” but “We
create engaging content,” or
“More people read (or rave about)
our stuff”
– Use sensory language to
visualize the desired end result
• If you can see, feel or hear it,
you can count it!

Adapted from the Performance Measure Blueprint Workshop, by Performance Measure Specialist
Stacey Barr (www.staceybarr.com)
 24

Remove Ambiguity From Intended
Results - Examples
• Intended Result: improve thought
leadership
Improve Brand – Too ambiguous: it is not clear exactly what
Image thought leadership means
• Not ambiguous: People rave about the
articles we write

• Intended Result: we deliver world class
Improve Service service
Delivery
– Too ambiguous: it is not clear exactly what
world class means
• Not ambiguous: Our service is delivered
on time and without defect

 25

Exercise 1: Which of the Following
“Intended Results” Still Contain Weasel
Words? And How Might They Be Improved?
Organization: Leading Tablet/Technology Innovator
Objective: Improve Product Development
Candidate Intended Results:
1. Quality increases
2. Enjoyment of our products is optimized
3. We understand our customers
4. Communications improves,
both internally and externally
5. We have a sexy user interface
6. We are on the bleeding edge of
technology
7. We humanize technology

26

Exercise 1: Intended Results
Possible Solutions

Organization: Leading Tablet/Technology Innovator
Objective: Improve Product Development
1. Quality increases Products meet specifications x or y at z consistency
2. Enjoyment of our products is optimized Customers tell us they love our
products
3. We understand our customers Voice-of-the-customer research is
consistently used in product development
4. Communications improves, both internally and externally Employees are
aware of key initiatives; our marketing advertisements are a hit on free
social media outlets
5. We have a sexy user interface Customers tell us they prefer our interface;
celebrities want to be seen using our products
6. We are on the bleeding edge of technology Our technology scanning team
identifies all important advances relevant to our products
7. We humanize technology Our customers tell us our products are intuitive;
non-technical customers use our products
27

Step 5.2: Describe the Measures

• Analyze each identified Intended Result using this leading
question:
– Can the Intended Result of the Objective be measured
“directly”? (i.e. is there a clear, discrete, unambiguous way
to capture the entire Intended Result?)
• Yes?
– Choose and define
the most appropriate
“direct” measures
• No?
– Define measureable
components that
thoroughly describe
the intended results

 28

Direct Measurement of the Intended
Result is Ideal - Examples
• Intended Result: Services are delivered on time:
Improve Service
– We know when the service is due; we know
Delivery when it is delivered; thus, we can directly
measure whether it was delivered on time
• Intended Result: Interagency is better integrated
Improve into planning:
Organizational – We know the regular planning activities that
Alignment occur; we know whether Interagency
representatives are included in those activities;
thus, we can directly measure whether there is
better integration
• Intended Result: All the products that we
Improve produce meet specifications:
Product Quality
– If we can define the specifications with physical
measures, and measure the products based on
those measures, we can measure directly

 29

If the Intended Result is a Complex
Concept with Multiple Components,
Measure it “Indirectly”

• If necessary, define components that could
thoroughly describe the intended result
• Select and define the appropriate indirect
measures

Identify those components by looking
for related results or for contributing
factors

30

Indirect Measurement of the
Intended Result May Be Necessary

• If the Intended Result can’t be
directly measured, develop
indirect measures based on a
hypothesis around:
– Correlation with intended
result : if desired result
improves, the indirect
measure does also
– Contribution to intended A
result : if A & B occur, the
result will be C; or A might also C
be a driving factor in B
B
31

Use a Proven Methodology to
Generate Candidate Measures

Logic Model
Input Process Output Outcome

Cause-Effect Analysis or Ishikawa
(Fishbone) Diagram
Resources Methods
Process Flow Analysis
Environment Effect

Systems Policies

 32

Consider the Purpose for Measuring
Performance When Selecting Measures

Promote

Budget Motivate

Control

Evaluate Celebrate

Learn Improve

Source: Adapted from Robert D. Behn, Why Measure Performance?, Public Administration Review, Sept./Oct. 2003, Vol. 63, No. 5.

33

Public Sector Managers Have Eight
Purposes for Measuring Performance
Common Measurement
Purpose Question Answered Characteristics
Evaluate How well is the agency performing? Outcomes, combined with inputs
Control Are subordinates doing what is asked? Inputs that can be regulated; compliance
Budget What should we spend money on? Efficiency (outputs/inputs) or effectiveness
(outcomes/inputs)
Motivate How can I motivate staff to improve? Real-time outputs compared with
production targets
Promote How can I convince stakeholders that my Easily understood; interesting and
agency is improving? How can I demonstrate meaningful for the intended audience
transparency?
Celebrate What accomplishments can we celebrate? Periodic and significant milestone targets
Learn What is working and what is not working? Disaggregated data that reveals deviances
from the expected
Improve What should we do to improve performance? Those that connect the dots between
(the overriding high level purpose for all) operations and desired outcomes

Source: Adapted from Robert D. Behn, Why Measure Performance?, Public Administration Review, Sept./Oct. 2003, Vol. 63, No. 5.

34

Define and Document all Selected
Measure(s)
• Use a Data Definition Table to capture and document all the
essential information about the measure:

35

Step 5.3: Describe the Desired
Performance Level and Interpretation
• Determine if “composite” measures are appropriate
using this leading question:
– Do the individual indicators from Step 5.2 offer useful
indicators of different dimensions or components of the
Objective’s Intended Result?
• If so, a composite
index will help
capture various
important dimensions of
the Intended Result
and combine them
into a single measure
• Develop targets &
thresholds for each
measure

36

Targets & Thresholds Help With
Interpretation

• Target: the desired level of
Target
performance for the
Green / Yellow reporting period in question
Threshold
• Threshold: the upper and
Yellow / Red
lower limits of desired
Threshold performance around a target
value (e.g., the exact point
that we want an indicator to
display green to indicate
good performance, yellow to
indicate satisfactory
performance, or red to
indicate poor performance)

37

Exercise 2: Performance Measurement
Development & Selection Quiz

• Answer each of the following questions
regarding performance measurement
development and selection.

38

• Organization: Leading Tablet/Technology Innovator
• Objective: Improve Product Development
• Intended Result: Voice-of-the-customer (VOC) research
is consistently used in product development
• Q: What is the most appropriate measurement?
a) Revenue related to new products in $
b) % of customers that love our products
c) % of products where VOC research procedure was followed
d) % of employees with project management skills

Why? The other measures, while needed, probably focus on other objectives
or intended results. Measure C DIRECTLY measures the intended result.

 39

• Organization: Professional Sports Team
• Objective: Improve Player Skills & Abilities
• Intended Result: Our players are skilled in their
respective roles
• Q: What is the most appropriate measurement?
a) Sales of shirts with shirts with star player names printed on
back (based on a hypothesis around correlation)
b) Our players lead the league in key player statistics relative
to their roles (correlation)
c) We win a lot of games (correlation)
d) We spend more on sabermetrics software (contributing
factor)

Why? There are too many other factors for A to be a helpful measure. B focuses
specifically on the outcome identified in the Intended Result. C is probably
covered in another objective. D is an input that doesn’t monitor follow-through.

 40

Contribution Vs. Correlate
Lemonade Stand Example

Strategy Map / Objectives Measures Targets
Profit is a ____ of Profit?
Financial • Profit • Enough to buy a a) direct measure of
Increase bicycle by the end
Profits b) correlate of
of summer
c) contributor to
Customer • Repeat business • 80% of target
Improve market (parents)
Customer Sugar content is thought
Satisfaction to be a _____ of Quality?
a) direct measure?
Internal • Quality: Sugar • 100% sugar
b) correlate?
Business content content exceeds
Improve c) contributor?
Process Improve • Image: “Awww” that of soda
Lemonade
Brand Image Index: % of drivers • 80% smile
Quality
that smile Mom’s anger is thought to
Organizational • Skills: % Mom is • 0% anger rate be a _____ of Lemon
Capacity angry due to • < 6 years old Squeezing Skills?
Increase wasted lemons a) direct measure?
Improve Lemon-
Youthful Work • Culture: Average
Squeezing Skills b) correlate?
Culture age of lemonade c) contributor?
stand workers

 41

• Organization: Local Elementary School
• Objective: Improve School Technology Capabilities
• Desired Result: Children have access to the technology
they will need to succeed in the world
• Q: Which of these is the most appropriate measure?
a) # of computers donated to the school (based on a
hypothesis around contribution)
b) Network up time (contribution)
c) Budget spent on technology and staff training (contribution)
d) % of students with access to technology that is less
than two years old (direct measure)

Why? It directly measures the Desired Result. The others
are inputs that don’t monitor follow-through.

 42

• Organization: State Literacy Program
• Objective: Improve Program Outcomes
• Desired Result: Literacy improves in the state
• Q: I’ve determined that the measurement purpose is
promotion, as I need to communicate progress to the
program sponsors and to external stakeholders. What is
the most appropriate measurement?
a) # of teachers trained in literacy coaching
b) Change in state literacy rate since program was enacted
c) # of students that attend the program per week
d) % of budget focused on literacy programs
Why? It is outcome focused and easily understood,
interesting and meaningful for the intended audience
 43

Step Seven is Now Called
“Performance Analysis”

Performance
7 Analysis

44

There Are Many Steps to Turning Data
Into Information and Knowledge

Continuously Review

Take Action

Share / Create Dialog

Interpret / Present

Analyze
Turn Data Into
Information & Store
Knowledge
Collect
Source: Adapted from Performance Measurement Diagnosis PuMP Mind Map, by Stacey Barr and The
Performance Measurement Cycle, Transforming Performance Measurement, Dean Spitzer.

 45

Take a Step-by-Step Approach to
Performance Analysis/Analytics
1. Focus on the highest value opportunities first
– Look to Strategic Themes, Results and Strategy
Map linkages for a starting point
2. Start with questions and decisions to make,
NOT DATA
3. Drive data-driven insights deep into the
organization
– Enable dialog
– Share, visualize & simulate
4. Expand (not replace) your analysis capacities
– Focus on both talent & culture
5. Use analysis to improve ongoing strategic
planning

Adapted From: Analytics: The New Path to Value – How the Smartest Organizations Are Embedding Analytics to
Transform Insights Into Action, LaValle, Hopkins, Lesser, Shockley & Kruschwitz, MIT Sloan Management Review and
the IBM Institute for Business Value, Fall 2010.

46

Performance Data Can Analyzed for
Actionable Correlations

Sears “mined” over 70 performance indicators for an
actionable correlations. Twenty-two months after the
analysis began, they identified three that were highly
significant:

Employee Customer Revenue
Attitude Impression Growth
(Satisfaction) (Satisfaction)
5 unit 1.3 unit 0.5 %
increase increase increase

15 months

Source: Harvard Business Review

 47

Gwinnett County Schools Used Performance Data
to Improve Graduation Rates
In 2002, the Gwinnett Count (GA) public school system decided to use performance
analytics to improve the (falling) graduation rates for at-risk kids.

Key Questions:
• What aspect of performance most Graduation
accurately predicted graduation?

?
• What is the strongest predictor of Pass
passing Algebra I?
Algebra I

Success at eighth-
?
grade creative writing

In Fall 2010, Gwinnett County won the Broad Prize for student
achievement and improvement while narrowing achievement
gaps between income and ethnic groups

Source: Big Data, Analytics and the Path from Insights to Value, LaValle, Lesser, Shockley, Hopkins & Kruschwitz, MIT Sloan
Management Review, Winter 2011.
48

Other Examples of Performance
Analysis Driving Better Decision Making
• Hilton Hotels found that a 5% improvement in customer retention would
result in a 1.1% increase in revenue at a typical property*
• Best Buy discovered that a tenth-of-a point increase in employee
engagement lead to an increase in store operating income by $100,000*
• The Baltimore Department of Works used data management to reduce
absenteeism to reduce operating costs by $13.2m.**
• Victoria’s Secret found that raising its average conversion rate (entering
customer who buy something) by 1% led to more than $35m in sales and
more than $15m in profit.*
• Store 24, a convenience store, confidently
abandoned its “entertaining service
experience” strategy after finding a
negative relationship between strategy
implementation and profits.*

Sources: *The Rise of Analytical Performance Management, Thomas H. Davenport, Harvard Business Digital / SAS Institute.
**The CItiStat Model-How Data-Driven Government Can Increase Efficiency & Effectiveness, Perez & Rushing, the Center for
American Progress, April 2007.
49

Use Performance Analysis to Improve
Ongoing Strategic Planning
• Use performance information to
“manage to results” and inform
decision making
• Look for new leading performance
indicators – incorporate learning into
strategy and business operations
• Communicate strategy effectively,
internally and externally
• Periodically check reward and
incentive systems for alignment with
desired results
• Use Performance Information during
Evaluation (step 9) to revise Strategic
Themes, Strategic Results, Strategic
Objectives, Strategy Maps, etc.
• Don’t be afraid to “test and learn”

 50

Q&A: What Are the Issues You
Are Facing in Performance
Measurement Development &
Analysis?

Thank You!

51

Strategic Performance & Change Management Conference
November 13-15, 2012 in New Orleans, LA

View the Strategic Performance
& Change Management program
here: http://tinyurl.com/8wxywgc

If you would like to receive the free IQPC brochure
directly email taryn.soltysiak@iqpc.com

Advanced Performance Measurement Workshop Develop Measures That Drive Performance By David Wilsey

More Related Content

What's hot (18)

Viewers also liked (8)

Similar to Advanced Performance Measurement Workshop Develop Measures That Drive Performance By David Wilsey (20)

More from Steven Bonacorsi (20)

Advanced Performance Measurement Workshop Develop Measures That Drive Performance By David Wilsey