Impact of Agile Quantified - Late 2014 Edition

The
Impact
Agile
of
Quantified
@LMaccherone | Larry@Maccherone.com @Tasktop |

Lean-Agile city.
This place runs on
folklore, intuition, and anecdotes.
If you want to know the truth
about this town, stick with me. I’ll
give you a tour you’ll never forget.
But if you don’t want your beliefs
challenged with facts, you’d better
beat it, kid. I don’t want to upset
you.

My sidekick down there? That’s
Larry Maccherone. He’s worked in
this town his entire professional life.

Credits
Many thanks to my friends at Rally Software
and their customers whose non-attributable
data is the source for much of this research.
Much of this material was previously
published when I was Director of Analytics
and Research at Rally Software, and much of
that is derived from my PhD work at
Carnegie Mellon, while other aspects come
from collaboration with the Software
Engineering Institute.

I’m going to give you the tools to find the real-world
numbers that can help you make the
economic case to get the resources you need
and get your people to commit to change.
Really.
-

The Seven Deadly Sins of Agile Measurement
1 Manipulating
Others
2 Unbalanced
Metrics
3 Quantitative
Idolatry
4 Overpriced
Metrics
5 Lazy
Metrics
6 Bad
Analysis
7 Linear
Forecasting

Sin #1
Manipulating
Others
Using metrics as a
lever to drive
someone else’s
behavior

Heavenly Virtue #1
Self
Improvement
Using metrics to
reflect on your own
performance

Linear
Sin #7
Forecasting
Forecasting without
discussing
probability and risk

Heavenly Virtue #7
Probability
Tools
Using the proper
tools to predict the
likelihood of results
(Not likely)

Monte
Carlo
Simulation

CAUTION:
Correlation
does not
necessarily mean
causation

CAUTION:
There are no
best practices
Only good
practices in
context

The investigation continues with ...
Iteration length

Crowd wisdom or shared delusion?
Iteration length Teams
using
1 week 6.2%
2 weeks 59.1%
3 weeks 23.4%
4 weeks 9.8%
5+ weeks 1.5%

SDPI current dimensions
Productivity
(Throughput)
Predictability
(Stability of
Throughput)
Responsiveness
(Time in Process)
Quality
(Defect Density)

Future SDPI dimensions
Customer/
Stakeholder
Satisfaction
(Late 2014)
Build-the- Right-
Thing metric
(2015)
Employee
Engagement/
Satisfaction
(Late 2014)
Code Quality from
Static Analysis
(2015)

Raw metrics → Percentiles = Index

Facts Discovered:
● Teams using two-week
iterations have the best
balanced performance
● Longer iterations correlate
with higher Quality
● Shorter iterations correlate
with higher Productivity and
Responsiveness
● However, some teams are
acting like “tough guys” by
pretending to operate at one-week
iterations when they
can’t back it up

Ratio of testers to
developers

Facts Discovered:
● More testers lead to better
Quality
● But they also generally lead
to worse Productivity and
Responsiveness
● Interestingly, teams that self-identify
as having no testers
have:
o The best Productivity
o Almost as good Quality
o But much wider variation
in Quality

Retrospectives

Motive

Evidence Found:
● Motive has a small but statistically
significant impact on performance
● Extrinsic motivation does not have
a negative impact on performance
● Executive support is critical for
success with Agile
● Teamwork is not the dominant
factor; talent, skills, and
experience are
● Those motivated by quality
perform best

Co-location

Evidence Found:
● Teams distributed within the
same time zone have up to
25% better productivity.
● Is distraction a problem?

One year earlier ...
rallydev.com/agilemetrics

Teams with low WiP have up to:
● 4x better Quality
● 2x faster Time to market
● But 34% worse productivity
Stable teams result in up to:
● 60% better Productivity
● 40% better Predictability
Dedicated teams: Teams made up of
people who only work on that one
team have double the Productivity
Smaller teams have better Productivity
Larger teams have better Quality

Future research

 Tasktop Data
– Provides a single aggregated stream of Data from connected tools:
• ALM, Build, SCM, Support, Sales, etc.
– Operates in real time.
– Maintains complete history.
– Normalizes data models. Resolves users and projects.
– Supports analytics and reporting tools.
– Does not directly provide visualization or analysis.
 Get insights like the ones in this talk for your teams

A fact without a theory
is like a ship without a sail,
is like a boat without a rudder,
is like a kite without a tail.
A fact without a figure
is a tragic final act.
But one thing worse
in this universe
is a theory without a fact.
~ George Schultz
©2014 Larry Maccherone

Tasktop booth
• Your questions
• Extracting insights from
your data
Panel discussion
• 4:30pm Grand Ballroom
• Dave West – Tasktop
Chief Product Officer
• Ken Pugh
• Mike Cottmeyer
• Steve Davis
• Jeff Estill

Additional
slides
@LMaccherone | Larry@Maccherone.com #RallyON14 @Tasktop |

SDPI dimensions

Productivity = Throughput
Throughput is simply the count of User Stories completed in a given time period.
Productivity (by default) is the percentile scoring of the raw Throughput metric for User
Stories normalized by team size.

Predictability = Stability of Throughput
Predictability measures how consistent you are at producing the same amount of work
each month as measured by the Coefficient of Variation (CoV) of Throughput.
Predictability (by default) is the percentile scoring of the raw CoV of Throughput.

Responsiveness = Time in Process
TiP shows how long it takes to get one work item through your system. It's the work
days that a User Story spends in development and testing. Similar to lead time or cycle
time.
Responsiveness (by default) is the percentile scoring of the raw Time In Process (TiP)
metric for User Stories.

Quality = Defect Density
Defect Density is a representation of the number of defects found in your code. It's the
count of defects found in a given time period, normalized by team size.
Quality (by default) is the percentile scoring of the raw defect density metrics for both
defects found in test as well as those found in production.

Team time together

Controlling WiP

Facts Discovered:
Teams that most aggressively
control WiP:
● Have ½ the Time in Process
(TiP)
● Have ¼ as many defects
● But have 34% lower
productivity

Recommendations:
● If your WiP is high, reduce it
● If your WiP is already low,
consider your economic
drivers
○ If Productivity drives your
bottom line, don’t push
WiP too low
○ If time to market or
quality drives your
bottom line, push WiP as
low as it will go

Estimating process

Estimating process
Process Type Teams Using
No Estimates 3%
Full Scrum 79%
Lightweight Scrum 10%
Hourly-Oriented 8%

Facts Discovered:
● Teams doing Full Scrum
have 250% better Quality
than teams doing no
estimating
● Lightweight Scrum performs
better overall, with better
Productivity, Predictability,
and Responsiveness

Recommendations:
● Experienced teams may get
best results from Lightweight
Scrum
● If new to Agile or focused
strongly on Quality, choose
Full Scrum

Team stability &
Dedication to one team

Another Fact Discovered:
One out of four team members
changes every three months!

Facts Discovered:
Stable teams result in up to:
● 60% better Productivity
● 40% better Predictability
Another Fact Discovered:
One out of four team members
changes every three months!

Recommendations:
● Dedicate people to a single
team
● Keep teams intact and stable

Team size

Balance your team’s Performance
Agile recommends that the ideal team size is 7± 2. How ideal is that when we actually look at the data?

Facts Discovered:
Small teams (of 1-3) people have:
● 17% lower Quality
● But 17% more Productivity
than teams of the recommended
size.

Recommendations:
● Set up team size of 7±2
people for the most balanced
performance
● If you are doing well with
larger teams, there’s no
evidence that you need to
change

Geography

Israel-based teams
● Find more defects overall
● But find fewer in production
● Theory: May correlate with
high use of static analysis tools
India-based teams
● Find more defects overall
● Released and unreleased
● Theory: May correlate with
high use of static analysis tools
● Theory: Could be recording
bias

Facts Discovered:
● Differences are slight but
statistically significant.
● Australia has the best overall
performance.
● India the worst. However,
there could be a reporting
bias for defects.
● Israel seems to catch the
most defects before
production. Heavy use of
static analysis?

Geography: US and Europe

Impact of Agile Quantified - Late 2014 Edition

More Related Content

What's hot (20)

Viewers also liked (19)

Similar to Impact of Agile Quantified - Late 2014 Edition (20)

Recently uploaded (20)

Impact of Agile Quantified - Late 2014 Edition

Editor's Notes