Bringing Science to Software Development

Janelle Klein
openmastery.org @janellekz
Tour Speaker
Bringing Science to
Software Development
How to Measure the PAIN
in Software Development
Janelle Klein

Janelle Klein
After ﬁve years…
Data-Driven Software Mastery
leanpub.com/ideaflow

Janelle Klein
==
2.0
Visibility Changes Everything
Visibility
+
Lean Manufacturing

“The Art of the Scientist”
Lean Startup Workshop, Ash Maurya
Ash:
“The goal isn’t learning, the goal is traction.”

==
2.0
+
Traction
(Lean Startup)
We get a design template for how to build a learning organization.
Visibility Changes Everything

News Flash!
For the ﬁrst time in our industry,
we have all the tools necessary
to pull this off.

Five Talks
Stop Getting Crushed By Business Pressure
How to Build a Learning Organization
Top 5 Reasons Why Improvement Efforts Fail
Make a F.O.C.O.L. Point!

This Talk…
Using Science to Fuel Software Mastery
(team & organizational level)

Five Disciplines of a Learning Organization
Personal Mastery
Mental Models
Shared Vision
Team Learning
Systems Thinking
These disciplines are emergent with science.

, Developer & Consultant for 15+ years
Specialized in Statistical Process Control (SPC)
and Supply Chain Optimization from Lean Manufacturing (I <3 cool data)
Continuous Delivery infrastructure, design & automation strategies
Janelle Klein
Who Am I?
Janelle Klein
leanpub.com/ideaﬂow
Data-Driven
Software Mastery
I’m also a hobbyist Cognitive Scientist

CTO @
Founder @
Industry Peer Mentorship Network
My Job

Summary from Talk #1
Learn & Adapt

Great Team
Disciplined with Best Practices
Constantly Working on Improvements+
Project FAILURE
About 8 years ago…

Technical Debt Mistakes
I thought the main obstacle was
Technical Debt

?
Most of our mistakes were in the
most well-written parts of the code.
Mistakes

We made significantly more mistakes
in code that we didn’t write ourselves.
Lower
Familiarity
More
Mistakes=
There had to be more to the story...

Complex(
So*ware(
PAIN
This is what I knew...
What made development feel painful?

Unexpected
Behavior
Problem
Resolved
Measure Painful Interaction with the Code (Friction)
Troubleshooting
Progress
5 hours and 18 minutes of troubleshooting...
PAINFUL

The amount of PAIN was caused by…
Likeliness(of((
Unexpected(
Behavior(
Cost(to(Troubleshoot(and(Repair(
High(Frequency(
Low(Impact(
Low(Frequency(
Low(Impact(
Low(Frequency(
High(Impact(
PAIN(

What Causes Unexpected
Behavior (likeliness)?
What Makes Troubleshooting
Time-Consuming (impact)?
Semantic Mistakes
Stale Memory Mistakes
Association Mistakes
Bad Input Assumption
Tedious Change Mistakes
Copy-Edit Mistakes
Transposition Mistakes
Failed Refactor Mistakes
False Alarm
Non-Deterministic Behavior
Ambiguous Clues
Lots of Code Changes
Noisy Output
Cryptic Output
Long Execution Time
Environment Cleanup
Test Data Creation
Using Debugger
Most of the pain was caused by human factors.
What causes PAIN?

Ambiguous Clues
Noisy Output
Cryptic Output
Long Execution Time
Environment Cleanup
Test Data Creation
Using Debugger
What causes PAIN?
Semantic Mistakes
Association Mistakes
Copy-Edit Mistakes
False Alarm

PAIN occurs during the process of
understanding and extending the software
Complex(
So*ware(
PAIN
Not the Code.
Optimize “Idea Flow”

My team spent tons of time working on
improvements that didn’t make much difference.
We had tons of automation, but the
automation didn’t catch our bugs.

My team spent tons of time working on
improvements that didn’t make much difference.
We had well-modularized code,
but it was still extremely time-consuming to troubleshoot defects.

The hard part isn’t solving the problems
it’s identifying the right problems to solve.
“What are the speciﬁc problems
that are causing the team’s pain?”

measures the time spent on:
Idea Flow
x
Troubleshooting
x
Learning
x
Rework
Quality Risk Familiarity Risk Assumption Risk

Idea Flow Mapping Tools
(Open Source, Supported GA ~June 2016)
github.com/ideaflow/tools

“What caused the pain
in this case?”
Categorize the Problems with #HashTags
#ReportingEngine
#Hibernate
#MergeHell

1. Test Data Generation
2. Merging Problems
3. Repairing False Alarms
1000 hours/month
Add up the Pain by Category

0%
100%
Release 1 Release 2 Release 3
Percentage Capacity spent on Troubleshooting (red) and Learning (blue)
Chaos Reigns
Unpredictable work stops
fitting in the timebox
Troubleshooting
Progress
Learning
Analyzing Long-Term Trends

Start%
Over%
Unmaintainable%
So0ware%
Realization:
Repeating the Same Mistakes

From the Outside:
This is the pattern of a
broken feedback loop.

We Work Under Constant Urgency

Time%
Pressure%
Compromise%
Safety%
for%
Speed%
Increase%
Number%&%
Severity%of%
Hazards%
%
More%Pain%
and%Higher%
Task%Eﬀort%
Constant'
Urgency'
Cycle of Chaos
High-Risk Decision Habits

0%
100%
Release 1 Release 2 Release 3
Troubleshooting
Progress
Learning
Percentage Capacity spent on Troubleshooting (red) and Learning (blue)
(extrapolated from samples)
Gambling Habit —> Gradual Loss of CONTROL

Fewer%
Problems%to%
Fix%
Stop%%
and%
Think%
Mi8gate%
the%
Risk%
Increased%
Produc8vity%
and%
Innova8on%
Safety'
Cycle of Safety
Low-Risk Decision Habits

Optimize for Safety
We don’t improve productivity by trying to go faster, we
improve productivity by improving control.
The Safety Principle

Product(
Requirements( So1ware(Task(
Possible(Decision(Paths(
So1ware(
Software is a Reﬂection of our Decisions

Product(
High%Risk%
Decision%Habits%
So1ware(
(and(all(our(problems)(
(
RESET

Product(
High%Risk%
Decision%Habits%
So1ware(
(and(all(our(problems)(
(
Technical Debt is just a Symptom of the Problem

Learn & Adapt
Just because we repair the car…
doesn’t mean we know how to drive.

Quality Target
Lower Control Limit
Upper Control Limit
X"
This is “Out of Control”
Lower Variability = Better Control
Steering Wheel = Process Control

“Pain Control” in Software Development
Optimal Friction
Upper Control Limit
X"
“Out of Control”
20min
0m
Average Pain per Incident
This is Better
Target
Control Limit
Categorize the Pain Type

Input:
Decision Constraints
Target: Optimize the Rate of Idea Flow
short-term looplong-term
loop
1.
Visibility
2.
Clarity
3.
Awareness
F ocus!
Output: “Friction” in Idea Flow
Improve Quality of Decisions
Idea Flow Learning Framework

Input:
loop
1.
Visibility
2.
Clarity
3.
Awareness
F ocus!
Target - The direction of “better”

Input:
loop
1.
Visibility
2.
Clarity
3.
Awareness
F ocus!
Input - The constraints that limit our short-term choices…

Input:
loop
1.
Visibility
2.
Clarity
3.
Awareness
F ocus!
Output - The pain signal we’re trying to improve

Input:
loop
1.
Visibility
2.
Clarity
3.
Awareness
F ocus!
Focus on the biggest pain…
F ocus!

Input:
loop
1.
Visibility
2.
Clarity
3.
Awareness
F ocus!
1. Visibility - Identify the specific patterns
1.
Visibility

Input:
loop
1.
Visibility
2.
Clarity
3.
Awareness
F ocus!
2. Clarity - Understand cause and effect
2.
Clarity

Input:
loop
1.
Visibility
2.
Clarity
3.
Awareness
F ocus!
3.
Awareness
3. Awareness - Stop and think to adjust habits

Input:
loop
1.
Visibility
2.
Clarity
3.
Awareness
F ocus!
4. Run Experiments to Learn What Works

Optimal Friction
Upper Control Limit
X"
20min
0m
This is Mastery
Target
Control Limit
This is the Process of Mastery

“The Art of the Scientist” Workshop, Ash Maurya
Ash:
“What’s the most important thing that scientists do?”
“Run Experiments!”
“Analyze Data!”
Getting Involved with Lean Startup

NO!
?
?
?
?
?
?
?

Ash:
“The most important things scientists do is build models.”

Ceiling of Achievement, Ash Maurya
Lots of learning, but not learning the “right things.”

Floor of Improvement?
Lots of improving, but not improving the “right things.”

Test the Model
(Experiments)
Refine the Model
(Patterns)
False&
Predic,on&
Scien,ﬁc&
Model&
Modeling
(Inductive)
Experimentation
(Deductive)
Empirical Data
The Scientiﬁc Method
We can’t ignore
this part.

Test the Model
(Experiments)
Refine the Model
(Patterns)
False&
Predic,on&
Scien,ﬁc&
Model&
Modeling
(Inductive)
Experimentation
(Deductive)
Empirical Data
Science is a Feedback Loop
that Fuels Discovery
Creating an explicit model makes
our beliefs testable and our discoveries additive.
We can’t ignore
this part.

Optimize the Rate of Flow in a Supply Chain
Focus on
Limiting
Constraint
(Goldratt’s Theory of Constraints, “The Goal”)

“The Customer Factory”, Ash Maurya
Optimize the Rate of Happy Customer Flow
Constraint

Optimal Friction
Upper Control Limit
X"
20min
0m
Average Friction per Customer
This is Traction
This is the Process of Lean Startup
High
Friction
Min
Friction
“No”
Threshold
Customer
Happiness
Categorize the “No” Type

“The Idea Flow Factory”
(supply chain model)
Optimize the Rate of Idea Flow
Across the Software Supply Chain

100 hours
50 hours
Troubleshooting
Learning
Rework
Focus on the Biggest Pain Type
Focused Effort = Visible Improvement

Optimal Friction
Upper Control Limit
X"
20min
0m
This is Traction
Target Rate
Control Limit
This is the Process of Idea Flow Mastery

We can apply the entire suite of cost & risk reduction tools
from Lean Manufacturing!!
Supply Chain Optimization
Statistical Process Control (SPC)
Optimal Friction
Upper Control Limit
X"
20min
0m
Target
Control
Limit
Throughput Accounting

Test the Model
(Experiments)
Refine the Model
(Patterns)
False&
Predic,on&
Scien,ﬁc&
Model&
Clarity
(Why)
Awareness
(How)
Visibility
(What)
Use Science to Fuel “Explicit Mastery”
Model “Decision Principles”

What’s a Decision Principle?
1. How do I evaluate my situation?
2. What should I optimize for?
Answers Two Questions

The Haystack Principle
Lots of unvalidated changes
Easier to ﬁnd the needle.
Decision Principle: The Haystack Effect
Optimize for small manageable haystacks.

Input:
loop
1.
Visibility
2.
Clarity
3.
Awareness
F ocus!
Systematically codify Decision Principles
from lessons learned

Three Stages of Mastery
Visibility
Clarity
Awareness

The Visibility Principle
Our mind ﬁlters what we see and experience
in terms of the “shapes” we know.

George Lakoff
Metaphors We Live By
Jeff Hawkins
On Intelligence
+
How do we “see” (and not see)?

Observe Pattern
“Do I know this pattern?”
Metaphors are patterns of conceptual “Shapes”

Three Different Types of Metaphors
Object Patterns
(Thing)
Context/Relationship Patterns
(Space)
Process Patterns
(Time)
We “see” the world through a fabric of metaphor.

Technical Debt
Debt
is a “Thing”

“This looks like
a tech debt thing.”
1. Categorize patterns by concept type

“This looks like
a painful tech debt thing.”
2. Categorize variations by intensity.
Pain level = 8

= Little Debt
= Medium Debt
= Big Debt
Our Beliefs about Developer Pain

PAIN occurs during the process of
understanding and extending the software
Complex(
So*ware(
PAIN
Pain occurs over TIME.

What do you think happens when we
make this transition?
Process Patterns
(Time)
Object Patterns
(Thing)

Take a Closer Look at the Patterns
Familiarity Mistakes
Semantic Mistakes
Copy-Edit Mistakes
False Alarm
Ambiguous Clues
Noisy Output
Cryptic Output
Long Execution Time
Environment Cleanup
Test Data Creation
Using Debugger

“What caused the pain
in this case?”
We Start to Experience the World Differently
#ReportingEngine
#Hibernate
#MergeHell

2.#Record#an#Idea#Flow#Map#
So#ware(Task(
Strategy((
Design(
Experience(
Review(
Iden:fy(the(Pa>erns(
1.#Clarify#the#Strategy#
So#ware(Task(
3.#Reﬂect#on#Decisions#
Focus(
Mee:ng(
4.#Share#Discoveries#
Visibility Framework

The Clarity Principle
Our understanding deepens when we explain
the similarities and differences of experiences
and the relationships of cause and effect

Journey
Beginning End
Think about the Developer’s Experience as a
Problem-Solving Journey

Journey
Constraints
What were the biggest challenges on the journey?

Friction
Journey

Journey Strategy

Journey Uncertainty
???

Situation Strategy Consequence
Developer Journeys
Breakdown the Patterns in Developer’s Experience

Archetype Journeys Archetype Journeys
Journeys that Repeat

Decision Principles
What are we trying to optimize for?

Pain Type
From the “Ten Pains” of Software Development

Circle Leader
Circle Member
Focus: What’s the problem to solve?
Observe: Ask questions about the facts
Conclude: What’s causing the pain?
Optimize: How do we reduce pain?
Learn: What questions does this raise?
Observation
Questions
Make a F.O.C.O.L Point!
Summarize lessons learned in a
#HashTagged Blog.
Deep Reﬂection in Open Mastery Circles

Clarity Framework (Why)
Repeat&
Weekly&
Focus&Mee1ng&
1.#Explain#Decisions# 3.#Search#for#Archetypes#
4.#Model#Decision#Principles#
Zoom In Zoom Out
2.#Model#the#Journey#
Focus#

Reset Password Login
Username:
Password:
To continue to our site, please login:
Task: Implement THIS

Bringing Science to Software Development

George’s Experience Review
1 hour 10 min to find this bug.

Me: "So first, George, can you describe what you were trying to do before you
encountered the conflict?”
George: "This was back when I was getting the token to work in the email. I
wrote the code to generate the token, edited the email template to include the
token in the email link, but the token wasn't showing up in the actual email."
Me: "So what did you do next?"
George: At first, I thought there was something wrong with the token variable
code, but the problem turned out to be a misspelled variable name in the
email template. Doh!"
Focus: What Major Factors Caused the Pain?
Observe: Ask Questions About the Facts.

Conclusions: What Caused the Pain?
Why did it take 1 hour 10 min to find this bug?

Painful Consequences: Consequence
Conclusions: What Caused the Pain?
Ran experiments by sending emails over and over again
(23 emails with a missing tokenId)
Redeployed application when uncertain about the cause
Ambiguous Clue
Manual Execution (slow)

Optimize: How could we have avoided the pain?
What’s an alternative strategy
that could have mitigated these risks?
George: My main strategy was to do a small #VerticalSlice by getting the
“email send” part working, before adding the token logic for the security part. I
understand how the #ManualExecution and an #AmbiguousClue caused the
friction, but I'm not really sure what I would do differently if I could do it again.
Me: "George, can you describe your original strategy to mitigate risk?”

What’s the strategy pattern we’re
trying to implement?
Alternative Strategy:
APIs for each email workflow
Wire up email last
Strategy

Alternative Strategy:
APIs for each email workflow
Wire up email last
Situation
What’s the situation we need to recognize
in order to decide to use the strategy?

Situation Strategy Consequence
Developer Journeys
#HiddenOutputs
Create
#ObservableInterfaces
Avoid #AmbiguousClues &
#ManualExecution

Learn: What questions does this raise?
What questions
should we ask ourselves in the future?
In what situations
should we ask the question?

Journey
Field
Guide
What questions should we add to the ﬁeld guide?
Situational
Questions

Look for Patterns Across Experiences
Journeys that Repeat

Describe the Generalized Causes of Pain

Experiment
Time
Our Perception of Time is WAY OFF
Setup Experiment Analyze Results &
Decide Next Experiment
Execute
waiting - time goes slow
doing - time zooms by

“Experiment Pain”
16:10
Optimizing for Human Cycles

The Code Sandwich Principle
The thickness of the
sandwich increases
troubleshooting
difficulty
Behavior Complexity
Observability
Ease of Manipulation
Optimize for experimentation.

Optimize for additive clues.
The Narrowing Cloud Principle

The bigger the cloud of possibilities,
the more we need a strategy.

The Awareness Principle
We can only avoid a mistake
if we predict the consequences
of the decision in the moment.

18:120:00
George’s Painful Experience
I know I made a big haystack,
I’ll do better next time.
This is hindsight bias, it’s not that easy!
The Experience Review

The Challenge
We tend to make auto-pilot decisions and do what’s familiar
How do we break decision habits?

Imagine your brain is a
decision-making engine
written in code.
Breakpoint
Stop and Think!
Changing Decision Habits

Breakpoint
Stop and Think!
Changing Decision Habits
We have to predict the big haystack
during the decision.

The Checklist Manifesto
Atul Gawande
Scaling Lean (Lean Startup)
Ash Maurya
+
Change Decision Habits with Strategy Experiments

14:230:00
I want to avoid this…
Thinking Checklist
Is my current approach likely to cause a big haystack?
Situation: start of subtask
Let’s Make a Checklist!
“What question could I ask my future self to
recognize similar risks in the future?”
“In what situation would I ask the question?”

0:00
Stop and Think:
Is my current approach likely to
cause a big haystack?
Predict: Small haystack
Strategy Experiments

18:120:00
Stop and Think:
Predict: Small haystack
False Prediction

18:120:00
Stop and Think:
False Prediction
High-Risk Situations
1. Unraveling sweater
2. Integration-heavy change
3. High state variation
4. Minimum scope is big
Q: Is my current approach likely to
Start of Subtask

Strategy Types (“do”)
1. DependencyAnalysis
2. IncrementalIntegrationTest
3. DataDrivenTest
4. IsolateHardToTestCode
High-Risk Situation Types (“see”)
1. UnravelingSweater
2. HeavyIntegrationLogic
3. HighStateVariation
4. CoupledExternalDependencies
Haystack Decisions
Codify What Works
DoSee

So#ware(Task(
Software TaskStrategy((
Design(
Thinking(checklists(in((
key(situa6onal(contexts(
!(
!(
!(
!(
!(
!(
!(
!(
!(
Experience(
Review(
!(
!(
3.#Reﬁne#the#
Checklists#
!(
!(
!(
Failed(
Experiments(
Successful(
Experiments(
2.#Run#Checklist#Experiments#
1.#Design#Thinking#Checklists#
+(
Awareness Framework (How)

Three Stages of Mastery
Visibility
Clarity
Awareness
See
Explain
Predict

Janelle Klein
These are the Blueprints!
++

Collaboration to Perform a Function (Team)
Social Network of Brains
Component

Input:
loop
1.
Visibility
2.
Clarity
3.
Awareness
F ocus!
How do my decisions affect your experience?

Management
Dev Team
Fire x10
What if our organization was a robot?
If the feedback loop is broken, we burn.
Pain Sensor

Bigger Social Network of Brains
Collaboration to Perform a Bigger Function (Organization)
Component
Component
Hub
Hub
Hub
The Challenge: Knowledge & Decision-Making are Distributed

Learn & Adapt
Role
Decision(
Type(
Required(
Knowledge(
Visibility and Decisions
Coupled
Visibility and Decisions
De-coupled
Role A
Required(
Knowledge(
Role B
Decision(
Type(
ROOT CAUSE of the Broken Feedback Loop

Manager
Alloca&on(
Decisions(
Knowledge(of(Risks(
Risk(Mgmt(
Decisions(
Developer
Communication
Breakdown
Broken Feedback Loop
(Manager Role)

So#ware(Task(
Mi.gate(the(Risks(
Product(
Development(
Product(
Work(Queue(
Risk(Mgmt(
Work(Queue(
Product(Owner(
Product(
Decisions(
Knowledge(of(
Customers(
Technical(Risk(Manager(
Risk(Mgmt(
Decisions(
Knowledge(of(
Risks(
Dev$Team$Capacity$
Manager
Alloca.on(
Decisions(
Turn Risk Management into an Investment Decision
Now we can steer!!

Three Dimensions of Organizational Knowledge
Discovery
(Expansion)
Peer Mentorship
(Diffusion)
Optimize the Whole
(Acceleration)
“Decision Principles”
(models)

Discovery
(Expansion)
Peer Mentorship
(Diffusion)
Optimize the Whole
(Acceleration)
Priority 1
Priority 2
Priority 3
Priority 1
Priority 2
Priority 3
Priority 1
Priority 2
Priority 3
Biggest Pains
$$
Biggest Skill Gaps
$
Limiting Constraints
$$$$
Turn Risk Management into an Investment Decision

Decisions are managed by a distributed
Network of Open Mastery Circles

4. Optimize the Whole
2. Design the Strategy
1. Identify Obstacles
(Focus)
3. Learn What Works
(Traction)
Steering
Wheel
Create a Steering Wheel to Drive the Organization!
$$ $$

Focus: Quality of Improvement Decisions
Focus Traction
$$$ $$
Focus: Quality of Investment Decisions
Risk
Translator
(interface)
Management
(Coordination)
Engineering
(Execution)
Open Mastery Learning Framework
Idea Flow Pain Sensor

If your Interested in doing this:
Free Support Network for

Open Mastery Learning Platform
Idea Flow Mapping Tools
Team Mastery Platform
Team
Joe
Sally
Mark
Eric
Community
Collaboration Platform
Anonymized
Data
Project
Tiger
Project
Bear
(REST)
Shared #HashTag Glossary
of Patterns & Principles
with Examples

The Conversation
is on Slack
We’re debating the patterns
and principles of Idea Flow in Wikipedia
It’s a Developer Party!
We’re sharing our ideas
over Blogs
(and managers are invited)

Industry Peer Mentorship Network
Companies
Community
Groups
HQ in
Austin
Open Mastery
Austin
meetup.com/Open-Mastery-Austin

Janelle Klein
openmastery.org @janellekz
Check out openmastery.org for details.
Read my Book.
Think About It.
FREE with
Reading GroupBuy It
Janelle Klein

Bringing Science to Software Development

More Related Content

What's hot (16)

Viewers also liked (17)

Similar to Bringing Science to Software Development (20)

Recently uploaded (20)

Bringing Science to Software Development