To Estimate or Not to Estimate, Is that the Question? (2017 Better Software Conference)
4 likes582 views
This document discusses different approaches to software project estimation, including arguments for and against estimating. It presents a spectrum of estimating approaches from never estimating to always estimating. It advocates for probabilistic and data-driven forecasting over deterministic estimates based on intuition. The key ideas are to reduce sources of variation, gather data over time to build probabilistic forecasts, focus on delivering value quickly, and establish service-level expectations rather than commitments.
To Estimate or Not to Estimate, Is that the Question? (2017 Better Software Conference)
- 1. # b e t t e r s o f t w a r e c o n | @ m a t t p h i l i p TO ESTIMATE OR NOT TO ESTIMATE Is that the Question?
- 2. @mattphilip #bettersoftwarecon HOFSTADTER’S LAW “It always takes longer than you expect, even when you take into account Hofstadter’s Law.”
- 3. @mattphilip #bettersoftwarecon PARKINSON’S LAW Work expands so as to fill the time available for its completion.
- 5. @mattphilip #bettersoftwarecon THE SPECTRUM OF ESTIMATING NeverEstimateAnything AlwaysEstimateEverything Estimate Default Behavior Don’t estimate Effort Factors All sources of variation Intuition Approach Data Deterministic Forecast Type Probabilistic Tasks in hours Level of Detail Work items in days High Effort Spent Low Estimating better Improvement Aim Reducing variation Commitment Management Service Expectations
- 8. @mattphilip #bettersoftwarecon WHAT NOESTIMATES IS NOT SAYING ▫︎ You are evil if you estimate ▫︎ All estimates are totally useless ▫︎ Stop doing your successful estimating practice ▫︎ Stop having the conversations to understand/analyze/ break down work ▫︎ Work items must be the same size ▫︎ You must place your full faith and confidence in Monte Carlo forecasts
- 9. @mattphilip #bettersoftwarecon WHAT NOESTIMATES IS SAYING ▫︎ Know why you are estimating ▫︎ Discover for yourself how good you are at estimating (measure) ▫︎ Keep doing the things that help you understand the work ▫︎ Upfront estimates need to be held loosely ▫︎ If you focus on delivering value quickly, you obviate the need for Iron Triangle considerations
- 10. @mattphilip #bettersoftwarecon MY NOESTIMATES MANIFESTO … We have come to value: Reducing sources of variation over Improving estimating Data over Intuition* Probabilistic over Deterministic MVP scope over Full scope That is, while there is value in the items on the right, we value the items on the left more. *Neil Killick uses “empiricism over guesswork”
- 12. @mattphilip #bettersoftwarecon DO YOU ASSUME CORRELATION? Is the initial sizing a good predictor for when you can get your stuff? In our case, the surprising truth was ”no.” — Mattias Skarin, Real-World Kanban
- 13. @mattphilip #bettersoftwarecon WHAT WE ESTIMATE Code
- 14. @mattphilip #bettersoftwarecon WHAT WE ESTIMATE Analyze+Design+Code+Test+Deploy
- 15. @mattphilip #bettersoftwarecon WHAT ACTUALLY HAPPENS Analyze Code Test DeployDesign Wait for approval Wait for staff availability Wait for rework Wait for API availability Wait for higher-priority work to finish Wait for approval Wait for blocker removal
- 16. @mattphilip #bettersoftwarecon WHAT’S GOING ON? Value-added time _________________________ Total elapsed delivery time Flow efficiency =
- 17. @mattphilip #bettersoftwarecon WHAT’S GOING ON? Low process efficiency (typically 5-15% in software delivery) means that even if we nailed the effort estimates … we would be accurately predicting 5-15% of elapsed delivery time! — Troy Magennis
- 18. @mattphilip #bettersoftwarecon SOURCES OF VARIATION How many can you name?
- 19. @mattphilip #bettersoftwarecon SOURCES OF VARIATION Waiting for availability Rework Team dependencies Stages in team development (Tuckman) Steps/ handoffs Selection policy Essential complication Accidental complication System dependencies Work In Progress Technology/ domain/ product Specialization Team composition Multitasking/ context switching Collaboration policy Blockers
- 20. @mattphilip #bettersoftwarecon WHAT CAN YOU DO ABOUT VARIATION How many remedies can you name?
- 21. @mattphilip #bettersoftwarecon WHAT YOU CAN DO ABOUT VARIATION ▫︎Lower WIP ▫︎ConWIP/System WIP ▫︎Five Focusing Steps ▫︎Blocker clustering ▫︎Reduce workflow stages ▫︎Explicit policies ▫︎Cost of Delay scheduling, sequencing and selection ▫︎Reduce “expedites” Lean-Kanban
- 22. @mattphilip #bettersoftwarecon WHAT YOU CAN DO ABOUT VARIATION ▫︎“Agile 101” (simple, decoupled design; thin vertical slices; pairing) ▫︎Identify/make visible/measure dependencies ▫︎Collaborate/Share work (Dimitar Bakardzhiev) ▫︎Spike and stabilize (Dan North) ▫︎Reduce accidental complexity (Liz Keogh) Team Why?
- 23. @mattphilip #bettersoftwarecon WHAT YOU CAN DO ABOUT VARIATION ▫︎Determine what actions would be different based on the estimate ▫︎Customer-based fitness criteria ▫︎Budgeting: Team run rate ▫︎Focus conversation on value, not cost ▫︎MVP and product ownership ▫︎Create probabilistic forecast ASAP (as soon as you have data) – together! ▫︎Service-Delivery Reviews ▫︎Teams: Keep teams together, dedicated (reduces context-switching, Tuckman stages) Business
- 24. @mattphilip #bettersoftwarecon POLICIES TO REDUCE VARIATION ▫︎We will only start new work at about the same rate that we finish old work. ▫︎We will make every reasonable effort to finish all work that is started and minimize wasted effort due to discarded work items ▫︎If work becomes blocked, we will do everything we can do unblock that work as expeditiously as possible. ▫︎We will closely monitor our policies around the order in which we pull items through our system so that some work items do not sit and age unnecessarily. — Daniel Vacanti, When Will It Be Done?
- 26. @mattphilip #bettersoftwarecon HUMAN BIAS ▫︎Recency ▫︎Availability ▫︎Gambler’s conceit ▫︎Anchoring ▫︎Halo effect ▫︎Motivational ▫︎Confirmation
- 27. @mattphilip #bettersoftwarecon FIRST, KNOW WHERE YOU ARE
- 28. @mattphilip #bettersoftwarecon KEOGH’S “SCALE OF IGNORANCE” 1. Just about everyone in the world has done this. 2. Lots of people have done this, including someone on our team. 3. Someone in our company has done this, or we have access to expertise. 4. Someone in the world did this, but not in our organization (and probably at a competitor). 5. Nobody in the world has ever done this before.
- 29. @mattphilip #bettersoftwarecon DATA OVER INTUITION
- 32. @mattphilip #bettersoftwarecon PROBABILISTIC OVER DETERMINISTIC Dec 28 — graphic courtesy of Larry Maccherone, (maccherone.com/lumenize)
- 33. @mattphilip #bettersoftwarecon VACANTI’S VERITY When making a forecast (predicting the future), you have to accept that there is more than one possible outcome. Therefore… A forecast is a calculation about the future that includes both a range and a probability of that range occurring.
- 34. @mattphilip #bettersoftwarecon PROBABILISTIC OVER DETERMINISTIC
- 35. @mattphilip #bettersoftwarecon PROBABILISTIC OVER DETERMINISTIC
- 36. @mattphilip #bettersoftwarecon PROBABILISTIC OVER DETERMINISTIC
- 37. @mattphilip #bettersoftwarecon PROBABILISTIC OVER DETERMINISTIC 2% 20% 21% 3% 16% 16% 9% 7% 2% 2% 2% Dec 28 Feb 17Nov 30 Jan 20
- 40. @mattphilip #bettersoftwarecon BACK TO YOUR SCATTERPLOT
- 41. @mattphilip #bettersoftwarecon STRAIGHT-LINE (AVERAGE) FORECASTS Team A Team B Work item #1 5 days 3 days Work item #2 1 day 3 days Work item #3 3 days 3 days Average 3 days 3 days 3-day delivery time expectancy 66% 100%
- 42. @mattphilip #bettersoftwarecon EXPECTATIONS, NOT COMMITMENTS
- 43. @mattphilip #bettersoftwarecon SERVICE-DELIVERY EXPECTATIONS Expectation Performance Expedite 90% 3 days or less Standard Urgency 85% 10 days or less Fixed-Date 90% On-time
- 44. 44 IN SUMMARY
- 45. @mattphilip #bettersoftwarecon TO ESTIMATE OR NOT TO ESTIMATE? NeverEstimateAnything AlwaysEstimateEverything Estimate Default Behavior Don’t estimate Effort Factors All sources of variation Intuition Approach Data Deterministic Forecast Type Probabilistic Tasks in hours Level of Detail Work items in days High Effort Spent Low Estimating better Improvement Aim Reducing variation Commitment Management Service Expectations
- 46. @mattphilip #bettersoftwarecon BETTER QUESTIONS TO ASK @mattphilip ▫︎In what context would estimates bring value, and what are we willing to do about it when they don’t? – Woody Zuill ▫︎How much time do we want to invest in this? – Matt Wynne ▫︎What can you do to maximize value and reduce risk in planning and delivery? – Vasco Duarte ▫︎Can we build a minimal set of functionality and then learn what else we must build? ▫︎Would we/you not invest in this work? If not, at what order-of-magnitude estimate would we/you take that action? ▫︎What actions would be different based on an estimate?
- 47. @mattphilip #bettersoftwarecon HOW TO GET STARTED (PROJECT IN PROGRESS) 1. Continue to discuss the work to analyze it and break it into thin vertical slices. 2. Start tracking two pieces of data for each work item: commit date and delivery date. 3. After accumulating 10 data points, run a Monte Carlo simulation to provide a probabilistic forecast. 4. Begin answering the “When” question using data, either at the project level or individual work-item level. 5. Validate whether the probabilistic forecasts match what actually occurs. 6. If the forecasts are helpful, wean your team off estimating. 7. Focus improvement efforts on reducing the sources of variation.
- 48. @mattphilip #bettersoftwarecon HOW TO GET STARTED (NEW PROJECT) 1. Ask the “better questions” above about the use and importance of estimates. 2. Use reference-class data (similar delivery times from other projects in similar tech stacks with similar teams) to provide an initial probabilistic forecast 3. As soon as you accumulate 10 data points, run a Monte Carlo simulation to provide an updated probabilistic forecast. 4. Continually reforecast when you get more information.
- 49. @mattphilip #bettersoftwarecon REFERENCES AND FURTHER EXPLORATION ▫︎ noestimatesbook.com (Vasco Duarte) ▫︎ infoq.com/articles/noestimates-monte-carlo (Dimitar Bakardzhiev) ▫︎ priceonomics.com/why-are-projects-always-behind-schedule ▫︎ http://scrumandkanban.co.uk/estimation-meets-cynefin/ ▫︎ ronjeffries.com ▫︎ Lizkeogh.com ▫︎ https://neilkillick.wordpress.com/ ▫︎ http://zuill.us/WoodyZuill/ ▫︎ mattphilip.wordpress.com/noestimates-game ▫︎ When Will It Be Done? (Dan Vacanti) ▫︎ focusedobjective.com (Troy Magennis) ▫︎ actionableagile.com (Dan Vacanti) ▫︎ kanbanize.com ▫︎ scrum.org