You want it when? Probabilistic forecasting and decision making
Download as PPTX, PDF6 likes2,633 views
The document discusses the importance of probabilistic forecasting in decision-making, highlighting how understanding probabilities can help overcome bias and improve the quality of choices. It emphasizes the need for effective measurement in agile environments and cautions against common pitfalls that can harm agile transformations. The presentation offers insights into using metrics for self-improvement and decision-making, while also providing visualization criteria to enhance data representation.
You want it when? Probabilistic forecasting and decision making
- 1. @LMaccherone @TheAgileCraft Probabilistic Forecasting When do you want it? Topic presented by: Larry Maccherone @LMaccheroneDecision Making
- 3. @LMaccherone @TheAgileCraft Bias eats good decisions for breakfast By understanding probabilistic decision making, we learn to trust and overcome bias
- 5. @LMaccherone @TheAgileCraft Every decision is a forecast!
- 6. @LMaccherone @TheAgileCraft You are forecasting that your choice will have better outcomes than the other alternatives
- 7. @LMaccherone @TheAgileCraft So… quality of decision depends upon: 1. alternatives considered, and 2. models used to forecast the outcome of those alternatives. Probabilistic models are superior
- 9. @LMaccherone @TheAgileCraft For a given alternative, let: Pg = Probability of good thing happening Vg = “Value” of good thing happening Then: Value of the alternative = Pg × Vg
- 11. @LMaccherone @TheAgileCraft $8M Best case (25%) $1M Likely case (50%) $1M Worst case (25%) 1 $2M$2M$1M2 Which strategy is best… …for your company? PW × VW = .25 × -$1.00M = -$0.25M PL × VL = .50 × $1.00M = $0.50M PB × VB = .25 × $8.00M = $2.00M ----------- $2.25M …for your career? PW × VW = .25 × $1.00M = $0.25M PL × VL = .50 × $2.00M = $1.00M PB × VB = .25 × $2.00M = $0.50M ----------- $1.75M
- 12. @LMaccherone @TheAgileCraft If you get only 1 project then strategy 2 is better 75% of the time If you get ∞ projects then strategy 1 is better 100% of the time How many projects do you need for strategy 1 to be better more often than not?
- 14. @LMaccherone @TheAgileCraft Play with it yourself at: http://jsfiddle.net/lmaccherone/j3wh61r7/
- 16. @LMaccherone @TheAgileCraft Emotion and bias plays a part
- 17. @LMaccherone @TheAgileCraft Did any of you get emotional about the $1M loss? Did any of you want to question the $8M number? We’ve totally… …eliminated fear from the equation …changed the nature of the conversation
- 18. @LMaccherone @TheAgileCraft Argument is about who is right. Decision making is about what is right.
- 20. @LMaccherone @TheAgileCraft Trained/Calibrated Untrained/Uncalibrated Statistical Error “Ideal” Confidence 30% 40% 50% 60% 70% 80% 90% 100% 50% 60% 80% 90% 100% 25 75 71 65 58 21 17 68 152 65 45 21 70% Assessed Chance Of Being Correct PercentCorrect 99 # of Responses We are inaccurate when assessing probabilities Copyright HDR 2007 dwhubbard@hubbardresearch.com But, training can “calibrate” people so that of all the times they say they are X% confident, they will be right X% of the time
- 21. @LMaccherone @TheAgileCraft Equivalent Bet calibration What year did Newton published the Universal Laws of Gravitation? Pick year range that you are 90% certain it would fall within. Win $1,000: 1. It is within your range; or 2. You spin this wheel and it lands green Adjust your range until 1 and 2 seem equal. Even pretending to bet money works. 90% 10%
- 22. @LMaccherone @TheAgileCraft An agile delivery date forecast example
- 23. @LMaccherone @TheAgileCraft Monte Carlo Forecasting What it looks like Live demo: http://lumenize.com (use Chrome)
- 24. @LMaccherone @TheAgileCraft Seek to change the nature of the conversation
- 25. @LMaccherone @TheAgileCraft Getting even more sophisticated 1. Only use slopes after it stabilizes. Discard the first N. (Lumenize has v-optimal algorithm for finding this inflection point) 2. Weight later slopes more heavily. 3. Markov chain pattern reproduction. Accomplishes 1 and 2 above automatically. 4. Simulate the movement of each individual work item through the system. Can find bottlenecks and help optimize your role balance. Troy Magennis has the expertise and tools for this.
- 26. @LMaccherone @TheAgileCraft Using measurement in an agile environment
- 28. @LMaccherone @TheAgileCraft … but for those brave enough to journey into the dangerous world of agile measurement there are great riches to be had. The trick is to slay the dragons.
- 29. @LMaccherone @TheAgileCraft The Dragons of Agile Measurement If you do metrics wrong, you will harm your agile transformation 1. Dragon: Measurement as a lever Slayer: Measurement as feedback 2. Dragon: Unbalanced metrics Slayer: 1 each for Do it fast/right/on-time, and Keep doing it 3. Dragon: Metrics can replace thinking Slayer: Metrics compliment thinking 4. Dragon: Expensive metrics Slayer: 1st work with the data you are already passively gathering 5. Dragon: Using a convenient metric Slayer: Outcomes Decisions Insight Metric (ODIM) 6. Dragon: Bad analysis Slayer: Simple stats and simulation 7. Dragon: Single outcome forecasts Slayer: Forecasts w/ probability
- 30. @LMaccherone @TheAgileCraft Manipulating Others Dragon #1 Using metrics as a lever to drive someone else’s behavior
- 31. @LMaccherone @TheAgileCraft Self Improvement Dragon slayer #1 Using metrics to reflect on your own performance
- 32. @LMaccherone @TheAgileCraft Dragon #5 Using a convenient metric aka “Lamp post metrics”
- 34. @LMaccherone @TheAgileCraft Good players? Monta Ellis 9th highest scorer (8th last season) Carmelo Anthony (Melo) 8th highest scorer (3rd last season)
- 35. @LMaccherone @TheAgileCraft Dragon slayer #5 ODIM O U T C O M E D E C I S I O N I N S I G H T M E A S U R E THINK EFFECT like Vic Basili’s Goal-Question-Metric (GQM) but without ISO/IEC 15939 baggage
- 36. @LMaccherone @TheAgileCraft The Dragons of Agile Measurement If you do metrics wrong, you will harm your agile transformation 1. Dragon: Measurement as a lever Slayer: Measurement as feedback 2. Dragon: Unbalanced metrics Slayer: 1 each for Do it fast/right/on-time, and Keep doing it 3. Dragon: Metrics can replace thinking Slayer: Metrics compliment thinking 4. Dragon: Expensive metrics Slayer: 1st work with the data you are already passively gathering 5. Dragon: Using a convenient metric Slayer: Outcomes Decisions Insight Metric (ODIM) 6. Dragon: Bad analysis Slayer: Simple stats and simulation 7. Dragon: Single outcome forecasts Slayer: Forecasts w/ probability
- 37. @LMaccherone @TheAgileCraft What? So what? NOW WHAT? My 2nd talk 3:45 Wednesday National Harbor 6/7
- 38. @LMaccherone @TheAgileCraft Top 10 criteria for great visualization 1. Answers the question, "Compared with what?” (SO What?) 2. Shows causality, or is at least informed by it. (NOW WHAT?) 3. Tells a story with whatever it takes. 4. Is credible. 5. Has business value or impact in its social context. 6. Shows differences easily. 7. Allows you to see the forest AND the trees. 8. Informs along multiple dimensions. 9. Leaves in the numbers where possible. 10. Leaves out glitter. Credits: • Edward Tufte • Stephen Few • Gestalt (School of Psychology)
- 39. @LMaccherone @TheAgileCraft Now what? Enter feedback in Sched App Come to the AgileCraft booth: • Questions answered • Demo of how AgileCraft surfaces probabilistic visualizations and is the best way to scale agile Come to my talk on Wednesday at 3:45: • Using data to influence • Top 10 criteria for great visualization
- 40. @LMaccherone @TheAgileCraft “They” say… Nobody knows what’s gonna happen next: not on a freeway, not in an airplane, not inside our own bodies and certainly not on a racetrack with 40 other infantile egomaniacs. – Days of Thunder Trying to predict the future is like trying to drive down a country road at night with no lights while looking out the back window. – Peter Drucker Never make predictions, especially about the future. – Casey Stengel
- 41. @LMaccherone @TheAgileCraft When you come to a fork in the road… take it! ~Yogi Berra
- 42. @LMaccherone @TheAgileCraft Now what? Enter feedback in Sched App Come to the AgileCraft booth: • Questions answered • Demo of how AgileCraft surfaces probabilistic visualizations and is the best way to scale agile Come to my talk on Wednesday at 3:45: • Using data to influence • Top 10 criteria for great visualization
Editor's Notes
- #10: Aspect of the Bellman Equation. Let’s say that getting a 1st down is worth 2 “points contributed” and the chance of getting a 1st down is 50%. The weighted value of this alternative is 1 point contributed. Now, let’s say that the value of putting the team back at their 20 yard line vs giving them the ball right here on say our own 40 yard line is 1 point contributed and the probability of that happening is 80% (could have a blocked punt or big run back). It’s weighed value is then 0.8 points contributed. For any given roll of the dice, the punt could come out better, but statistically, over the long haul, going for it in this situation is the better alternative.
- #12: Now a real example that shows two different risk profiles in a business scenario
- #22: READ THIS 1687
- #41: They all get it wrong. You are already predicating the future every time you make a decision. That last one sounds like something Yogi Berra would say. Only he didn’t. Instead, this is what he said…
- #42: Every decision is a forecast. You are now armed with ways to avoid cognitive bias, creating models that produce probabilities, translating proxy variables into money using “values”. Make better decisions.