Stop Flying Blind! Quantifying Risk with Monte Carlo Simulation
9 likes4,944 views
The document discusses the importance of quantifying product risks for startups using Monte Carlo simulation and practical metrics. It emphasizes understanding risk through impact and probability, with examples of different scenarios affecting product development and decision-making. The author, Sam McAfee, provides a framework for leveraging business metrics to assess risks and optimize product lifecycle profits.
Stop Flying Blind! Quantifying Risk with Monte Carlo Simulation
- 1. Stop Flying Blind! Quantifying Startup Product Risks with Monte Carlo Simulation. Sam McAfee, Principal - @sammcafee
- 2. Slides available at bit.ly/riskiest_assumption
- 3. Who is this guy? • Dot-com boom survivor. • Ran a dev shop for 10 years after that. • Agile has always been the “norm”. • Read ‘4 Steps’ before #LeanStartup. • Participated in 3 different startups. All dead. • Ran Change.org engineering. • Learned LeanUX from @clevergirl. • Wore plaid and facial hair before it was cool. Tweet every word: @sammcafee
- 4. A Typical Scenario Maybe you’ve experienced something like this?
- 5. A typical scenario The CEO
- 6. “ ” Our competitors have it, so we should have it too!
- 7. A typical scenario Product Management
- 8. It takes too long to get new features out. “ ”
- 9. A typical scenario Engineering
- 10. “ ” We have to refactor all the things we rushed out the door last time.
- 11. How do we prioritize competing objectives?
- 12. Understanding Risk It’s kind of important for startups.
- 13. Understanding Risk Why Is Risk Important?
- 14. Startups are risky! You are participating in one of the highest risk business endeavors there is. Risk is a business term. You are in business. Let’s use terminology that business people use to make decisions. Ignore it at your peril. Choosing to ignore risk will not protect you from it.
- 15. Understanding Risk What Is Risk?
- 16. Risk = Impact * Probability
- 17. Understanding Risk Stop Using Your Gut!
- 18. “ ” All people, including experts and managers, are very bad at assessing the probabilities of events. — Doug Hubbard
- 19. Risk = Impact * Probability
- 20. How Do We Define Impact? Hint: For startups, it can probably be found on the bottom line.
- 21. Economic framework. Allows you to compute the impact of any change in the system into a single unit of measure. Start with your P&L. Your CEO or finance people already have a framework for you to start with. Include them. Apply risk scenarios. Calculate the impact of likely scenarios on the total product life-cycle profits, or similar KPI.
- 22. Scenario 1 Cost of Delay
- 23. Cost of Delay Scenarios • Projects will have different cost of delay curves. • What is the effect on total life-cycle profits for each unit of delay? • Delay in product development is overwhelmingly affected by queues between steps rather than job duration at a given step. • Cost of Delay enables you to calculate the cost of queues.
- 24. Scenario 2 Sequence of Work
- 25. Work Sequence Scenarios • The order in which you do work can have a substantial impact on the total cost of queues. • If cost of delay is homogeneous, do the shortest job first. • If job duration is homogeneous, do the highest cost of delay first. • You can combine job duration and cost of delay using weighted shortest job first.
- 26. Tangent: which is the shortest job? • Job duration is an estimate. • Estimates are probabilistic, not deterministic. • Use cycle time and throughput to compute a probability distribution for likelihood of job duration.
- 27. Scenario 3 Capacity vs. Queue Cost
- 28. Capacity Utilization Scenarios • You can reduce the cost of queues by adding additional capacity. Obviously, additional capacity has a cost too. • Additional capacity increases transaction costs, but lowers holding (delay) costs. • Is the cost of adding additional capacity justified by the gains in product development throughput and lower queue costs?
- 29. How Do We Define Probability? Use your metrics, Luke!
- 30. Use business metrics. You are probably already tracking tons of data about how you acquire new customers and how they enter and leave your product funnels. Value each conversion. Use your historical data to calculate values for each step in a product funnel. Note areas of variance. The steps in your funnel that have consistent, regular conversion rates are unlikely to change. They represent lower information content.
- 31. How Do We Value Information? Hint: It’s probabilistic too.
- 32. Information Has Value • 10,000 visitors, 10% sign-up, $20/month. • Each visitor is worth $2/month (10% x $20/month). • A test to increase rate to 15% is worth: • An additional $10/month * probability of test success.
- 33. Experiments Have Value • Experiments attempt to capture new information. • Experiment value = expected benefit - cost of running the test. • Expected benefit = increase in KPI * probability of success. • Cost of running the test = cost of delay * job duration.
- 34. Your backlog is a prioritized list of experiments.
- 35. Monte Carlo Simulations It’s not just for insurance geeks anymore!
- 36. Monte Carlo Simulation • A series of dependent variables, each with probability distributions. • Randomly selects a value from each variable, and computes output. Repeats thousands of times. • End result is another probability distribution, in the unit of measure that you care about. • It’s 2014. You can do this in a spreadsheet, in about an hour.
- 37. Basic funnel Using time series data from your funnel, create probability distributions for each step toward capturing revenue. Basic histogram After 5,000 or so simulations, display a histogram of the expected output. 1 2 1 2
- 38. References +
- 39. Thank you!
- 40. Slides available at bit.ly/riskiest_assumption