How to Use Experimentation to Optimize Your Product and Business
3 likes344 views
The document outlines methods for optimizing products and businesses through experimentation and metrics analysis, emphasizing the importance of product-market fit. It describes three phases of product development: pre-market fit, post-market fit, and growth, while discussing qualitative and quantitative learning in each phase. Key topics include identifying high ROI metrics, improving retention rates, and the significance of holistic analytics frameworks to enhance product performance.
How to Use Experimentation to Optimize Your Product and Business
- 1. How to Use Experimentation to Optimize Your Product and Business Dan Olsen 3-28-19
- 2. n Engineering background: EE, submarine design n Stanford MBA n Product Management leader at Intuit & startups n CEO & Cofounder, TechCrunch winner YourVersion n Product Management consultant & trainer n Founder, Lean Product Silicon Valley Twitter: @danolsen My slides: https://dan-olsen.com My Background Copyright © 2019 @danolsen
- 3. Spiderman’s motto: “With great power comes great responsibility” Copyright © 2019 @danolsen
- 4. Product Manager’s motto: Spiderman’s motto: “With great power comes great responsibility” “With great responsibility comes NO power” Copyright © 2019 @danolsen
- 5. Product Teams Are Responsible For Achieving Product-Market Fit AND Growth Copyright © 2019 @danolsen
- 6. The 3 Phases of Your Product Copyright © 2019 @danolsen BEFORE Product-Market Fit AFTER Product-Market Fit BEFORE your product is live Phase 1 AFTER your product is live Phase 2 #ofCustomers Time 0 lots Phase 3
- 7. Quantitative vs. Qualitative Learning QualitativeQuantitative OprahSpock
- 8. Qualitative Complements Quantitative Copyright © 2019 Olsen Solutions Qual Why? Quant What? How many?
- 9. The 3 Phases of Your Product BEFORE Product-Market Fit AFTER Product-Market Fit Growth Mainly Quantitative BEFORE your product is live Phase 1 Test hypotheses, gain confidence before building Mainly Qualitative AFTER your product is live Phase 2 Improve product- market fit Qualitative & Quantitative Goal: Methods: Phase 3
- 10. The 3 Phases of Your Product BEFORE Product-Market Fit AFTER Product-Market Fit Growth Mainly Quantitative BEFORE your product is live Phase 1 Test hypotheses, gain confidence before building Mainly Qualitative AFTER your product is live Phase 2 Improve product- market fit Qualitative & Quantitative Goal: Methods: Phase 3 What phase are you in?
- 11. That’s Why I Wrote Copyright © 2019 @danolsen
- 13. The Product-Market Fit Pyramid
- 14. The Product-Market Fit Pyramid
- 15. The Product-Market Fit Pyramid
- 16. The Product-Market Fit Pyramid
- 17. The Lean Product Process
- 18. The Lean Product Process
- 19. The Lean Product Process
- 20. The Lean Product Process
- 21. The Lean Product Process
- 22. The Lean Product Process
- 23. The Lean Product Process
- 24. The Lean Product Process
- 26. Don’t Just Throw Spaghetti
- 27. Local vs. Global Maximum
- 28. Theory Experiment Data Evidence Supports or refutes A/B Testing Isn’t Enough Revise You need to understand WHY A outperformed B Your mental model of causality
- 29. Copyright © 2019 @danolsen
- 30. Identify highest ROI idea Design and Implement Analyze How the Metric Changes Brainstorm Ideas to Improve Metric Copyright © 2019 @danolsen Lean Product Analytics Process Identify What Your Metrics Are Measure Metrics Baseline Values Evaluate Metrics Upside Potential Global Level Metric Level Select Top Metric Learn & Iterate
- 31. Valuable to Have a Holistic Analytics Framework: Dave McClure’s AARRR!
- 32. Valuable to Have a Holistic Analytics Framework: Dave McClure’s AARRR!
- 33. Valuable to Have a Holistic Analytics Framework: Dave McClure’s AARRR!
- 34. Valuable to Have a Holistic Analytics Framework: Dave McClure’s AARRR!
- 35. Valuable to Have a Holistic Analytics Framework: Dave McClure’s AARRR!
- 36. At any point, one metric offers the highest return-on-investment (ROI) “Metric that Matters Most” (MTMM) Focus on right metric at right time Copyright © 2019 @danolsen
- 37. If you just launched a new product, which would you optimize first? Acquisition: attracting prospects Conversion: turning prospects into customers Retention: ensuring customers remain active Copyright © 2019 @danolsen
- 38. If you could track only 1 metric to measure Product-Market Fit, which metric would it be? Copyright © 2019 @danolsen
- 39. Retention Rate n Retention rate tracks what % of your customers are still active over time ~80% never use app again Curve either goes to zero or flattens out
- 40. Cohort Analysis Copyright © 2019 @danolsen
- 41. Improving Retention Rate Over Time= Increasing Product-Market Fit
- 42. Improving Retention Rate Over Time= Increasing Product-Market Fit
- 43. Improving Retention Rate Over Time= Increasing Product-Market Fit
- 45. Profit = Revenue - Cost Unique Visitors x Ad Revenue per Visitor Impressions/Visitor x Effective CPM / 1000 Visits/Visitor x Pageviews/Visit x Impressions/PV New Visitors + Returning Visitors Unique Visitors in Last Period x Monthly Retention Rate Define the Equation of your Business Peeling the Onion Advertising Business Model: Copyright © 2019 @danolsen
- 47. View Each Metric as a Gauge Copyright © 2019 @danolsen Minimum Possible Value Maximum Possible Value Current Value
- 48. Identifying Your Metric that Matters Most n What is the upside potential of each metric? n How much do we think we can “move the needle”? n What would the revenue impact be? n How many resources will it take to move the needle”? n Developer-days, time, money n Which metric offers best Return on Investment (ROI)? Copyright © 2019 @danolsen
- 49. Copyright © 2019 @danolsen Prioritizing Product Ideas by ROI Investment (developer-weeks) Return(ValueCreated) Idea C Idea B Idea D Idea A Idea F 1 1 2 3 4 2 3 4 ?
- 50. Types of Metric ROI SituationsReturn Investment Return Investment Return Investment Metric A Good ROI Metric B Bad ROI Metric C Great ROI Copyright © 2019 @danolsen
- 51. Case Study Copyright © 2019 @danolsen
- 52. • Which metric has highest ROI opportunity? Case Study: Optimizing Friendster’s Viral Loop Active Users Prospective Users Invite Click Succeed Invite click-through rate Conversion rate Don’t Click Fail Invites per sender % of users sending invites • Multiplied together, these metrics determine your viral ratio Users % of users who are active = 15% = 2.3 = 85% Registration Process Copyright © 2019 @danolsen
- 53. The Upside Potential of a Metric 0 100% 0 100% 0 ? Registration Process Yield % of users sending invitations Avg # of invites sent per sender 2.3 85% 15% Max possible improvement 0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?% Copyright © 2019 @danolsen
- 54. Is anyone feeling a sense of déjà vu right now?
- 55. The Upside Potential of a Metric 0 100% 0 100% 0 ? Registration Process Yield % of users sending invitations Avg # of invites sent per sender 2.3 85% 15% Max possible improvement 0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?% Copyright © 2019 @danolsen Metric B Bad ROI Metric A Good ROI Metric C Great ROI
- 56. Okay, so how can we improve the metric? n How do we increase the average number of invites being sent out per sender? n For each idea: n What’s the expected benefit? (how much will it improve the metric?) n What’s the expected cost? (how many engineer- days will it take?) n You want to identify highest ROI idea Copyright © 2019 @danolsen
- 57. After Launching Address Book Importer… Copyright © 2019 @danolsen
- 58. After Launching Address Book Importer… Copyright © 2019 @danolsen
- 59. After Launching Address Book Importer… Copyright © 2019 @danolsen
- 60. Identify highest ROI idea Design and Implement Analyze How the Metric Changes Brainstorm Ideas to Improve Metric Copyright © 2019 @danolsen Lean Product Analytics Process Identify What Your Metrics Are Measure Metrics Baseline Values Evaluate Metrics Upside Potential Metric Level Select Top Metric Learn & Iterate Global Level
- 61. Upping Your Experimentation Game n Simultaneous tests vs. before-and-after n Rollout to any % of users vs. only 100% Feature flags n Testing multiple variants vs. 1 alternative n Experimentation platform: buy vs. build n Experimentation velocity learning velocity Copyright © 2019 @danolsen
- 62. Ken Norton, Google Ventures Today: Thu March 28th in Mountain View What Product Teams Can Learn from Jazz Musicians
- 63. Thanks! @danolsen https://dan-olsen.com Lean Product Meetup: meetup.com/lean-product Happy to connect on LinkedIn