Dan mc kinley_final
8 likes1,283 views
This document summarizes Dan McKinley's talk at the Lean Startup Conference about using data-driven decision making at Etsy. Some key points: 1) In the early days of Etsy, product decisions were based on "coolest ideas" but later shifted to using A/B testing and metrics to evaluate ideas. 2) Examples show how estimating metrics like conversion rates and order values can help determine if ideas are worth pursuing. 3) This approach of using basic math and estimates led to systemic improvements in Etsy's sales and product decisions over time.
Dan mc kinley_final
- 1. Data Driven Done Right Dan McKinley @mcfunley mcfunley.com Lean Startup Conference -‐ San Francisco, CA December 10th, 2014
- 2. IntroducHons
- 4. 2007 2003 2014 1998
- 5. 2007 2003 2014 1998
- 6. Handmade & vintage marketplace. $1.3 Billion in Gross Sales in 2013. 600ish Employees.
- 7. Less than 20 people in 2007.
- 8. photo: Ian Norman on Flickr.
- 9. “What do we build?” photo: Wilhelm Joys Andersen on Flickr.
- 10. “We should be data-‐driven.”
- 11. We should be data driven. This is data. ∴ we should be driven by this.
- 12. Data-‐Driven PrioriHes This is a thing that’s possible. ! I fell ass-‐backwards into learning a thing or two about it.
- 13. War Stories
- 14. In the Beginning
- 15. Skateboard ramp demoli;on, 2007.
- 18. Ideas that sound awesome are awesome. -‐ Etsy product philosophy in the early days. photo: srslyguys on Flickr.
- 19. photo: JusDn Brockie on Flickr.
- 20. Products in the Free-‐Growth Universe (They RULE) Released a thing Released a thing Released a thing Oh man that thing we released, I tell you what
- 22. Product Release Plan 1. Build feature. 2. Rent warehouse for launch party. 3. Release feature. 4. Launch party. 5.Wait 20 months. 6.Delete feature for lack of usage. photo: James Willamor on flickr.
- 23. 2008
- 24. 2011
- 25. A/B tesHng: a soluHon?
- 27. Product selecHon: sHll “coolest idea.” photo: nanpalmero on Flickr.
- 28. nanpalmero on Flickr. Product selecHon: sHll “coolest idea.” But the measurement had changed! photo: masterpunk.etsy.com
- 29. “Only Hope stayed under the rim of the jar.” photo: ketrin1407 on Flickr.
- 30. Something that Actually Started to Work
- 31. Time 2005 Idea Code Release
- 32. 2005 2010 Idea Code Release Idea Code A/B test Release?
- 33. “Oh, crap” 2005 2010 Idea Code Release Idea Code A/B test Release?
- 34. Wasted effort (maybe). THE HELL WITH IT! 2005 2010 Idea Code Release Idea Code A/B test Release?
- 35. 2005 2010 2012 Idea Code Release Idea Code A/B test Release? Idea Validate Prototype A/B Test Refinement A/B test Release
- 36. Use data here. 2005 2010 2012 Idea Code Release Idea Code A/B test Release? Idea Validate Prototype A/B Test Refinement A/B test Release
- 37. 2005 2010 2012 Idea Code Release Idea Code A/B test Release? Idea Validate Prototype A/B Test Refinement A/B test Release Possibly quite crappy.
- 38. Made less crappy here. 2005 2010 2012 Idea Code Release Idea Code A/B test Release? Idea Validate Prototype A/B Test Refinement A/B test Release Possibly quite crappy.
- 39. 2005 2010 2012 Idea Code Release Idea Code A/B test Release? Idea Validate Prototype A/B Test Refinement A/B test Release Maybe give up. Maybe give up. Maybe give up (hopefully not).
- 40. Examples
- 41. Here are two real ideas. Neither should sound completely nuts. photo: machineisorganic on Flickr.
- 42. Idea #1 Let’s build a landing page for local furniture.
- 44. Step 1: Who would see it?
- 45. Lis$ng Search Shop Home Other “A significant fracDon” Traffic Percent (%) Browse
- 46. Lis$ng Search Shop Home Other Traffic Furniture Percent (%) Browse
- 47. Lis?ng Search Not a significant fracDon. Shop Home Other Traffic Items sold Percent (%) Browse
- 49. Step 2: What would change imply?
- 50. visits × conversion rate × order value × liH ≈ ?
- 51. visits × conversion rate × order value × liH ≈ ? Es#mated Modeled
- 52. 50K × 0.25% × $400 × lig ≈ ? +2%? +1%? (fake numbers, by the way) $500,000 $375,000 $250,000 $125,000 $0 +0.5%? One Year
- 53. 50K × 0.25% × $400 × lig ≈ ? $500,000 $375,000 $250,000 $125,000 $0 +2%? +1%? +0.5%? One Year (fake numbers, by the way)
- 54. 50K × 0.25% × $400 × lig ≈ ? $500,000 $375,000 $250,000 $125,000 $0 +2% ≈ $360,000 +1%? +0.5%? (fake numbers, by the way) One Year
- 55. 50K × 0.25% × $400 × lig ≈ ? $500,000 $375,000 $250,000 $125,000 $0 $360,000 $13K (our cut) (fake numbers, by the way) One Year
- 56. Verdict: Let’s do something else.
- 57. Idea #2 Let’s email people that give up in the middle of purchases.
- 59. users × order value × resurrec?on rate ≈ ?
- 60. 20K × $40 × resurrecHon rate ≈ ? $20,000,000 $15,000,000 $10,000,000 $5,000,000 $0 +5%? +2%? +1%? One Year (fake numbers, again)
- 61. 20K × $40 × resurrecHon rate ≈ ? $20,000,000 $15,000,000 $10,000,000 $5,000,000 $0 +5%? +2%? +1%? One Year (fake numbers, again)
- 62. 20K × $40 × resurrecHon rate ≈ ? $20,000,000 $15,000,000 $10,000,000 $5,000,000 $0 $3,000,000! One Year (fake numbers, again)
- 63. 20K × $40 × resurrec?on rate × 3.5% ≈ ? $20,000,000 $15,000,000 $10,000,000 $5,000,000 $0 $3,000,000 One Year $100K! (our cut) (fake numbers, again)
- 64. Verdict: Let’s build it!
- 65. Abributed Source of Etsy’s Sales Other My email (epic)
- 66. Systemic improvement through basic arithmeHc!
- 67. GeneralizaHons
- 69. Who? ! How many?
- 70. Close is good enough.
- 71. YMMV
- 72. Startup Mature Company Moonshots Strategic Projects Vebed OpportuniDes h/t Marc Hedlund
- 73. Conclusions
- 74. “What do we build?” photo: Wilhelm Joys Andersen on Flickr.
- 75. Many product ideas sound awesome. photo: mindonfire on Flickr.
- 76. “Upon what meat doth this our Caesar feed, that he is grown so great?”
- 77. “He who refuses to do arithme;c is doomed to talk nonsense.” -‐ John McCarthy