Iterate. Calculate. Communicate.
1 like398 views
This document discusses how to close the gap between measuring product metrics and learning from those metrics through an iterative process. It emphasizes establishing the right metrics to track, ensuring metrics are actionable by tying them to goals that change behaviors, and communicating metrics and plans to take action on results across teams. The document provides examples of metrics to track at different stages of product development and recommends establishing a single source of truth for metrics as well as communication checklists.
Iterate. Calculate. Communicate.
- 2. CLOSING THE GAP BETWEEN MEASURE + LEARN
- 3. Linked In: /in/xplusx Medium & Twitter: @laurex LAURE PARSONS SENIOR PRODUCT MANAGER
- 4. “THE GOAL OF BUILD-MEASURE-LEARN IS NOT TO BUILD A FINAL PRODUCT TO SHIP OR EVEN TO BUILD A PROTOTYPE OF A PRODUCT, BUT TO MAXIMIZE LEARNING THROUGH INCREMENTAL AND ITERATIVE ENGINEERING” Steve Blank, author of The Four Steps to the Epiphany BE ACTIONABLE!
- 5. 1. WHAT SHOULD YOU TRACK? 2. WHAT MAKES A METRIC “GOOD?” 3. HOW CAN YOU GET ALIGNMENT WITH YOUR TEAM AND LEADERSHIP? 4. WHAT SHOULD YOU DO TO ENSURE YOU LEARN? ‣ CASE STUDY: BUILDING A NEW FEATURE!
- 8. THE ONE METRIC From Lean Analytics by Benjamin Yoskovitz & Alistair Croll
- 9. UX TESTS MVP A/B TESTSCLASSIC LEAN MEASUREMENT
- 10. WHO, WHAT, WHEN, WHERE, WHY & HOW ▸WHO CUSTOMERS ARE ▸WHAT CUSTOMERS DO AND/OR TRY TO DO ▸WHEN DO USERS DISENGAGE ▸WHERE CUSTOMERS COME FROM ▸WHY CUSTOMERS CONVERT ▸HOW MUCH YOU CAN BUILD
- 11. GOOD METRICS
- 13. RATIOS & RATES
- 16. BE ACTIONABLE! TWO TYPES OF ACTIONABLE DATA ▸“Accounting” metrics can make your predictions more accurate ▸“Experimental” metrics help optimize product, pricing, or market. Agree on what will change before you collect the data.
- 17. “IF YOU MEASURE SOMETHING AND IT’S NOT ATTACHED TO A GOAL, IN TURN CHANGING YOUR BEHAVIOUR, YOU’RE WASTING YOUR TIME.” LEAN ANALYTICS by Ben Horowitz and Alistair Kroll BE ACTIONABLE!
- 19. OKRS
- 21. WHAT ACTIONS WILL YOU TAKE?
- 24. THE DATA TOOL YOUR TEAM WILL ACTUALLY USE
- 25. ‣A single source of truth ‣Company-wide common understanding and definitions ‣Appropriate permissions ‣Transparency ‣Collaboration via shared data and commenting ‣Automated integration from your tools ‣Sharing in Slack, where your team is already talking ‣Dashboards to track the important things for each team ‣One-click reports to share with management and stakeholders
- 26. CASE STUDY
- 27. CASE STUDY QUESTIONS DISCOVERY & LEAN CUSTOMER DEVELOPMENT ▸ Based on our problem hypothesis, what percentage of the 6-10 interviews we conduct confirm what we think is the problem?
- 28. CASE STUDY QUESTIONS MVP ▸ Solves a problem in the simplest way possible ▸ Could actually be better than the intended product, with “concierge onboarding" ▸ Asks a question, which should be clear and measurable.
- 29. CASE STUDY QUESTIONS EFFORT TO BUILD ▸ How much dev effort is anticipated in points or cost per developer hour? ▸ Reduce scope to the most essential parts of the feature. Build “nice to haves” when you have signal ▸ Track velocity, throughput, cycle times to anticipate & understand building costs
- 30. EFFORT TO BUILD
- 31. CASE STUDY QUESTIONS WHO CARES? ▸ Beta test the feature with customers who’ve requested it and/or with freemium or opt-in customers ▸ If you have a large enough customer base, survey your customers with a tool like Intercom or Qualaroo when they use the feature ▸ Measure feature adoption in Mixpanel or other event-based tools. See how often customers try the feature and how many times they re-use it.
- 32. CASE STUDY QUESTIONS COSTS BEYOND DEVELOPMENT ▸ Track support requests tagged with the feature name ▸ Track marketing spend on launching the feature ▸ Track escaped bugs
- 33. CASE STUDY QUESTIONS THE BOTTOM LINE ▸ Do a cohort analysis of customers before and after the feature was introduced. ▸ Track number of signups, churn, and referrals. ▸ If the feature is plan-gated, you can also track by pricing segments.
- 35. COMMUNICATION CHECKLIST ARE YOU COMMUNICATING? ▸ Does your whole team and company know why you are tracking these metrics? ▸ Do you have agreed-upon actions based on the results? ▸ Do you have to track down people to get the data? ▸ Does someone own the metric?
- 36. DON’T STOP NOW! LEARN MORE ▸Lean Analytics by Benjamin Yoskovitz & Alistair Croll ▸Getting Started with OKRs & Metrics Guide ▸The School of Little Data ▸The AARRRT of Product Metrics
- 37. THANK YOU!