Milion Dollar Impact Through Metrics, Analytics & A/B Testing
- 1. Million Dollar Impact Through Metrics, Analytics, & A/B Testing
- 2. Hello! I am Azhar Bande-Ali Software Engineer turned PM with a passion for people and data You can find me at: @AzharBA
- 3. In this talk ◎ Discuss web products at scale ◎ Narrow approach due to time constraints ◎ Principles can be adapted to your world Feel free to ask questions
- 9. Metrics Define your measurement criteria
- 10. Top Down Metrics Measure what your business cares about. Everything else is vanity.
- 11. Place your screenshot here E-commerce Website Improve yearly revenue from online sales
- 12. Metrics for growing revenue
- 13. Conversion Rate Metrics for growing revenue Drop-off Rate Return Rate
- 14. Conversion Rate % of unique sessions with a submitted order Metrics for growing revenue Drop-off Rate For every page in the ordering flow: % of sessions that don’t graduate to next step Return Rate % of new users who repeat order Compare each metric to your target/goal and industry average!
- 16. OR
- 17. Getting started with Analytics ◎ Needs code change - development cost ◎ Measure: ○ Conversion rate of funnels and ○ Drop off rates on pages ◎ Event tracking for more intricate analysis
- 18. Marry analytics to business goals Conversion Rate Drop off Rate Return Rate
- 19. Measure Frameworks ◎ Funnels ◎ AIDA ○ Awareness ○ Interest ○ Desire ○ Action
- 20. Measure Frameworks ◎ Funnels ◎ AIDA ○ Attention ○ Interest ○ Desire ○ Action Events
- 21. Drop-offs
- 22. ROI
- 23. How to get buy in from your boss and partners
- 24. $1,085,136 How to get buy in from your boss and partners
- 25. A/B Test Validate your hypothesis
- 27. Hypothesis Making “Continue” button more prominent will improve the checkout conversion rate.
- 28. Variant 1 Remove “Cancel Order” Button
- 29. Variant 2 Remove “Cancel Order” button and rename “Continue” to “Checkout”
- 30. And the winner is..
- 31. And the winner is.. ..neither!
- 32. Truth about A/B testing Diminishing Return Even a successful A/B test variant will eventually start failing for a variety of reasons. No Winners It is possible that none of your variants cause a big enough change to be declared winner Success Rate According to KissMetrics, only 1 out of 8 A/B tests have a valid winner Significance Sample being tested on should be big enough to represent population Control Group Always show a ‘no change’ state to a control group to compare understand what would happen if you did nothing Representation The test should be conducted on the same group of users that are a sample of the actual user set
- 33. Next Steps Org Support Ensure that your organization recognizes the value of the opportunity enough to prioritize it Iterate Be ready to come up with several hypothesis about which features create value for users Failure Even the best teams have a 12.5% chance of success. Know that it isn’t easy and you’ll fail a lot
- 35. Thanks! Any questions? You can find me at: @AzharBA pcampatl@azharb.com