Meetup Turin (UX Torino and Optimizely)
1 like664 views
This document summarizes lessons learned from over 686,000 A/B tests. It provides six key lessons: 1. Define quantifiable metrics for success upfront such as sign-ups, donations, or engagement. 2. Reduce choices in experiments to get clearer results. Too many variations can dilute the effect. 3. Carefully craft the call to action as wording matters significantly. 4. Seek the global maximum by testing combinations of elements rather than individual factors. 5. Fail fast by killing experiments that are not working to focus resources on more promising ideas. 6. Challenge assumptions and test everything as nothing should be considered sacred.
Meetup Turin (UX Torino and Optimizely)
- 2. Lessons learned from over 686,000 A/B testsMarco Sgrignuoli (Software Engineer) Marco@optimizely.com Pedro B. Meyer (Country Manager – Italy, Spain & Portugal) Pedro@optimizely.com
- 5. Lesson #1: Define Quantifiable Success Metrics
- 6. $ Per RecipientVisitors Per Day Visit Sign-Up Rate Email $$
- 7. Media Button
- 8. Button ● Sign Up ● Learn More ● Join Us Now ● Sign Up Now Splash Page Experiment’s Variations Media • “Get Involved” Image • “Family” Image • “Change” Image • Barack’s Video • Springfield Video • Sam’s Video
- 13. Button ● Sign Up ● Learn More ● Join Us Now ● Sign Up Now Splash Page Experiment’s Variations Media • “Get Involved” Image • “Family” Image • “Change” Image • Barack’s Video • Springfield Video • Sam’s Video
- 20. Button ● Sign Up ● Learn More ● Join Us Now ● Sign Up Now Splash Page Experiment Media • “Get Involved” Image • “Family” Image • “Change” Image • Barack’s Video • Springfield Video • Sam’s Video
- 24. Splash Page Experiment Email Subscriptions Volunteers Amount Raised Original 7,120,000 712,000 $143,000,000 +40.6% +2,880,000 +280,000 +$57,000,000 New 10,000,000 1,000,000 $200,000,000
- 25. Lesson #1: Define Quantifiable Success Metrics
- 26. Lesson #2: Less Is More. Reduce Choices.
- 31. Lesson #2: Less Is More. Reduce Choices.
- 32. Lesson #3: Words Matter! Focus on your Call to Action
- 33. Donation Button Experiment Never Signed-Up Signed-Up No Donation Previously Donated 0.0% 0.0% 0.0% 2.3% 27.8% 16.3% -27.8 N/A N/A 15.2% -24.6% 11.9% 8.5% 2.9% 18.4%
- 35. Lesson #3: Words Matter! Focus on your Call to Action
- 36. Lesson #4: Seek the Global Maximum
- 42. Lesson #4: Seek the Global Maximum
- 47. Lesson #6: Nothing is Sacred. Challenge Assumptions.
- 50. Lesson #6: Nothing is Sacred. Challenge Assumptions.
- 51. Lesson #7: Get Started Today!
- 58. Live Demo!