Optimization Summer Games - Test Ideation
- 2. Generating High-Impact Experiments Hazjier Pourkhalkhali Senior Strategy Consultant @Hazjier
- 3. Optimizely is the world’s largest digital laboratory
- 5. We researched the world’s top companies Interviewed experts § Spoke with thought leaders in the industry § Met with top customers Built maturity model § Developed comprehensive maturity model § Assessed 100 customers § Surveyed 50+ customers Held 100+ Business Reviews § Met with executives at over 100 Optimizely clients Analyzed Top Revenue Experiments § Analyzed the 100 best experiments run on Revenue Per Visitor § Asked customers to describe test and process
- 6. User acquisition: 1.5m User retention 33% Add-to-Cart: 4.00% Cart Checkout: 50.0% Average quantity: 2.00 Average per unit price: €25.00 Average Order Value: €50.00 Conversion Rate: 2.00% Revenue Per Visitor: €1.00 Visitors: 2.0m Revenue: €2.0m Understand the revenue drivers
- 7. Goals Strategies Tactics Cart Checkout Rate Wording / messaging Location / size Design / color / iconograp hy Establish a visual hierarchy Remove non- critical content Limit number of choices Select/ target the right USP’s Refine the messaging Crystallize visually Clear and consistent UI Structure, naming, and order of sections Describe visually Test ideal product visuals Use the right review systems Descriptions Reviews Images Emphasize the Primary Call to Action Minimize Distractions Communicate Unique Selling Points Design Intuitive Navigation Market Products Effectively Build a Sense of Urgency Show quantity left Push temporary offers Promote buying before set times
- 8. Testing requires freedom LOW FREEDOM HIGH FREEDOM “Every change we make to our website has to be approved by our senior staff” -Financial Services company
- 9. Testing requires freedom LOW FREEDOM HIGH FREEDOM “Our Brand Styleguide is key. We cannot make changes that do not fit our design book” -Financial Services company
- 10. Testing requires freedom LOW FREEDOM HIGH FREEDOM “We would change our company logo if it converted better” -Travel Company
- 12. NO BEARD CAVEMAN .30% .29% .28% .32% .25% .74% .11% .28% .24% .32% .09% .59% .74% .59%
- 13. Minimize Distractions: Remove Non-Critical Content +14% Revenue Per Visitor
- 14. BASED ON KEYWORDCURATED +24% Click to Sale +30% Add-to-Cart +210% Conversions
- 17. § The user is reminded repeatedly of price § The user must remember the product benefits § The user must scroll down to the CTA when their basket is 3 or more Problem Solution Result § Reduce size of price § Highlight discounts § Add product visuals § Duplicate CTA at top of page § Increase Conversion Rate § Increase Conversion Rate § Increase CTR to checkout Build powerful hypotheses
- 18. § Analyze one section of your application / website § Document 10 different problems for your users § Group them by common root causes § Prioritize by urgency Problem Solution Result Optimizely Hypothesis Workshop
- 19. § Analyze one section of your application / website § Document 10 different problems for your users § Group them by common root causes § Prioritize by urgency Problem Solution Result Optimizely Hypothesis Workshop § Select 1 concrete problem § Design 10 solutions per theme § Choose 4 preferred solutions
- 20. The problemwith “A/B Testing” 2 A B 4 A B C D
- 21. Most people test only 2 variations 0 10,000 20,000 30,000 40,000 50,000 60,000 2 3 4 5 6 7 >8 Tests 65% 17% 17% # of Variations
- 22. Most people test only 2 variations 0 10,000 20,000 30,000 40,000 50,000 60,000 2 3 4 5 6 7 >8 Tests # of Variations
- 23. Test 5 variationsand nearly double your win rate! Win Rate # of Variations 14% 18% 20% 27% 20% 22% 23% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 2 3 4 5 6 7 >8 +90%
- 24. Running more variationswill significantlyimprove your testing!
- 25. § Analyze one section of your application / website § Document 10 different problems for your users § Group them by common root causes § Prioritize by urgency Problem Solution Result Optimizely Hypothesis Workshop § Select 1 concrete problem § Design 10 solutions per theme § Choose 4 preferred solutions § Primary Goals: What do you seek to impact immediately § Secondary Goals: What would you like to impact, but do not have to § Monitoring Goals: What are metrics you want to track
- 26. Optimizely funnel analysis Homepage Flight Results Book Flight Passenger Details Flight Options Payment Page Step 45% 30% 65% 65% 90% 70% 40% 40% 15% 20% 5% 25% 15% 30% 20% 15% 5% 5% Exit Rate Divergence Completion
- 27. Optimizely funnel analysis Homepage Flight Results Book Flight Passenger Details Flight Options Payment Page Step Primary / Secondary Goals § Searches Submitted § Category tabs § Deals § Frequent Flyer § Flight Selection § Filter Usage § Refine Search § Sort Order § Continue button § Edit search § Upsells § Field completion rate § Continue button § Continue button § Upsells § Seat selection § Conversion rate § Form field completion § Payment Type Monitoring Goals § Checkboxes § Flight tracker § Home page clicks § Menu interactions § Site search § Home page clicks § Menu interactions § Site search § Field error rate § Home page clicks § Menu interactions § Site search § Shopping cart pageviews § Home page clicks § Return to previous page § Edit address, information § Error rate
- 28. § Analyze one section of your application / website § Document 10 different problems for your users § Group them by common root causes § Prioritize by urgency Problem Solution Result Optimizely Hypothesis Workshop § Select 1 concrete problem § Design 10 solutions per theme § Choose 4 preferred solutions § Primary Goals: What do you seek to impact immediately § Secondary Goals: What would you like to impact, but do not have to § Monitoring Goals: What are metrics you want to track