Greg Kelton - "Multivariate Testing at Orange"
3 likes541 views
This document discusses how multivariate testing (MVT) was used by Orange, an online telecommunications company, to improve the performance of their website. MVT analyzes thousands of variations of web page elements like images, text, layouts, and calls to action to determine which combination performs best. For Orange, testing different creative elements on their shopping page led to a 75% increase in click-through rates. On other pages, testing things like animated images versus static images increased conversion rates by up to 10%. The document advocates that MVT allows companies to continuously improve their online operations and revenue by getting direct feedback from customers on what content and designs they engage with most.
Greg Kelton - "Multivariate Testing at Orange"
- 1. Multivariate Testing at Orange Driving more revenue online Greg Kelton Managing Director, Autonomy Optimost EMEA Interwoven Confidential
- 2. Quiz! Why Multivariate Conversion Analysis Case Studies (Orange, Delta, etc.) How it works Q&A Slide 2 2
- 3. 1 A A Control B B +75% 3
- 4. Generating a 75% increase in Clickthroughs • 14 template permutations analyzed Control (creative 2) Winner (creative 11) UAR-Shop = x% UAR-Shop = x% +75% 4
- 5. 2 A A +6.97% B B Control 5
- 6. Generating a 53.42% lift (99% confidence) Total Recipes Analysed: 2,938 B4 F=1 Control generating a 53.42% lift (99% confidence) 6
- 7. 3 Flash (animated) A A B B Control +4.1% 7
- 8. Winning creative beat control by 10.0% on UAR-Congratulations and 8.2% on UAR-Profile Information Over 1 billion template permutations analyzed Control (creative 121) Winner (creative 271) UAR-Congratulations = x% UAR-Congratulations = x% +10.0% UAR-Profile Information = x% UAR-Profile Information = x% +8.2% 8
- 9. Who should control your marketing? Shareholders/ Shareholders/ Investors Investors Product Product Marketing Marketing Managers Managers Dept Dept Creative Creative CEO CEO Agencies Agencies Executives Executives Analysts Analysts 11
- 10. Who should control your marketing? Shareholders/ Shareholders/ Investors Investors Product Product Marketing Marketing Managers Managers Dept Dept None of the above Creative Creative CEO CEO Agencies Agencies Executives Executives Analysts Analysts 12
- 11. Who should control your marketing? Your marketing should be driven by your customers Your Your Customers Customers 13
- 12. 14
- 13. Quiz! Why Multivariate Conversion Analysis Case Studies (Orange, Delta, etc.) How it works Q&A Slide 15 15
- 14. The Content Selection Process is Complex & Faulty How do you make web content decisions? • Speculation • HiPPO • Intuition • Agency experience • Argument • Quasi-objective • Focus groups What if Your Customers • Usability testing Could Tell You What • Web Surveys Content They Like? 16
- 15. 17
- 16. Old Facebook New Facebook 18
- 17. Top facebook Applications See 25 Percent Drop In Traffic Since Redesign facebook Poll: 94% Of Users Don’t Like Redesign 19
- 18. 20
- 19. 22
- 20. 23
- 21. 24
- 22. 25
- 23. Quiz! Why Multivariate Conversion Analysis Case Studies (Orange, Delta, etc.) How it works Q&A Slide 29 29
- 24. But…. What’s the point? • Your online competitive advantage • Your Sales Funnel: • Ease of buying & customer experience? • Multi-channel influences? • Technology investment (Flash, AJAX, Adobe AIR?) • Your Operational Advantage: • Change management? • Cost reduction? • Putting business in control of your site? • Speed to market? • Your Brand & Offering: • Regional customers • Price sensitivity? By region? By demographic? 30
- 25. Affiliate Marketing Page Simplify the Buying Process • Simple version increased conversion by 3.86% 32
- 26. Putting business in control of your online channel Merchandising Challenge - Brands, price brackets, colours vary depending on product category •How many price brackets? •Should the number of products in stock be displayed? •How many brands? •Which order is best? •Are collapsible menus preferred? Benefit –understand how your customers prefer to interact with site technologies eg. reviews, search, navigation, shopping carts. 33
- 27. Shop Home Page 35
- 28. Shop Home Page C A, B 36
- 29. 30 total Page Recipes analysed Quiz! B Control A +17% 39
- 30. Product Page 40
- 31. Product Page 41
- 32. 60 total Page Recipes analysed Quiz! A -9% B +13% 43
- 33. Quiz! Why Multivariate Conversion Analysis Case Studies (Orange, Delta, etc.) How it works Q&A Slide 47 47
- 34. 1. Pick a Page to Test 48
- 35. 2. Add Single Javascript Callout to Page <script optimost.displayModule("main"); </script> 49
- 36. 3. Brainstorm ideas to test 1 2 3 vs None 4 51
- 37. 4. Optimost Generates Page Recipes 52
- 38. 5. Run the Experiment Live on your site! 53
- 39. 6. Optimost declares the Best Recipe This Version Increases Original Version Conversions by 28.4% 54
- 40. 7. REPEAT 55
- 41. Summary – MVT Allows you to: Listen to your Customers Improve Revenue… consistently Avoid Cost and Risk of Redesigns 57
- 42. Thank You! Greg Kelton gkelton@interwoven.com 58