Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014
3 likes1,735 views
Haymarket Media Group is establishing a testing culture to improve user experience through analytics and optimization, led by Jon Warden and Shruti Ahuja. They have been using Optimizely to conduct A/B tests and have learned valuable lessons about the testing process, including the importance of simplicity and stakeholder communication. Moving forward, they aim to create a dedicated optimisation team and embrace a mindset that prioritizes continuous improvement and learning from both successes and failures.
Case Study by Jon Warden and Shruti Ahuja of Haymarket - Optimizely Experience London 2014
- 1. HOW HAYMARKET IS ESTABLISHING ! A TESTING CULTURE Jon Warden Head of User Experience ! Shruti Ahuja Head of Analytics and Optimisation
- 2. Jon Warden Head of User Experience! Haymarket Media Group ! uk.linkedin.com/in/jonwarden/! @jonwarden My background! 10 years at News UK in various design & UX roles 5 years as a UX consultant, freelancing at various organisations Haymarket to date: Initial set up, and 2 years management of internal UX team
- 3. Shruti Ahuja Head of Analytics and Optimisation! Haymarket Media Group ! uk.linkedin.com/in/shrutiahuja/ My background! 8 years experience in Analytics and Optimisation across Media, eCommerce, CE, Automotive, Travel and Financial services Haymarket to date: Develop analytical infrastructure across 20 B2C brands Haymarket this year: Expand Analytics to B2B brands and strengthen Optimisation structure for B2C brands
- 4. “One accurate measurement is worth a thousand expert opinions” – Grace Murray Hopper, Computer scientist and US Navy Admiral
- 5. Who are Haymarket Media Group?
- 6. AN INTERNATIONAL PUBLISHING HOUSE We create award winning specialist content for international audiences • 72 market leading brands • 20 offices • Across six countries • 1500 + staff • Business Media division • Consumer Media division • For Print, Digital, Live Media and events
- 7. AN INTERNATIONAL PUBLISHING HOUSE Some of our digital brands
- 8. AN INTERNATIONAL PUBLISHING HOUSE Some of our digital brands
- 9. AN INTERNATIONAL PUBLISHING HOUSE Some of our digital brands
- 10. Our journey with Optimizely
- 11. WITH OPTIMIZELY SINCE MAY 2013 • We started off on a month-by-month Gold package ! • We completed very basic experiments on: Whatcar.com PistonHeads.com CaravanSiteFinder.com ! • We quickly found we were regularly hitting our traffic allocation • In April 2014 we upgraded to Platinum
- 12. WHY WE UPGRADED • We needed dedicated support • We needed unlimited projects, user accounts, and more features to use on all our brands • Other areas of the business were interested in using Optimizely, including international divisions • We now have dedicated accounts for: Haymarket Business Media! Haymarket Consumer Media ! PistonHeads.com
- 14. OPTIMISATION & BUSINESS PRACTICE Each company has their own process for A/B testing Test report created Success/fail Assumption/ hypothesis reported to Product team Publisher/feature request Business criteria UX / UI / FE create experiment Analytics reports Test runs for 2 weeks minimum Product / UX / Analytics define test No Yes Test report created Sent to business for approval Budget agreed to create dev Pushed to live work for sprint Test fail
- 15. OPTIMISATION & BUSINESS PRACTICE Each test goes on the testing road map
- 16. OPTIMISATION & BUSINESS PRACTICE Each test has a reporting document and test ID
- 17. OPTIMISATION & BUSINESS PRACTICE • The reports
- 18. OPTIMISATION & BUSINESS PRACTICE • The reports
- 19. OPTIMISATION & BUSINESS PRACTICE • The reports
- 20. IN AN IDEAL WORLD… • Over all our process is a good one, but… • We’ve found that we’ve had a few issues • Maintaining focus, and learning is difficult, we’re a busy group • We don’t have a dedicated Optimisation team, yet! • Everyone has their own day jobs, business, analytics, product, ux, editorial • Resourcing - we have small analytics and UX teams • Business see the validity, but it isn’t business critical in certain disciplines • Getting budget released or time for iteration is difficult due to dev roadmap
- 21. SOLUTIONS • We’re trying to change the mindset and the culture • We need to create an Optimisation process ‘all’ buy into • We’ve enlisted the help of Optimizely experts to help us • Getting onsite training for the team • Discussing testing strategies with senior management • In the past we’ve hired a third party CRO agency to get more focus • We need to create an in house dedicated team to focus efforts
- 22. Experiments - what we’ve learnt
- 23. WHAT WE’VE LEARNT 1. Do - ‘Start simple’, until you know what you’re doing 2. Do - Create as many tests as possible, and run them consecutively 3. Do - Measure the numbers, and then learn about the results 4. Do - Leave experiments to run as long as needed 5. Do - Embed a testing culture into your business 6. Do - Ask for help to learn more
- 24. WHAT WE’VE LEARNT 1. Don’t - run before you can walk, but learn as you go 2. Don’t - try experiments with too many goals 3. Don’t - create A/B and MVT tests with too many variants at first, without having the proper knowledge 4. Don’t - launch multiple experiments on the same site, at the same time, they have a tendency to clash with each other, causing experiments to fail 5. Don’t - forget to inform stakeholders you’re running tests
- 25. WE’VE SIMPLIFIED OUR TESTS DOWN, ! AND HAD SUCCESS • We recently created an experiment on one of our sites • Using two styles and four variants Variant A Variant B Variant C Variant D
- 26. WE’VE SIMPLIFIED OUR TESTS DOWN, ! AND HAD SUCCESS • The results were astounding! • Variant D had an uplift of +197.6%
- 27. PISTONHEAD: ANALYSIS-LED OPTIMISATION Shruti Ahuja Head of Analytics and Optimisation
- 28. ABOUT PISTONHEADS.COM • Connects car enthusiasts to dealers and one another • An automotive classifieds platform • One-stop car shop
- 29. HOW DID WE CREATE OUR OPTIMISATION ROADMAP?
- 30. WHAT IS OUR OPTIMISATION PROCESS? Insights Generation Roadmap Creation Test Learning Results Implementation Analysis
- 31. WHAT DETERMINES OUR TESTING ROADMAP? • Identify KPI’s that support the Business Model • Investigate issues that hinder KPI conversion • Generate ideas to test and improve performance on an on-going basis !!!!!! Visitors !!!!!! Inquiries !!!!!! Leads !!!!!! Sales
- 32. WHAT INSPIRES OUR TEST IDEAS? • Monthly dashboard reviews • Ad-hoc analysis • Product related issues
- 33. OUR MOST SUCCESSFUL TEST Objective:! Determine if introducing a new layout with an inbuilt Contact Seller form and a large image increases site conversion ! Hypothesis:! Currently users have a 2-step process to contacting sellers. The simplification of this and introduction of larger images will improve interaction with the ad details pages ! Test Description:! A/B test. Control (Existing content: 2 step form & small images), Alternate B (In-page form, in-page images) ! Success Metrics:! • Traffic • Retention • Contact Seller Conversion • Revenue
- 34. OUR MOST SUCCESSFUL TEST
- 35. WHEN TESTS FAIL… Objective:! Understand how user behaviour is affected by using a car like VRM on the parts and plates landing page. ! Hypothesis:! The current parts and plates landing page design mirrors the old style SSI landing page. The new VRM form style has been seen to perform better for cars. This test will confirm that the design works for parts and plates too. ! Test Description:! A/B test. Control, variation B - car-like VRM ! Success Metrics:! Proceed to next step Ad creation conversion rate
- 36. WHEN TESTS FAIL…
- 37. WHAT WE LEARNT • Optimisation is a learning process • Never assume / preempt • You win some, you lose some • In either instance, test further!!