Beyond Simple A/B testing
Download as PPTX, PDF0 likes190 views
The webinar covered various topics related to optimizing an A/B testing program including improving program performance over time using metrics, supporting innovation through processes and methodology, developing strong research and experiment strategies, and evaluating technology and team capabilities. Key points included using metrics like test capacity, velocity, win rate to monitor program quality; implementing processes like an experiment charter and centralized project management; developing hypotheses grounded in qualitative and quantitative customer research; and ensuring the team includes strategists, analysts, designers, copywriters and developers. The webinar provided advice on optimizing an A/B testing program through monitoring performance, standardized processes, rigorous research-based experimentation, and building a cross-functional team.
Beyond Simple A/B testing
- 1. ratiopartners.co.uk 1 Webinar – Beyond simple A/B testing
- 2. Topics in todays webinar ratiopartners.co.uk 2 • ‘Optimising Optimisation’ or how to improve your program performance over time • Supporting innovation with Processes and Methodology • The types of research that will deliver the best experiment hypothesis & strategy • The capabilities (people) needed to excel • Technology and evaluating the suitability of a product for your organisation
- 3. Topics in todays webinar ratiopartners.co.uk 3 • ‘Optimising Optimisation’ or how to improve your program performance over time • Supporting innovation with Processes and Methodology • The types of research that will deliver the best experiment hypothesis & strategy • The capabilities (people) needed to excel • Technology and evaluating the suitability of a product for your organisation
- 4. ratiopartners.co.uk 4 There’s a natural tendency to focus on the outputs of Optimisation… Like increases in Conversion Rate, AOV, CLV, Customer satisfaction, repeat purchase, etc… Topic 1: Optimising Optimisation
- 5. ratiopartners.co.uk 5 But are these outputs levers that we can pull to drive more success from our program? Will they tell us what we are doing well or poorly, or where we are going wrong? Not really..
- 6. ratiopartners.co.uk 6 A best in class optimisation program needs a set of performance metrics that will help us to understand where the weak points are in our program as a whole, not just report on how any one individual experiment happened to perform
- 7. Metric 1: Test Capacity ratiopartners.co.uk 7 The more effective experiments we run, the more results our program will deliver.. Test capacity helps us to understand the upper limit of how many tests we can run
- 8. Metric 1: Test Capacity ratiopartners.co.uk 8 52 Average test duration ( in weeks) # of simultaneously testable funnels or pages
- 9. Metric 1: Test Capacity ratiopartners.co.uk 9 52 5 5 Average test duration ( in weeks) # of simultaneously testable funnels or pages 52 tests (annual capacity)
- 10. Metric 2: Test velocity ratiopartners.co.uk 10 Where experiment capacity showed how many experiments can be run, experiment velocity is a measure of how many experiments have been run It helps us understand how well we have capitalised on our opportunities 0 1 2 3 4 5 6 7 Tests Program velocity tracker Test Velocity Test Capacity
- 11. Metric 3: Zero test tracker ratiopartners.co.uk 11 Traffic that doesn’t see a test is a missed opportunity – permanently. To help encourage a culture of “Always be testing” and minimise lost opportunities, track how many days it’s been since your teams last day with zero experiments live. 23 days Since the last “Zero experiments” day
- 12. Metric 4: Win rate ratiopartners.co.uk 12 As well as monitoring and increasing quantity of experiments delivered, it’s crucial to understand the quality of those tests Monitoring test win rate keeps our finger on the quality pulse 0% 20% 40% 60% 80% Test Win Rate tracker Win rate Inconclusive Rate Loss rate
- 13. ratiopartners.co.uk 13 Processes and methodology are a key lever in your quest to improve experiment velocity Improving the processes that sit around your experimentation will help you deliver more output from finite resources Simply put, robust processes support rather than stifle innovation Topic 2: Process and Methodology
- 14. ratiopartners.co.uk 14 An Experimentation charter is a document which lays out in explicit detail why and how the optimisation program will be run Amongst other things it clearly defines: • Roles and responsibilities • Experimentation process from ideation to go live • Which stakeholders are involved at each step and what their committed turn around time is Example 1: Experimentation Charter
- 15. ratiopartners.co.uk 15 Centralise planning & communications around the CRO project using a PM tool, even if it’s a free one like Trello. Build out a workflow that matches the process agreed in your optimisation charter Example 2: Using a centralised PM tool
- 16. ratiopartners.co.uk 16 A thorough QA checklist will help you to minimise the amount of time each experiment will spend in QA as well as minimise mistakes made through human oversight Example 3: QA checklist QA Checklist Goals implemented and triggering correctly Only the correct audience can enter test Functionality/design works across key browsers Functionality/design works across key device types New links all point to working pages Additional GA tracking implemented
- 17. ratiopartners.co.uk 17 Example 4: Post test analysis & knowledge repository Conduct meaningful post test analysis, communicate it within the organisation & create a system to record this information for posterity. This will maximise the future value derived from each completed experiment Post test analysis What was the hypothesis? What pages/touch points was it executed across? Did it win or lose? What behaviours were Were there any other insights Did any customer segments over or under perform? Did VOC tools provide any additional insight?
- 18. ratiopartners.co.uk 18 Example 5: Qualitative research Qualitative research is an important step in developing hypothesis that deliver results. Try to standardise the way you collect this information and optimise your questions/process over time to help isolate the questions which inspire the most helpful information from customers What was your reason for visiting today? Option A Option B Option C Option D (Free text) What was your reason for visiting today? (free text input) 0.25% response rate 1% response rate
- 19. ratiopartners.co.uk 19 If your optimisation program is to deliver meaningful changes in user behaviour, you need to make meaningful changes to their user experience, with hypothesis grounded in qualitative and quantitative data Move beyond button colour testing! Topic 3: Research and Strategy
- 20. ratiopartners.co.uk 20 Developing hypothesis the right way The best test hypothesis will have a grounding in both the qualitative and quantitative In general, digital analytics data can tell you “what” is happening, qualitative data will give you insights as to “why” it happened
- 21. ratiopartners.co.uk 21 Digital analytics Digital analytics data shines at sizing the opportunity – for example, “where is the biggest drop out point in the purchase funnel for mobile users”? If you don’t have a “traditional” funnel, try to think about the objective of the page you’re looking at and what metrics might help you evaluate possible headroom for optimisation Landed on Homepage Category listing Product Page 18% Purchase 75% 80% 82% 25% 20% Responsive Funnel
- 22. ratiopartners.co.uk 22 Digital analytics Clicked hamburger nav Opened top level category in nav 35% 65% Landed on Homepage Implemented well, web analytics products can also help you to isolate where within that funnel step users are having the most problems But at this point, we still don’t know what the problem is and developing a hypothesis with no other information will mean we are in danger of applying our own assumptions/biases 80% 20% Click through to category 80% 20%
- 23. ratiopartners.co.uk 23 Voice of customer ‘VOC’ tools will help you understand what it is about that step that users struggle with For example, session replays or user testing might reveal that users do not understand the category links in the navigation are clickable
- 24. ratiopartners.co.uk 24 Useful VOC tools and common use cases • User testing • Test new designs on users before building them out and putting them into production • Interviews with sales / customer service staff • Understanding customer frustrations with website • Understanding common customer objections and how they are overcome • Surveys (email/on site) • Identify visitor goals and reasons for non completion • Uncover the reasons that customers buy to communicate to others • Visitor replay technology • See the specific challenges users are having and how they are interacting in real time • Heat/scroll maps • Identify if users are missing key information within the page due to position
- 25. ratiopartners.co.uk 25 A cautionary tale - how not to approach optimisation & the danger of HIPPOs
- 26. ratiopartners.co.uk 26 CEO The navigation menu is our priority! I think it’s a big opportunity Head of marketing The navigation menu looks dated & is not on brand Head of Product It doesn’t surface all of our categories
- 27. ratiopartners.co.uk 27 Adventure Creative Driving Flying Getaways Gourmet Indulge Adventure subcategories Subcategory 1 Photo/link Subcategory 2 Photo/link Subcategory 3 Photo/link Subcategory 4 Photo/link Subcategory 5 Photo/link Subcategory 6 Photo/link (2 months of design & development later)
- 28. ratiopartners.co.uk 28 But the test lost, and we had no demonstrable idea why, just more of our own assumptions…
- 29. ratiopartners.co.uk 29 Some of the problems with our approach Our hypothesis sucked - Or rather, there wasn’t one! The decision to rebuild the navigation was based on Highest paid persons opinions and not a data driven hypothesis
- 30. ratiopartners.co.uk 30 Learning: Good test hypothesis should set out to solve actual customer problems, highlight or clarify your value proposition, reduce anxiety or introduce urgency At a minimum, we should have set metrics to understand how the existing nav performed to validate “headroom” for improvement existed “I can’t find pricing information” “It’s not clear why I would take your service over a competitors” “How do I know that your organisation can be trusted with my money?” “I don’t know what this sizing means” “I cant see how to buy it”
- 31. ratiopartners.co.uk 31 Some of the problems with our approach Our hypothesis sucked - Or rather, there wasn’t one! The decision to rebuild the navigation was based on Highest paid persons opinions and not a data driven hypothesis We didn’t quantify the opportunity or prioritise this hypothesis against any others
- 32. ratiopartners.co.uk 32 Learning: Fundamentally, test hypothesis should be prioritised based on expected return vs expected effort
- 33. ratiopartners.co.uk 33 Some of the problems with our approach Our hypothesis sucked - Or rather, there wasn’t one! The decision to rebuild the navigation was based on Highest paid persons opinions and not a data driven hypothesis We didn’t quantify or prioritise this hypothesis against others We had no way of validating what behaviour was occurring with the new navigation – back to our own assumptions to interpret the results
- 34. ratiopartners.co.uk 34 Learning: Make sure you are thinking about what tracking you will need to maximise test learnings and iterate on your hypothesis It needs to be in place before the test goes live!
- 35. ratiopartners.co.uk 35 Research/Strategy takeaways: • Use digital analytics data: • To quantify and prioritise the key drop out points within your funnel • To identify possible drivers of this drop out by looking at in page interactions • Use customer research to: • Identify customer pain points & UX barriers • Identify customer objections and the persuasion elements that can help you overcome same • Identify what actual customers view as your value proposition to inform how you communicate to prospective customers • When developing hypothesis: • The most powerful hypothesis will come from both qualitative + quantitative understanding • At the most fundamental level, prioritise your hypothesis based on estimated impact and effort to execute • When executing: • Make sure you have enough tracking in place that you can get a complete understanding of the behaviours your test influenced, win or lose
- 36. ratiopartners.co.uk 36 The Strategist • Strong understanding of the business context • User research skills to uncover the qualitative gold • Can integrate disparate data sources (qualitative + quantitative) and formulate hypothesis • Must be data driven (identify the problems) and creative (ideate the solutions)
- 37. ratiopartners.co.uk 37 The Analyst • Very strong web analytics platform knowledge – using it for deep dive analysis of customer journeys, not just reporting • Can identify the patterns and stories behind the numbers • Good command of statistics (statistical significance & power, probability, sample sizes etc)
- 38. ratiopartners.co.uk 38 The Designer • Translates test hypothesis into designs • Strong understanding of persuasive design principles • Understanding of responsive design principles
- 39. ratiopartners.co.uk 39 The Copywriter • Translates user research findings in to persuasive copy which adheres to brands tone and style • Understanding of the conversion funnel/buyer journey and has an ability to write according to funnel stages • Familiarity with industry specific language • Comfortable with SEO principles
- 40. ratiopartners.co.uk 40 The Developer • Modern testing platforms allow a lot to be done without development, but at some point you will start to exhaust these capabilities • Look to bring in a front end developer who is very strong at HTML/css/javascript to execute on more complex design changes • They should be comfortable with delivering an ‘MVP’ product to validate the hypothesis is valid
- 41. ratiopartners.co.uk 41 Design Front end Team member BTeam member A Data Strategy Copy- writing • There can certainly be overlap between these roles, particularly when the team is just starting out. But it’s important that they are all present in some way!
- 42. Questions to help evaluate technology vendors ratiopartners.co.uk 42 Is your priority optimisation of individual touch points or personalisation of the entire customer journey? If your priority is personalisation, can the technology link together behaviour between different touch points & devices over multiple visits and use this knowledge to drive personalisation? If your priority is optimisation, how much flexibility will the solution give you to alter design and layout of pages? Will it allow the optimisation project to be abstracted from existing development efforts? (release cycles, development resources, etc) Can the technology help you identify opportunities? Eg, through integrated VOC testing tools (VWO) or Machine Learning (Sitecore) Can the technology provide you with any other customer intelligence? Do you have other data which could enrich personalisation (eg CRM) and if so how readily can this be integrated with the testing platform?