Optimism Webinar 2 - Moving from AB testing to true experimentation
1 like591 views
The webinar presented by Lisa Rohlfs outlines the transition from A/B testing to a comprehensive culture of experimentation within organizations. It emphasizes the importance of creating a structured strategy, building a dedicated team, and integrating business metrics to drive innovation and growth. The session also highlights best practices for executing experiments, analyzing results, and fostering a collaborative team environment to enhance experimentation efforts.
![17
Example testing charter/Onboarding Guide
Team Culture Strategy Technology
CompanyName Testing Methodology
Introduction & Program Goals
[Add an introduction paragraph]
β’β Problems: The Optimizely testing platform was purchased to help us [describe the rationale for purchase here]
β’β Outcomes & Success: Our testing program will be successful if we are able to [describe what success looks like
here]
β’β This Document: This document is the source of truth for our organization when it comes to all things related to
our testing program. If you have additional suggestions for it please donβt hesitate to edit this document or let
[Testing Program Managerβs Name] know so it can be added to this page.
β’Team
β’β Executive Sponsor - [name and email]
β’β Program Manager/Power User - [name and email]
β’β Primary Developer - [name and email]
β’β Primary Designer - [name and email]
β’β Testing QA Lead - [name and email]
β’β [add other team members as needed]
β’Process
β’[Add an introduction paragraph describing the testing process generally.]
β’[Example image]
β’[Add your steps from your process below]
β’β Brainstorm: [who, where, why, what, when, how]
β’β Prioritization: [who, where, why, what, when, how]
β’β Documentation & Design: [who, where, why, what, when, how]
β’β Setup: [who, where, why, what, when, how]
β’β QA & Approval: [who, where, why, what, when, how]
β’β Run: [who, where, why, what, when, how]
β’β Analysis & Review: [who, where, why, what, when, how]
β’β Results Sharing: [who, where, why, what, when, how]
in the testing team and their respective answers]
β’Meetings & Meeting Cadence
β’[You may include this as part of your process described above or you may note it here separately.]
β’Test Velocity
β’The aim is to run at least [##] tests per [timeframe]. However testing quality is important as well and we should aim
to learn something from every test. Weβll start with this initial goal but testing velocity may [increase/decrease]
depending on our teamβs agility and itβs goals for any given quarter.
β’Test Plan & Results Templates
β’Each test plan should include the following elements and be based off [link to a downloadable template].
Consistently documenting the tests we run so we can have quarterly summary meetings and effectively learn from
our tests. We aim to have a strong testing results archive.
β’Links Key
β’β Roadmap
β’β Templates
β’ β Test Plan
β’ β Test Results
β’β Results Archive
β’β Goals Definitions
β’β Targeting Definitions
β’β Audience Definitions
β’β Optimizely Knowledge Base
β’β Optimizely Community
β’β Optimizely Learning Academy
β’β [add other items as needed]
β’
β’Other Guidelines and FAQs
β’β [list general guidelines for the testing program]
β’β [list common questions for people](https://image.slidesharecdn.com/optimismwebinar2-movingfromabtestingtotrueexperimentationlr-190122161556/85/Optimism-Webinar-2-Moving-from-AB-testing-to-true-experimentation-17-320.jpg)
Optimism Webinar 2 - Moving from AB testing to true experimentation
- 1. Moving from A/B testing to true experimentation Lisa Rohlfs β’Manager, EMEA Customer Success
- 2. 2 β’ Webinar will be recorded and shared along with the slides β’ Q&A β Use your control panel to ask questions Housekeeping
- 3. Create a culture Adopt technology Design a strategy Build a team
- 4. 4 BUSINESS VALUE VELOCITY/VOLUME LEVEL 1 Executional Start LEVEL 2 Foundational Growth LEVEL 3 Cross-Functional Advancement LEVEL 4 Operational Excellence LEVEL 5 Culture of Experimentation
- 5. 5 BUSINESS VALUE VELOCITY/VOLUME LEVEL 1 30% LEVEL 2 45% LEVEL 3 20% LEVELS 4 & 5 5%
- 6. 1. Discover a framework to evaluate your journey so far. 2. Gain actionable insight to move your program to the next level. 3. Create the culture of experimentation that drives innovation β and powers growth.
- 7. 7 Maturity Model Categories Team Culture Strategy Tech
- 8. 8 Why do we aim for balance?
- 9. 9 Moving to the next Level
- 10. Level 2 β You can report on business metrics β Your Analytics are integrated with your testing platformTechnology β Standardized process across a team β Roadmap document that includes structured hypothesis & prioritization criteria β Experiments are more often targeted to specific personas/segments β Using indirect and direct data for ideation β Non-standardized process β Roadmap is a simple list of experiment ideas β Experiments are targeted infrequently to specific audiences β Sometimes leverages web analytics or past experimentation results for ideation Strategy β Management actively drives experimentation and leverages results for decision making β Teams experiment cross- functionally β Management is aware but not actively driving experimentation β More than one team in one part of the business is experimenting Culture β More than 1 full-time resource on experimentation, access to developers β Variety of roles involved β Less than 1 full-time resource on experimentation β Foundational roles in place Team Level 3
- 11. Level 2 β You can report on business metrics β Your Analytics are integrated with your testing platform Technolog y β Standardized process across a team β Roadmap document that includes structured hypothesis & prioritization criteria β Experiments are more often targeted to specific personas/segments β Using indirect and direct data for ideation β Non-standardized process β Roadmap is a simple list of experiment ideas β Experiments are targeted infrequently to specific audiences β Sometimes leverages web analytics or past experimentation results for ideation Strategy β Management actively drives experimentation and leverages results for decision making β Teams experiment cross- functionally β Management is aware but not actively driving experimentation β More than one team in one part of the business is experimenting Culture β More than 1 full-time resource on experimentation, access to developers β Variety of roles involved β Less than 1 full-time resource on experimentation β Foundational roles in place Team Level 3
- 12. 12 Team 1. Time spent on experimentation
- 13. 13 Increasing Time Spent on experimentation Team Culture Strategy Technology Link experimentation to existing strategic initiatives Existing reach
- 14. 14 Increasing Time Spent on experimentation Team Culture Strategy Technology Link experimentation to existing strategic initiatives Showcase experiment ROI Existing reach
- 15. 15 Culture 1. Team involvement 2. Management Awareness
- 16. 16 Getting more teams to experiment Analyze required Skill Sets Create internal Onboarding Guide Share Optimization Successes Team Culture Strategy Technology Expand reach
- 17. 17 Example testing charter/Onboarding Guide Team Culture Strategy Technology CompanyName Testing Methodology Introduction & Program Goals [Add an introduction paragraph] β’β Problems: The Optimizely testing platform was purchased to help us [describe the rationale for purchase here] β’β Outcomes & Success: Our testing program will be successful if we are able to [describe what success looks like here] β’β This Document: This document is the source of truth for our organization when it comes to all things related to our testing program. If you have additional suggestions for it please donβt hesitate to edit this document or let [Testing Program Managerβs Name] know so it can be added to this page. β’Team β’β Executive Sponsor - [name and email] β’β Program Manager/Power User - [name and email] β’β Primary Developer - [name and email] β’β Primary Designer - [name and email] β’β Testing QA Lead - [name and email] β’β [add other team members as needed] β’Process β’[Add an introduction paragraph describing the testing process generally.] β’[Example image] β’[Add your steps from your process below] β’β Brainstorm: [who, where, why, what, when, how] β’β Prioritization: [who, where, why, what, when, how] β’β Documentation & Design: [who, where, why, what, when, how] β’β Setup: [who, where, why, what, when, how] β’β QA & Approval: [who, where, why, what, when, how] β’β Run: [who, where, why, what, when, how] β’β Analysis & Review: [who, where, why, what, when, how] β’β Results Sharing: [who, where, why, what, when, how] in the testing team and their respective answers] β’Meetings & Meeting Cadence β’[You may include this as part of your process described above or you may note it here separately.] β’Test Velocity β’The aim is to run at least [##] tests per [timeframe]. However testing quality is important as well and we should aim to learn something from every test. Weβll start with this initial goal but testing velocity may [increase/decrease] depending on our teamβs agility and itβs goals for any given quarter. β’Test Plan & Results Templates β’Each test plan should include the following elements and be based off [link to a downloadable template]. Consistently documenting the tests we run so we can have quarterly summary meetings and effectively learn from our tests. We aim to have a strong testing results archive. β’Links Key β’β Roadmap β’β Templates β’ β Test Plan β’ β Test Results β’β Results Archive β’β Goals Definitions β’β Targeting Definitions β’β Audience Definitions β’β Optimizely Knowledge Base β’β Optimizely Community β’β Optimizely Learning Academy β’β [add other items as needed] β’ β’Other Guidelines and FAQs β’β [list general guidelines for the testing program] β’β [list common questions for people
- 18. 18 Reporting Experiment Wins/Losses Program Reporting/ROI Communication Team Culture Strategy Technology Create Management Awareness
- 19. 19 Communication Experimentation forums Newsletters Infographics Business Updates Team Culture Strategy Technology Create Management Awareness
- 20. 20 Communication Experimentation forums Newsletters Infographs Business Updates Management Awareness Reporting Experiment Wins/Losses Program Reporting ROI Team Culture Strategy Technology
- 21. 21 Strategy 1. Set Business Goals 2. Process best practices
- 22. Company Metric Business Unit Optimization Goals Experiment Goals Business Metrics vs Experiment Goals Team Culture Strategy Technology
- 23. 23 Integrating Business Metrics Revenue: $2.0m Revenue Per Visitor: $1.00 Average Order Value: $50.00 Average quantity: 2.00 Average per unit price: $25.00 Conversion Rate: 2.00% Add to Cart Cart Checkout Rate Visitors: 2.0m User acquisition: 1.5m User retention 33% Team Culture Strategy Technology
- 24. 24 Integrating Business Metrics Revenue: $2.0m Revenue Per Visitor: $1.00 Average Order Value: $50.00 Average quantity: 2.00 Average per unit price: $25.00 Conversion Rate: 2.00% Add to Cart Cart Checkout Rate Visitors: 2.0m User acquisition: 1.5m User retention 33% Define strategic program KPIs (company/team) Ways to leverage a goal tree Compare QoQ/YoY snapshots Identify metric βtrade-offsβ Team Culture Strategy Technology
- 25. 25 Process Best Practices Team Culture Strategy Technology
- 26. 26 Process Best Practices Direct data β Web analytics β Experiment results β Voice of the customer β Heatmaps β User personas β Usability tests β Customer decision model β Metrics Indirect data β Competitor sites β Best-in-class experiences β Blogs and webinars β User groups β Academic literature Leverage Data 01 Team Culture Strategy Technology
- 27. 27 Process Best Practices Strong Hypotheses 02 Team Culture Strategy Technology Follow the Hypothesis Framework
- 28. 28 Hypothesis Framework Team Culture Strategy Technology
- 29. 29 Process Best Practices Generate more ideas 03 Run an ideation workshop Team Culture Strategy Technology
- 30. 30 Ideation Workshop Team Culture Strategy Technology Post-Workshop 1. Compile ideas for a prioritized backlog/roadmap 2. Get attendees involved in execution of experiments 3. Keep everyone informed on progress & successes! Preparation 1. Invite different areas of the business 2. Set up whiteboards/virtual collaboration 3. Define area of website to ideate on (page- specific) Execution 1. Introduce hypothesis concept (*see slide 25) 2. Assign area of the website to the team 3. Identify all βuser problemsβ 4. Brainstorm possible solutions 5. Define metrics of success
- 31. 31 Ideation Workshop Team Culture Strategy Technology Post-Workshop 1. Compile ideas for a prioritized backlog/roadmap 2. Get attendees involved in execution of experiments 3. Keep everyone informed on progress & successes! Preparation 1. Invite different areas of the business 2. Set up whiteboards/virtual collaboration 3. Define area of website to ideate on (page- specific) Execution 1. Introduce hypothesis concept (*see slide 25) 2. Share area of focus on your website 3. Identify all βuser problemsβ 4. Brainstorm possible solutions 5. Define experiment metrics
- 32. 32 Ideation Workshop Team Culture Strategy Technology Post-Workshop 1. Compile ideas for a prioritized backlog/roadmap 2. Get attendees involved in execution of experiments 3. Keep everyone informed on progress & successes! Preparation 1. Invite different areas of the business 2. Set up whiteboards/virtual collaboration 3. Define area of website to ideate on (page- specific) Execution 1. Introduce hypothesis concept (*see slide 25) 2. Share area of focus on your website 3. Identify all βuser problemsβ 4. Brainstorm possible solutions 5. Define experiment metrics
- 33. 33 Ideation workshop outcomes Team Culture Strategy Technology Lots of new ideas Reach of experimentation
- 34. 34 Process Best Practices Run an ideation workshop Allow for easy submission of ideas Team Culture Strategy Technology Generate more ideas 03
- 35. 35 Idea Submission Team Culture Strategy Technology
- 36. 36 What β Prioritized experiments β Balanced testing schedule Roadmap Best Practices: What to include Team Culture Strategy Technology
- 37. 37 When β Timeline β Highlight interferences What β Prioritized experiments β Balanced testing schedule Roadmap Best Practices: What to include Team Culture Strategy Technology
- 38. 38 Who β Availability of resources β Dependencies When β Timeline β Highlight interferences What β Prioritized experiments β Balanced testing schedule Roadmap Best Practices: What to include Team Culture Strategy Technology
- 39. 39 ObjectivesWho β Availability of resources β Dependencies When β Timeline β Highlight interferences What β Prioritized experiments β Balanced testing schedule Roadmap Best Practices: What to include β Funnel coverage β Strategic theme Team Culture Strategy Technology
- 40. 40 Roadmap Example *template available here Team Culture Strategy Technology
- 41. 41 Process Best Practices Create & QA 01 Introduce re-usable templates (events, elements...) Create a QA checklist Team Culture Strategy Technology
- 42. 42 QA Checklist Team Culture Strategy Technology
- 43. 43 Process Best Practices Analysis 01 Be strict on statistical significance Team Culture Strategy Technology
- 44. 44 Process Best Practices Analysis 01 Be strict on statistical significance Understand why certain changes lead to your desired outcomes (wins) Team Culture Strategy Technology
- 45. 45 Process Best Practices Analysis 01 Be strict on statistical significance Understand why certain changes lead to your desired outcomes (wins) Analyse Inconclusive & Losing Tests Team Culture Strategy Technology
- 46. 46 Process Best Practices Analysis 01 Be strict on statistical significance Understand why certain changes lead to your desired outcomes (wins) Analyse Inconclusive & Losing Tests Document and share takeaways Team Culture Strategy Technology
- 47. 47 Results Sharing Team Culture Strategy Technology
- 48. 48 Targeting 1st & 3rd Party Data Location Based Targeting Demographic Targeting DCP Data Real Time Information Behavioral Targeting Team Culture Strategy Technology
- 49. 49 Technology
- 50. 50 Integrating Business Metrics Clarify the metrics you need to track Evaluate if and where you are able to track them Team Culture Strategy Technology Revenue: $2.0m Revenue Per Visitor: $1.00 Average Order Value: $50.00 Average quantity: 2.00 Average per unit price: $25.00 Conversion Rate: 2.00% Add to Cart Cart Checkout Rate Visitors: 2.0m User acquisition: 1.5m User retention 33%
- 51. 51 So, what can you do tomorrow?
- 52. 52 Watch your program mature! Leverage the recording of this session & our academy for guidance on activities Identify areas of opportunity Take the Optimizely Maturity Assessment (itβs free!)
- 53. 53 Webinar 3: Been there, done that. Learn how leading brands have taken their digital to new heights. Friday 25th January 2019 12:00 GMT / 13:00 CET
- 54. Thank you for listening. β’ Lisa Rohlfs Manager, EMEA Customer Success