Optimizely & Photobox - DON'T PANIC: The No-Confusion Experimentation Startup Guide
Download as PPTX, PDF0 likes281 views
The document provides a guide for businesses to start experimentation without confusion, emphasizing a structured approach to optimization as if it were a startup. It advocates for developing a mindset for experimentation, enhancing team capabilities, and employing a model to assess success while pivoting as needed. The key takeaway is to treat experimentation like a mini-business, focus on enabling teams, and continuously measure and adapt strategies for improvement.
Optimizely & Photobox - DON'T PANIC: The No-Confusion Experimentation Startup Guide
- 1. Ellie Hughes DON’T PANIC The no-confusion experimentation startup guide Twitter: @EllieExperiment medium.com/@EllieExperiment
- 2. PROBLEM: You’re a business who want to start experimenting, but don’t know where to start: - Where should you invest your time, money or resources first? • In these cases, panic often sets in - this leads to a scatter gun approach to trying to get experimentation going: - Let’s buy a tool - Let’s pay an agency - Let’s hire a specialist - Let’s add more tools or get rid of an existing tool - Let’s get everyone in the business to come up with ideas PANIC!!
- 3. SOLUTION: This talk! Grab your towel and listen to a guide who has done the hitch-hiking so you don’t have to: • Advice on approach • A strategy to get started - Mindset - Enablers • Finally, how to assess how things are going DON’T PANIC!!
- 4. 1. Think of optimisation like a mini-business or services startup within your business: whose customers are the product and tech teams, commercial, marketing, the leadership team…… anyone who could be more successful if they can run experiments. 2. Develop your optimisation startup like a real business: - Hire a great CEO: get a CRO owner with vision and expertise - They develop & rollout a strategy: give them ownership of the programme and strategy - Invest: get the tools, people and skills they need to make the strategy successful - Run and assess: Let the programme run and see what is working - Double-down or pivot: keep doing what makes your customers happy, and fix the things that don’t work Advice
- 5. • Build a Mindset for experimentation: getting everyone to think about experiments constructively - The How, Why and What of Experimentation • Utilise a Model to enable experimentation: - Across the experiment lifecycle try different approaches and ways to enable experiments - Measure the results • Quick aside: The structure of your team doesn’t matter… What is my strategy for my experimentation startup?
- 6. Mindset Experiments consist of 3 key variables or decisions: ‣ What you are testing: Changes you think will impact your KPI ‣ Why you are testing: The impact you want to make on your KPI/metric or goal ‣ How you will test that effectively and rigorously: - Process/Planning - Ideation - Execution - Analytics - Value - … many more The How, Why and What of Experimentation
- 7. Mindset In many teams, the decision about what to test often comes first They are employing the following strategy: The How, Why and What of Experimentation: As-is
- 8. Mindset In many teams, the decision about what to test often comes first They are employing the following strategy: The How, Why and What of Experimentation: As-is • What phase: The team are doing ‘feature-led’ testing, where they decide what to test and then justify why they are testing it. • Why phase: the team look around for KPIs to support their pre-selected idea: And so increase conversion… • How: Finally, they work out how to test their feature, or whether it is even possible/feasible to run the experiment. This happens well after choosing the experiment.
- 9. Mindset How: Do we find out customer behaviour and problems from our metrics? Do we ideate experiments? Do we plan experiments? How do we do experimental design? Do we build experiments? Do we test statistically and analytically? Do we design the feature? Do we know an experiment has won? In actual fact, this hierarchy needs to be inverted: - Enable the team by teaching the principles of How to Test, which leads naturally into the Why and What to Test. The How, Why and What of Experimentation: To-be
- 10. Mindset • After the How, it becomes clear which KPIs can be influenced and whether these are actually important: The Why. • This leads to a growing number of much better ideas: The What. • The correct training and enablement free the team to be more autonomous and agile with their experiments. In actual fact, this hierarchy needs to be inverted: - Enable the team by teaching the principles of How to Test, which leads naturally into the Why and What to Test. The How, Why and What of Experimentation: To-be
- 11. Enablement This is designed to show how we can inject the How Why and What into our ways of working, and help us diagnose where this is going wrong. The enablement model Having got the right mindset, we then need to do the second half of our strategy: Utilise an enablement model across the experiment lifecycle
- 13. Team Goal Impacted Team is successful and Healthy @ellieexperiment Business is Successful
- 14. Team Goal Impacted Team is successful and Healthy EXPERIMENT DESIGN SKILLS/ TRAINING ANALYSIS TECHNOLOGYEXECUTION IDEATION @ellieexperiment Business is Successful
- 15. @ellieexperiment
- 16. @ellieexperiment
- 17. Enablement • Lack of Impact on goal over time or over many experiments • Poor team health • Using experiment Retros How do you diagnose a failure of How, Why, What?
- 18. Enablement • An experiment retrospective looks back at what happened during the experiment, and what the team did and how well they did it. • Make it quantitative! What is an experiment Retro? • Ask a lot of questions: - How did the experiment go? - Was it a winner? - Did it impact their goal/KPI? If not did it impact there metrics instead? - Where did they feel it went wrong? - What knowledge and skill gaps were revealed? - What processes need to change for next time? - ….
- 19. Enablement • Using a simple spider scorecard, I work with our teams to reflect where they need additional help • For all the areas where they score 1 out of 5, we focus to bump that up to a 2 or 3. Diagnose a failure of How, Why, What
- 20. Enablement Diagnose a failure of How, Why, What • Low Maturity: The team biases towards their comfort zones: technical skills and execution/delivery. @ellieexperiment
- 21. Enablement Diagnose a failure of How, Why, What • Low Maturity: The team biases towards their comfort zones: technical skills and execution/delivery. • Medium Maturity: The team starts to focus more on analytics learnings and quality of ideation, leading to more impact on their goal. @ellieexperiment
- 22. Enablement Diagnose a failure of How, Why, What • Low Maturity: The team biases towards their comfort zones: technical skills and execution/delivery. • Medium Maturity: The team starts to focus more on analytics learnings and quality of ideation, leading to more impact on their goal. • High maturity: The team invests heavily in training (whole team), analysis & product insights, and experiments with planning/process models and ideation techniques @ellieexperiment
- 23. Enablement Diagnose a failure of How, Why, What So what? The retro shows: - Specifically how the team are struggling - Where the “How, Why, What” has failed - How they feel about it - The impact on their goal Now what? - Use the enablement model, and invest in the lagging areas.
- 24. Hopefully this has made it easier not to panic! In Summary: - Treat optimisation like a start-up inside your business - Get the experimentation mindset right - Enable experimentation by enabling the teams running experiments - Measure and fix! Need more ideas and tools you can use to improve experimentation, or want to try them out? - come talk to me! - checkout my blog post on medium Twitter: @EllieExperiment Medium.com/@EllieExperiment DON’T PANIC!!
- 25. Q&A
- 26. Thank you!
- 27. Session Exit Slide (Do Not Remove) Up next… ▪ Announcement 1 ▪ Announcement 2 ▪ Announcement 3
Editor's Notes
- #3: A common problem I find when I join a business who want to start experimenting, is that they do not know where to start: Where should they invest their time, money or resources first? This is an experience which can be incredibly costly: I once started at a B2B travel provider who spent >£200k/yr on their analytics suite, but only had 5000 active users. In these cases, panic often sets in - this leads to a scatter gun approach to trying to get experimentation going: Let’s buy a tool Let’s pay an agency Let’s hire a specialist Let’s add more tools or get rid of an existing tool Let’s get everyone in the business to come up with ideas The typical comments in the business as a result are: We Built lots of tests but don’t seem to be getting anywhere We aren’t building any tests or not fast enough We aren’t able to analyse our tests We don’t have enough ideas or we have too many! The literature provided by most AB testing vendors, agencies and consulting firms doubles the confusion: An inspirational aspirational pitch deck about your future optimisation maturity and growth A survey about your business or optimisation maturity A plethora of optimisation maturity models (saying almost but not quite the same thing?)
- #4: This talk is designed to give you some help and advice to avoid the mistakes many businesses make. So, grab your towel and listen up to a guide who has done the experimentation hitch-hiking so you don’t have to!
- #5: Don’t undermine your “CEO” by buying a tool or bringing in an agency/consultants before they join: this gives them less ownership of the strategy and they may have recommended a different/better approach.
- #6: The type of team you run is actually irrelevant, so long as you get the MINDSET and ENABLEMENT right. Go for squads, feature teams, product teams, marketing/commercial teams with a specialist, 1 internal person running an outsourced agency team… I am not going to sell you some magic beans and make you build a bean stalk too So first up: How, Why and What
- #8: In many teams, the decision about What to test often comes first, especially as it is the variable over which the team has the most control.
- #9: In many teams, the decision about What to test often comes first, especially as it is the variable over which the team has the most control.
- #10: By Understanding the methods and constraints of experimentation, and so success and failure, this gives product teams the right mindset. Laying that ground work gives teams the ability to “play” in the space, getting an intuitive idea of the KPIs to test (Why) and therefore What to Test (impacting their KPI). After the How, it becomes clear which KPIs can be influenced and whether these are actually important: do they indicate a customer behaviour we want to change or problem to fix. This leads to a growing number of much better ideas: The What. The correct training and enablement free the team to be more autonomous and agile with their experiments.
- #11: By Understanding the methods and constraints of experimentation, and so success and failure, this gives product teams the right mindset. Laying that ground work gives teams the ability to “play” in the space, getting an intuitive idea of the KPIs to test (Why) and therefore What to Test (impacting their KPI). After the How, it becomes clear which KPIs can be influenced and whether these are actually important: do they indicate a customer behaviour we want to change or problem to fix. This leads to a growing number of much better ideas: The What. The correct training and enablement free the team to be more autonomous and agile with their experiments.
- #13: Business is Successful Team Goal Impacted Team Successful and Healthy Ideation Experimental design Execution Technology Training Analysis
- #14: Business is Successful Team Goal Impacted Team Successful and Healthy Ideation Experimental design Execution Technology Training Analysis
- #15: Business is Successful Team Goal Impacted Team Successful and Healthy Ideation Experimental design Execution Technology Training Analysis
- #19: Running a retro is common practice for a lot of teams Used in agile/scrum to check team health and improve the way the team works An experiment retrospective looks back at what happened during the experiment, and what the team did and how well they did it. Make it quantitative!
- #25: This talk is designed to give you some help and advice to avoid the mistakes many businesses make. So, grab your towel and listen up to a guide who has done the experimentation hitch-hiking so you don’t have to!