Digital Data Tips Tuesday

Editor's Notes

• #2: Thanks a lot for having me here, I’m excited to give you some insights into how to build and/or improve the testing culture within your organisation to make sure it looks easy for you to keep on testing.
• #3: What I’d like to talk about is how we set up the culture around this, how we are able to scale this fast and how will of this help you in improving your testing program and it’s culture.
• #4: So I think with the a culture and in any testing program these 4 main topics are the most important. Having the right team in the structure and dynamics. Creating experiments and based on your prioritisation create a planning. Obviously after this you have to create them and analyse them.
• #5: So I think with the a culture and in any testing program these 4 main topics are the most important. Having the right team in the structure and dynamics. Creating experiments and based on your prioritisation create a planning. Obviously after this you have to create them and analyse them.
• #10: So I think with the a culture and in any testing program these 4 main topics are the most important. Having the right team in the structure and dynamics. Creating experiments and based on your prioritisation create a planning. Obviously after this you have to create them and analyse them.
• #11: So how does your team structure look like? What I see a lot is that it contains so many silos. 1 Project manager to make absolutely sure that office politics is handled. He or She comes up with an idea and runs it by legal, yes this happens, I bet some people here are also to scared to admit it ;-). They ask designers to create a prototype, preferably have 2 rounds of changes. Have it coded by their development team and have their QA person check the site. Total time to run through 1 test: probably around 3 weeks. But who can design, code and analyse? So you can save yourself some time? Let’s do a show of hands on that!
• #12: An overview of how a part of our TNW structure looks like. Design and Development are sitting next to each other and only marketing is sitting on different islands. Mostly because we just have too many people by now.
• #13: Our core team working on CRO consist of 4 people. They’re supported by 1 designer and 1 front-end developer who jump in whenever needed when we want to do stuff that goes above our heads in terms of knowledge.
• #14: Meet our culture, oh and these are our founders/ CEO + COO Boris and Patrick.
• #15: So in our case we turned around these ‘quotes’ as we didn’t have any of them. I don’t even know what the name of our laywer is. We have access right to our live servers to push winning variants live immediately. And in case of design reviews, it’s more that we can’t keep up with coding them then that we’re waiting for them, luckily!
• #16: So in our case we turned around these ‘quotes’ as we didn’t have any of them. I don’t even know what the name of our laywer is. We have access right to our live servers to push winning variants live immediately. And in case of design reviews, it’s more that we can’t keep up with coding them then that we’re waiting for them, luckily!
• #17: So how does your team structure look like? What I see a lot is that it contains so many silos. 1 Project manager to make absolutely sure that office politics is handled. He or She comes up with an idea and runs it by legal, yes this happens, I bet some people here are also to scared to admit it ;-). They ask designers to create a prototype, preferably have 2 rounds of changes. Have it coded by their development team and have their QA person check the site. Total time to run through 1 test: probably around 3 weeks. But who can design, code and analyse? So you can save yourself some time? Let’s do a show of hands on that!
• #18: Every presentation needs a quote by Tony Robbins. We highly believe that all the excuses you make belong to this problem. You can always find a solution for your problems whatever the issue is.
• #19: HIPPO, the Highest Paid Person in the Organisation. Worst case I saw is that in calculating the averages the HIPPO’s opinion had more weight. You gotta be kidding me!
• #20: Our HIPPO works in a different way, meet Boris again, our CEO. He obviously likes to share his opinion as well but does that in a different way. It happens at least monthly that he walks up to me with another crazy idea where we completely change the whole structure again. I rather have this over him prioritising our tests ;-)
• #21: So I think with the a culture and in any testing program these 4 main topics are the most important. Having the right team in the structure and dynamics. Creating experiments and based on your prioritisation create a planning. Obviously after this you have to create them and analyse them.
• #22: So how does your team structure look like? What I see a lot is that it contains so many silos. 1 Project manager to make absolutely sure that office politics is handled. He or She comes up with an idea and runs it by legal, yes this happens, I bet some people here are also to scared to admit it ;-). They ask designers to create a prototype, preferably have 2 rounds of changes. Have it coded by their development team and have their QA person check the site. Total time to run through 1 test: probably around 3 weeks. But who can design, code and analyse? So you can save yourself some time? Let’s do a show of hands on that!
• #23: So I have to admit we don’t have that really structured planning, we might want to have it. But as we can test on a quite fast pace it can be that all the new ideas that we come up with can be tested within a month. Which means at this point we prefer not to make our prioritisation process to hard on ourselves.
• #24: That’s where our ‘support troops’ come in handy. Our developers and designers can assist us wherever possible in finding us new improvements in what we could improve and also make sure it’s coded.
• #25: But what if you could have an almost unlimited number of tests running at the same time against at almost 0 costs.
• #26: That’s why creating experiments is easy, we can code and design them ourselves and only if it’s a really big change we try to involve our design and development team as well. Which makes it clear that we want good code, to make sure we can’t be blamed for hickups on the site. Which would also shoot us in the foot.
• #27: So I think with the a culture and in any testing program these 4 main topics are the most important. Having the right team in the structure and dynamics. Creating experiments and based on your prioritisation create a planning. Obviously after this you have to create them and analyse them.
• #28: Our technical setup makes this earlier quote very easy, all testing for now is ran in Google Tag Manager and supports both A/B testing and MVT testing. So we write the custom jQuery for creating our changes which any tool would do for us anyway. Want to do segmentation on a certain user segment? Sure, probably our dataLayer is already supporting this and sending the right variables for this to GTM. Where we like to improve? We want to be able to do this all from our backend, why? We want to have less flickering, more caching which improves speed and less JS that’s needed to be loaded.
• #29: So I think with the a culture and in any testing program these 4 main topics are the most important. Having the right team in the structure and dynamics. Creating experiments and based on your prioritisation create a planning. Obviously after this you have to create them and analyse them.
• #30: We decided to built a small platform where we could one 1 side keep a backlog of tests, move them to real testing documentation if time would come and also keep the data on metrics there. So we only needed 1 tool to keep track of the test and it’s documentation. Cutting out 2 other tools. And this is how it looks like as it retrieved the data for a ‘dummy’ experiment.
• #31: So I think with the a culture and in any testing program these 4 main topics are the most important. Having the right team in the structure and dynamics. Creating experiments and based on your prioritisation create a planning. Obviously after this you have to create them and analyse them.
• #32: It’s build on top of the Google Analytics API. As we send all our data for variants to custom dimensions in Google Analytics and data on all our KPIs with event tracking we have all the metrics in there already. Which makes the tool perfect to do automated reporting for us. Creating ultimate pleasure for the team as they don’t have to pull endless reports with the data and then start analysing it.
• #33: It’s build on top of the Google Analytics API. As we send all our data for variants to custom dimensions in Google Analytics and data on all our KPIs with event tracking we have all the metrics in there already. Which makes the tool perfect to do automated reporting for us. Creating ultimate pleasure for the team as they don’t have to pull endless reports with the data and then start analysing it.
• #34: Adding a new idea for a test is easily, we keep track of the dates of the test and the templates/ user segments (or swim lanes, thanks to Andy Nelson of Moz I learned that term) the test is running on. We choose an objective metric and if needed provide more details on the right side of the screen.
• #35: What’s next? We’re currently already growing the marketing team, which means we’ll get more traffic to test on, which means more testing. We might embrace cross functional a bit more. Currently we’re already working more with our developers and designers to make them part of the process and use their skills to fasten up the process. We might even get them on the marketing team later on to make ourselves cross-functional. For the analysis part we like to automate even more and there’s still a lot of improvement.
• #36: So the take aways: creating a great culture where people are taking full responsibility and not blaming silos. Have your whole team work on the process and make sure you completely understand the full process from design to analysis and back. Running more than X tests a week or month. See what you could analyse, otherwise you might get stuck in repetitive work and burn yourself out in the long run.
• #38: Thanks a lot for you time and have fun the rest of the day. Slides are available at this URL, or come talk to me then I’ll make sure you’ll get a link in your inbox. Thanks!