What your testtool doesn't tell you

Editor's Notes

• #2: First of all, I’d like to thank Jackie for inviting me to this Conversion Elite conference in Manchester.
• #3: For everyone who doesn’t know me, I’ll start with a short introduction. I can keep it real short, because Eline did the most work for me. As said I’ve worked in the field of webanalytics for over 7 years now and have worked for several companies. I like the travel industry the most, that’s why I’ve recently switched jobs. Right now, I’m Conversion Manager at Tix.nl and am responsible for the Abtesting program. I love running experiments and optimize websites based on data.
• #4: As said, the travel industry really appeals to me. And that’s not just work related.  My life motto is basically: work, save, travel, repeat. The thing that make me the most happy en thrilled is wandering the world. Right now I’m in the work phase, but tomorrow it’s travel time. I have extended my stay until this weekend so if you have any tips for me please let me know during drinks.
• #5: Last summer we went to Northern Norway. We visited Tromso and the Lofoten, but also went to a lesser known island called Senja. There were no travelguides about this area so we googled to find out what we could do. We found a hike going to one of the highest mountains of Senja, which should have an amazing view. It was supposed to be a 2 hour hike, so very doable.
• #6: The trail started off rather easy and we were opportunistic and enthousiastic to reach the top. But our enthousiasm soon declined. The trail was getting steeper and steeper and there were several times that I wanted to call it a day and return to the car. My limbs were sore and I was totally out of breath.
• #7: But we persevered and made it to the top. The view was to die for. And the whole journey towards the top was totally worth it.
• #8: It would have been soooo much easier if there was an easy route, like a cable car. But then we wouldn’t have overcome the challenges and learned to persevere, and we wouldn’t have been as thrilled to made it to the top.
• #9: This hike for me stands for the way we look at A/B-testing as well. We are inclined to only have the end goal in mind and take the easy route. We buy a testingtool and immediately expect amazing outcomes. We let the marketeer run the test program. The testtool can make the variation so we don’t need developers, the testtool will tell when the test is cooked so we don’t need data scientists and the testtool will tell you if it’s a winner so we don’t have to do the analysis ourselves. Easy does it! But running a proper CRO program can’t be done by skipping the hiking trail and taking the cable car to the top. You need to put time and effort in it. Because there are important things our testingtool doesn’t tell us.
• #10: The first thing testtools don’t tell you is what’s worth to test? Where should you start? You can set up a test on a page with 50 visits per day, but such a test would run for ages and the results would then still be questionable! So how do you determine where you should A/B-test on your site and where you shouldn’t?
• #11: Well, there’s a rough rule of thumb to keep in mind. Basically, if you have less then 1,000 conversions per month, it’s not worth the trouble. Because in order to find a significant effect between A and B you will need a conversion rate uplift of around 15%. And these kind of uplifts are not easy to accomplish. So if a page has fewer than 1,000 conversions per month, you shouldn’t spend your resources on it, but find other ways to validate your idea (with other types of more qualitative research). But this benchmark of a 1000 conversions per month isn’t specific enough.
• #12: Before I start with an optimization project I first start with a test power determination. This is basically a fancy word for determining whether of not you can run a test on a specific page with a reasonable chance to find a winner. This reasonable chance to detect a winner is called ‘Statistical Power’
• #13: I know this sounds a bit cryptic, but what you should remember is
• #14: you need a certain amount of people in your test and the change should make an impact on behavior to be able to proof your hypothesis. the more people in your test, the higher the power and the bigger de difference between A and B the higher the power.
• #15: For each page template you want to test on you should determine the unique weekly visitors and the unique weekly buyers. Given the 80% Power critical value and the pre-determined confidence level you can calculate the needed sample size and corresponding effect size. Suppose you have 32.000 weekly visitors and 800 buyers. These are nice numbers to perform an Abtest on right?
• #16: Well, in order to have a Power of at least 80% you can detect uplifts of 9% and higher in 4 weeks time. If you expect that your variation will only result in a 5% uplift, then your sample size needs to be way higher to garantee 80% power. You then need to run the test for 13 weeks. And this is not something you want to do. The adviced maximum test duration is 4 weeks. If you would test shorter than this, then the chance to detect a winner will be far lower than 80%. You need to be aware of the of the effect you need from your test and how long the tests needs to run. If you know what impact you need to make, you can design your test accordingly. If you know you need an uplift of 10% you will probably test bolder changes.
• #17: The 2nd thing your testing tool doesnt tell you is how you should code your variations.
• #18: A couple of months ago we did an Abtest where we showed our destinations that are highlighted in the tv-commercial on the homepage. The code for these 6 blocks was written by our front-end developer, but during the test the visuals and prices needed to be updated. My colleague was responsible for this and used the WYSIWYG editor for this.
• #19: The result after 4 weeks of testing was this monstrosity of a code.
• #20: This is really dirty code and front-end developers will probably hate you when you show them this. Because, it doesn’t only look ugly, it is ugly as well. Because the longer the code, the slower it will load. And it’s unreliable in different browsers. Some adjustments will work perfectly find in Chrome, but will not work in Internet Explorer. And where you have these kind of codes it becomes hard to QA And the third drawback of the WYSIWYG editor is that it’s rather limited. You cannot add new functionalities or completely change the lay-out.
• #21: For easy tests you can code this yourself using html/css and javascript and jQuery. You can quickly learn the basics on Codecademy.com. For the more complicated tests you really need development resources. There’s no way around it.
• #22: The 3rd thing testing tools don’t address is the possibility of validity problems.
• #23: Validity is the extent to which a conclusion or measurement is well-founded and corresponds accurately to the real world. If you have run a valid experiment, the results can be generalized for the whole population, but there are a couple of checks you need to do.
• #24: You need to make sure that each group in the sample is representative of the total population.  Especially if you have a smaller sample size this might be an issue and when you have groups in your population with very different conversion rate. For example, new visitors normally have a far lower conversion rate then returning visitors. This means that when you have relatively more returning visitors in one of your variations the measured uplift may well be caused by the difference in composition of the samples. So you need to check the representativeness of each sample before drawing conclusions. This is something a testtool doesn’t do for you.
• #25: All the testing tools in the market will call a winning result regardless of how long the test has run. Of course they take into account sample size, but they don’t take into account day-of-the-week effects. If you take a look at your data over a longer period of time and big differences appear in conversion rates per day, you should always test for full weeks. Because a winning test based on only weekdays, may well underperform in the weekend. Again: these results cannot be generalized over all the days of the week.
• #26: And another important one: you cannot generalize results found in a period where user behavior is different than normal. If you found a winning result around Christmas, you should at least re-test this in another time of the year. Or wait for the next Christmas to implement the winner.
• #27: Another threat to running a successfull testing program is the possibility of sample pollution. We wrongfully asume that we can build the perfect scientific experiment online.
• #28: There are several reasons why unique users might end up in both variations of an experiment. First of all visitors can delete their cookies or browse incognito. When visitors return to the site they have a 50% chance (when you just run A vs B) that they will see the other variation. Secondly, more and more visitors will use more than 1 browser of device in their quest for your product. Especially cross-device usage is a big polluter of your sample. And lastly, if the orientation phase for the product you are selling is long, then visitors will probably have seen the original page before they ended up in your test variation.
• #29: So suppose you did an experiment and you have found that B has a higher conversion rate than A
• #30: Now you know a proportion of both the samples is polluted. Some visitors in A have also seen B and vice versa.
• #31: We don’t know what the conversion rate is of the ABBA group, but we make the assumption that their conversion rate is probably the average of A and B. This means that the actual conversion rate of A is even lower (because it positively influenced by the ABBA group). And the actual conversion rate of B is probably higher – because it’s negatively influenced by the ABBA group.
• #32: So you measured this uplift in your test
• #33: But the actual uplift is far higher! This means that if you have a lot of pollution in your test, you will have a hard time finding the effect.
• #35: Especially if your win ratio is at the lower end (like lower than 20%) you need to do research how big these issues are.
• #36: If you have found out that the pollution rate is very high, you need to adjust your testing strategy.
• #37: And the last thing I want to address it that you need a basic understanding of the statistics that are used by your testtool. Because every A/B-test uses its own statsengine and you need to be aware what this means when drawing conclusions.
• #38: There are 2 types of statistics that have been used in A/B-testing: frequentist and Bayesian.
• #39: Historically most testtools used frequentist statistics, but over the last couple of years more and more tools switched to Bayesian statistics. And this is not without reason. Using frequentist statistics has a couple of challenges. I’ll explain this.
• #40: Frequentist testing can be compared with court trial in the US. The null hypothesis says that the defendant is innocent. The defendant is innocent until proven guilty! and the alternative hypothesis says that the defendant is guilty. We then present evidence or, or in other words, collect data. Then, we judge this evidence and ask ourselves the question, could the evidence have happened by chance if the defendant is innocent? Is there reasonable doubt?
• #41: This principle is used in Frequentist statistics as well. You have a null hypothesis stating that there is no difference in conversion rate and you try to disprove this claim. You run your test and you judge the results based on the p-value. You try to answer the question: could the data plausibly have happend if the null hypothesis is true? If there still is doubt then you fail to reject the null hypothesis and conclude that there is no difference. But if there’s no doubt (or very little) than you reject the null hypothesis.
• #42: Snoop Dogg has an excellent line for this: if the p is low, the ho must go
• #43: I will give an example how this translates to an A/B-test. So, suppose you did an experiment and the p-value of that test was 0.01. You remembered: the p-value is very low, so the H0 needs to go. But what is the exact conclusion of this test? With the use of frequentist statistics you can only conclude how surprising the results are based on the hypothesis that A and B perform exactly the same. I don’t know about you, but this confuses the hell out of me! This is really hard to explain – not only to fellow optimizers but mainly to your boss. And besides the confusion, I’m actually not interested in “how unlikely it is that I found these results.” I just want to know whether variation B is better than A. Frequentist statistics are counter intuitive.
• #44: The other challenge with using frequentist statistics is that an A/B-test can only have 2 outcomes: you either have a winner of no winner. In other words, you can either reject the null hypothesis or fail to reject it. And there is no wiggle room. If you take a look at this test-result you would conclude that there is no winner, that it mustn’t be implemented and that the measured uplift in conversion rate wasn’t enough. So you will see this a loser and move on to another test idea. However, there seems to be a positive movement (the measured uplift is 5%), but it isn’t big enough to recognize as a significant winner.
• #45: The alternative to using frequentist statistics, is Bayesian statistics. And as said most test tools have switched to using Bayesian (or using flavors of Bayesian). And that’s not without reason: Bayesian statistics makes more sense, since it far better suits the underlying business question.
• #46: When you use Bayesian statistics, to evaluate your A/B-test, then there is no difficult statistical terminology involved anymore. There’s no null hypothesis, no p-value or z-value et cetera. It just shows you the measured uplift and answers the question what the chance is that B is better than A. Easy right? Everyone can understand this. Based on the same numbers of the A/B-test I showed you earlier you have a 89,2% chance that B will actually be better than A. Probably every manager would understand this and will like these odds.
• #47: When using a Bayesian A/B-test evaluation method you no longer have a binary outcome like the t-test does. A test result won’t tell you winner / no winner, but a percentage between 0 and 100% whether the variation performs better than the original. In this example 89,2%. The question that remains is: is this enough to be implemented?
• #48: Well that depends on a couple of things. If you would implement a test variation with a probability of 51% then you’re not doing much better than just flipping a coin. The risk of implementing a losing variation is quite high. Depending in the type of business you may be more or less willing to take risks. If you are a start-up you might want to take more risk then a full grown business, but still we don’t really like the chance to lose money so what we see with our clients that most need at least a probability of 75%. But it also depends on the type of test. If you only changed a headline then the risk is lower, then when you need to implement a new functionality on the page. This will consume much more resources. Hence, you will need a higher probability.This deliberation isn’t presented in your testtool, you need to make this decision yourself. For some tests you need 95% probability, whereas for other you are happy with 75%.
• #49: What you can do is make a risk assessment. You can calculate what the results mean in terms of revenue. When the client decides to implement the variation they have a 10.8% chance of a drop in revenue of 200.000 in 6 months time (and an average order value of 175) But on the other hand, they also have a 89.2% chance that the variation is actually better and brings in nearly 650.000 euro. You can show this table to your boss and ask whether he would place the bet. So it’s not the testingtool which decides if it’s a winner or not it’s you (or your boss).
• #50: So, to sum up there are a lot of things your testing tool doesn’t tell you: - you need to: 1,2,3,4,5,6….It’s not an easy path and there will be times you want to call it a day, but if you persevere you will make it to the top and fully enjoy what you have accomplished.