The guide to A/B testing
- 1. consultingSTARTUP MINDED THE GUIDE TO A/B TESTING
- 2. 2 CONTENTS A - WHAT IS A/B TESTING ?..................................................................................................3 ThePrincipleofA/BTesting..............................................................................................................................3 Improving the conversion rate: an important leverage effect................................................................6 B - HOW TO PUT A/B TESTING INTO PLACE: THE MILESTONES.................................7 Identify the opportunites for optimisation and construct hypotheses...............................................7 Prepare the tests: conception of variants and planning.........................................................................9 Interpret the results........................................................................................................................................11 C - GUIDELINES FOR GETTING STARTED: ......................................................................12 Adapttheorganization...................................................................................................................................12 Choosetherighttools.....................................................................................................................................13 consultingSTARTUP MINDED
- 3. 3 A/B Testing is a technique that enables testing of two versions of the same device in which a single variable is different. A/B Testing therefore measures the impact of that modified variable on the achievement of a given objective (rate of conversion, rate of clicks, lead generation...). In the american presidential race of 2008, Barack Obama’s team implemented an innovative approach for boosting fundraising by relying on A/B Testing. The obtained result: 60 million dollars of additional collection, which is around 8% of the total 760 million dollars spent during the campaign! During the long months, Obama’s team tested multiple variants of communication supports in order to retain only the most successful. In the example below, to optimize the registration page, several visuals and texts for the button were tested: the visual on the right, with Obama and his family, generated 13% additional conversion. ILLUSTRATION 1 : VARIANTS USED FOR FUNDRAISING FOR OBAMA’S 2007 CAMPAIGN (SOURCE : OPTIMIZELY) THE PRINCIPLE OF A/B TESTING A - WHAT IS A/B TESTING? A B
- 4. 4 Beyond the field of political communication, a growing number of sites will now implement A/B Testing techniques: e-commerce sites, service sites, content sites... Dell conducted a number of tests on their contact page which illustrate what is possible in this area: A/B Testing allows the user to be very efficient in the conception phases. In fact, the dedicated meeting for the conception of application screens, or other marketing materials, are often interspersed with long exchanges over ergonomic choices, the content of catch phrases, the position and color of buttons... Pieces of advice collide, returning to this or that «leading» practice of a key player, without being able to prove that it is really transposable / relevant in the given context. A/B Testing can often provide an objective response to this problem. The approach consists in comparing several versions of a support to determine which performs best according to a given criterion: the rate at which it is «added to cart», the rate of inscription, or any other measurable reference indicator. The example on the next page shows a schematic representation of two variants in a conversion funnel: the replacement of a text by a visual in the second version leads to an improvement in the rate of conversion of 4 points. Result: the version B enables a gain for 2 KPIs: -27% rebound rate, 36% form submission from version A ILLUSTRATION 2 : DELL CASE, CONTACT PAGE OPTIMIZATION FOR DELL ENTERPRISE (WWW.WHICHTESTWON.COM) A B
- 5. 5 Concretely, the completed tests can affect the optimisation of a large number of elements. The most common are given in the table below: ILLUSTRATION 3 : TWO MODELS ARE TESTED AND EVALUATED BASED ON THE OBTAINED CONVERSION RATE (SOURCE : KELEY CONSULTING) STANDARD MEDIAS Screen > Web page > Mobiel App screen/ touchpad Direct Marketing > E-mail > SMS > Mobile App push notification Advertising Support > Banner > Sponsored link, including Adwords STANDARD ITEMS Hook > Title > Subtitle > Tagline Text > Description > Benefits > Testimonials, social proof > Awards Image > Icons > Photos > Videos Chart > Color schemes > Size of the items Ergonomy > Zoning > Scroll vs sequence > Menu > Links TABLE 1 : THE USUAL DIGITAL ELEMENTS TO A/B TEST (SOURCE : KELEY CONSULTING) Model A (control) Model B (variant)
- 6. 6 The current guide concentrates on A/B Testing in the context of digital materials, but the me- thod also applies to numerous areas of «offline» marketing, notably: > Direct marketing: tests for mail, coupons... > Merchandising: changes in stalls of large distribution, organization of rays... > Packaging: during the testing phase, presentation of different variants > TV Advertisement: comparison of multiple commercial concepts > Product development: tests of variants in food, perfume... A/B Testing enters into the large family of methodologies and tools of CRO (Conversion Rate Optimization). Enabling the continual optimisation of the performance of digital devices and relying on the scientifc method, A/B Testing offers both significant ways to save time in arbitrating complex decisions, and a «depersonalization» of the debate. Fairly often, the economic effect of an increase in the conversion rate is underestimated, the efforts primarily focusing on the increase in traffic. However, the improvement of the transformation rate of a device often produces a significant leverage effect on the margin rate of an operation. In the example on the following page, corresponding to an online site selling physical products, we can see that an increase of 20% in daily sales steps up the net margin by 80% (leverage x4), hence a boosted ROI and a project that funds itself much quickly. IMPROVING THE CONVERSION RATE: AN IMPORTANT LEVERAGE EFFECT
- 7. 7 B - HOW TO PUT A/B TESTING INTO PLACE: THE MILESTONES The first step consists of clarifying the objectives of the devices to optimize, as indicators of improvement, notably: rate of inscription, rate of addition to cart, cart abandonment rate, average basket... It is then necessary to identify the best optimization opportunities: those that will have the strongest impact on performance. We are traditionally interested with the analysis of weak points, via the Web Analytics tool, for example: the pages with the strongest rebound rate, the steps generating the most cart abandonment. IDENTIFY THE OPPORTUNITIES FOR OPTIMAZATION AND CONSTRUCT HYPOTHESES COSTS Paid traffic share 60% 60% 60% 60% 60% Paid visits / day 1200 1200 1200 1200 1200 Cost / paid visit 0,6 € 0,6 € 0,6 € 0,6 € 0,6 € Paid visits cost 720 € 720 € 720 € 720 € 720 € Acquisition cost 18,0 € 17,1 € 16,4 € 15,0 € 13,8 € Production cost share 50% 50% 50% 50% 50% Production cost / basket 40 € 40 € 40 € 40 € 40 € Production cost 1 600 € 1 680 € 1 760 € 1 920 € 2 080 € Total costs 2 320 € 2 400 € 2 480 € 2 640 € 2 800 € NET MARGIN Net margin / day 240 € 288 € 336 € 432 € 528 € Net margin / year 87 660 € 105 192 € 122 724 € 157 788 € 192 852 € Net margin variation - 20 % 40 % 80 % 120 % Annual earnings via A/B Testing 0 € 17 532 € 35 064 € 70 128 € 105 192 € An increase of 20% of turnover excl. taxes ( 0.40 points of conversion rate) enables the increase in the net margin of 80% in the featured model. TABLE 2 : EFFET OF A CONVERSION RATE INCREASE ON NET MARGIN (SOURCE : KELEY CONSULTING.) Initial situation Conversion rate increased by 0.10 point Conversion rate increased by 0.20 point Conversion rate increased by 0.40 point Conversion rate increased by 0.60 point Conversion rate 2% 2,10% 2,20% 2,40% 2,60% Turnover variation excl. taxes - 5% 10% 20% 30% INCOME Visits / day 2000 2000 2000 2000 2000 Sells / day 40 42 44 48 52 Average basket 80 € 80 € 80 € 80 € 80 € Turnover excl. taxes / day 2 560 € 2 688 € 2 816 € 3 072 € 3 328 €
- 8. 8 Sources of optimization can likewise be identified by comparison with benchmarks from the competition. It therefore focuses on improving aspects that clearly underperform, compared to competitive sites or references. Once the priority optimizations and the objectives are identified, it is necessary to identify hypotheses for the brakes to raise and the levers to activate to increase the performance of the items concerned. To identify these hypotheses, one can carry out the following activities: > Internally: workshops with the marketing teams and the selling power > With the customers: online questionnaires, chats, interviews by telephone or in person The I.A.R. model published by Conversionista is useful for forming hypotheses: ILLUSTRATION 4 : DISPLAY OF THE PAGES WITH THE STRONGEST REBOUND RATE ON GOOGLE ANALYTICS Insight As we observe that: (Result) By : (observation method) Action We wish to: (implement this change) for: (this segment / this user profile) Result Which should result in: (this change in behaviour) And the effect will be measured in the following manner : (indicators / impacted metrics)
- 9. 9 Furthermore, there are numerous online resources, notably the site whichestwon.com, which provide A/B Testing data bases, put together by a variety of organizations. ILLUSTRATION 5 : WWW.WHICHTESTWON.COM, THE WEBSITE THAT PROVIDES MANY EXAMPLES OF A/B TESTING Once the testable hypotheses are identified, the variants of pages can be designed. In order to accelerate the implementation of the tests, it is recommended to use the tools of the online edition, provided by the solutions of A/B Testing, bypassing the purely graphic models. As a matter of fact, the tools of A/B Testing generally allow the creation of variants to test, without the need to program: modifications can be presented in a visual way. We could, for example, displace the elements and change their size, modify the content (texts, visuals, videos...), add or delete elements. PREPARE THE TESTS: VARIATIONS DESIGN AND PLANNING
- 10. 10 ILLUSTRATION 6 : OPTIMIZELY INTERFACE THAT ALLOWS TO CREATE VARIATIONS FROM A WEBPAGE Once the variants are created in the tool, it is important to make a recipe for controlling their operation in all the provided terminals, notably cellphones and tablets. In fact, certain modifications sometimes cause dysfunctions that must be corrected manually by a technical team, directly in the A/B Testing tool. Finally, before running the tests, it is advised to inform the teams in charge of customer relations, if the impacts are significant, so that they can properly support users as needed. A/B Testing does not degrade the natural referencing of a site as long as it is put in place with respect to a few simple rules. Google also encourages the implementation of A/B tests via the distribution of tools such as Google Analytics. Search engine teams communicate by rules of implementation: > Avoid the “cloaking”: Presenting a different version of a Web page to indexing robots of search engines and to visitors. This practice is discoverable by search engines which can degrade the ranking of a suspected site or even dereference it from SERPs. Therefore, one should not use A / B testing to target the crawlers, such as Google. > Avoid the “duplicate content”: Pages that exist under multiple variants can be signaled to engines using the tag «rel=canonical». This confirms to the engine that the variants are connected to the principle page. > Do not slow down the loading of the site: The implementation solution does not have to slow down the loading of the pages, an important factor taken into account by the engines to establish the ranking of pages in the search results.
- 11. 11 Interpretation of the results is a delicate stage because it requires the mastery of basic statistical concepts, including on the representativeness of the results. Indeed, a performance variation can only be considered statistically valid if it has been observed on a representative sample of the population studied. Fortunately, online tools facilitate interpretation by performing the calculations automatically, including the following tool: Sample Size Calculator (Evan’s Awesome A / B Tools). Sample Size Calculator (Evan’s Awesome A/B Tools). THE GUIDE TO A/B TESTING ILLUSTRATION 7 : ONLINE TOOL FOR EVALUATING IF A TEST IS STATISTICALLY SIGNIFICANT Moreover, it happens regularly that the A/B tests are not relevant, especially for the following reasons: > Not enough users tested the proposed variant, so the test is not statistically significant > There were several changes in the same test and they are offset > The modification did not have a connection with the measurement used to evaluate the result of the test When these situations present themselves, it is necessary to document them, as to capitalise on their lessons and use them for the next tests.
- 12. 12 C - GUIDELINES FOR GETTING STARTED The organizations that effectively use A/B Testing often carry out hundreds of tests per month. The website Etsy, for example, generates 25 A/B tests per day and Shop Direct, more than 100 per day. To reach this pace, the organization must be adapted to avoid the pitfalls associated with classic cycle projects : validation by several hierarchical layers, design review meetings... To save time, it is thus recommended to appoint officials who will work with independent tests within the framework imposed by a A/B Testing charter, of which the terms are negociated with the stakeholders of the company usually called to intervene in the lifecycles of the projects. Typical categories of an A/B Testing charter : ADAPT THE ORGANISATION Rules Description Examples Authorized pages List the pages that should not be A / B tested No test on the following pages: the values of the company, CSR policy ... Texts List restrictions on certain texts Terms & Conditions (GTC), Terms of Service (TOS) Visuals List restrictions on visuals Only use visuals which are in the directory selected by the internal communication Periods List the restrictions on testing periods No test during sales weeks I It is also useful to create a shared file to document the passed tests and conserve a memory of accumulated knowledge in the framework of the tests : the hypotheses tested, the obtained results...
- 13. 13 Numerous tools appeared to complete the A/B tests, they can be classified into three categories : 1. Dedicated SaaS solutions for A / B testing: Optimizely, who knew strong international success, or French A/B Tasty, with important parts of the market on the national territory; 2. SA/B Testing solutions included in a marketing suite: Certain digital marketing suites present a dedicated tool for A/B testing. This is the case with the Adobe Marketing Cloud suite with Adobe Target, and with the Google Analytics 360 suite with Google Analytics; 3. Internal solution: The tools are sometimes implemented by the agencies or the internal teams of big businesses. It is recommended to take precautions regarding the internal solutions for several reasons: > To limit the dependence on a provider, if the solution is proposed by an agency, > Because the A/B Testing solutions are complex softwares, which require substantial investments, difficult to mobilize by teams who haven’t achieved a certain critical success. In April 2016, the ranking established by TrustRadius put the solutions of A/B Testing of Optimizely and AB Tasty in the forefront. To learn more about A/B Testing and to meet our teams, contact us : Keley Consulting, 28 rue du Dr Finlay, 75015 Paris Phone : 01.80.48.26.20. - Email : contact@keley-consulting.com CHOOSE THE RIGHT TOOLS ILLUSTRATION 8 : MAPPING TOOL PROVIDED BY TRUSTRADIUS (HTTPS://WWW.TRUSTRADIUS.COM/AB-TESTING).