Conversion Jam
1 like2,478 views
1) The document discusses how The Next Web scaled up their A/B testing program from 20-30 tests in 2014 to around 200 tests per year. 2) They started by using Google Analytics Content Experiments but then hit limits, so they developed their own process and tools using Google Tag Manager and JavaScript. 3) Key aspects of their approach included setting a clear vision, structuring their team for testing, communicating results, continually innovating their process, and focusing on quality over just quantity of tests.
Conversion Jam
- 1. How to run 200 tests a year & what do you learn? Martijn Scheijbeler / Director of Marketing / martijn@thenextweb.com / @MartijnSch
- 2. Emerce Conversion - @MartijnSch Title
- 3. Know why, you test and how you scale what matters: @MartijnSch WHAT YOU GET OUT OF THIS? We’ll be sharing what tools + process we have at The Next Web to get started!
- 4. How it all Started @MartijnSch THE START
- 5. Why we feel The need for testing? @MartijnSch#ConversionJam
- 6. “So what would we have to do to grow x10?” Boris (CEO) “I have no idea what I’m doing” @MartijnSch / #ConversionJam
- 8. Why am I showing you this slide? @MartijnSch#ConversionJam
- 9. But how did we get to 200 A/B Tests a Year? @MartijnSch#ConversionJam
- 10. Testing Figures Data from 2015 2,5CRO Team in 2015 38.000.000Sessions with Tests 0CRO Team in 2014 5-7Tests per week PricelessAmounts of fun 35%Traffic v.s. Tests @MartijnSch
- 11. @MartijnSch Our roadmap for getting to 200 tests a year Jan ‘15 Let’s do more testing! After running 20-30 tests in 2014 we knew we had to scale up. Resources were low, but with the addition of a web analist we could grow. Apr ‘15 Hitting some limits Google Analytics Content Experiments (JS) was our first tool to use. But soon we hit the limits of the tool. May ‘15 Setting the vision After running 40 tests by May we needed a beter process that could scale and that aligned with our testing vision. Dec ‘15 Reaching XXX tests With the testing vision and updated testing program we were able to run XXX tests in total in 2015. 3 Months 1 Month 7 Months
- 12. Quantity v.s. Quality? @MartijnSch But why would you like to run at high velocity? Why not go for quality? #ConversionJam
- 13. Testing Figures Data from 2015 26%Winning Tests in 2015 @MartijnSch :(:)
- 14. “If you double the number of experiments you’re going to double your inventiveness” - Jeff Bezos @MartijnSch
- 15. What makes it different: Publishers v.s. E-commerce? @MartijnSch#ConversionJam
- 16. The opportunity is 350-400 tests a year. As ROI per winning test is lower we need to run a lot to have a positive return. Efficiency <> Costs @MartijnSch / #ConversionJam
- 17. How is our (marketing) team Structured? @MartijnSch#ConversionJam
- 19. F*CK it, we’ll do it live It’s not rocket science “Innovate through in-house solutions” @MartijnSch
- 21. We failed, not once but also not twice… @MartijnSch#ConversionJam
- 22. We needed to learn, Knowledge building? @MartijnSch You got to love working with people who are like minded. #ConversionJam
- 23. Thanks for all your support! We learn from you! @MartijnSch / #ConversionJam
- 24. The more we test the more we innovate and the higher inventiveness. Find the perfect process @MartijnSch / #ConversionJam
- 25. The opportunity is 350-400 tests a year. As ROI per winning test is lower we need to run a lot to have a positive return. Know your metrics? @MartijnSch / #ConversionJam
- 26. @MartijnSch 1 2 Unique Users Growing our unique user base will increase the potential to reach Generation T. 2 3 3 Our metrics/KPIs for success; User Engagement: PV/Session A higher engagement with our user base will increase the number of pageviews per session. User Retention: Returning % Increasing the percentage or returning users will increase our brand awareness. Newsletter Signups To get a relationship with our users we’d like to have their email address for the newsletter. Social Shares More shares will influence our reach amongst new and returning users.
- 27. The opportunity is 350-400 tests a year. As ROI per winning test is lower we need to run a lot to have a positive return. Do you have a swimming pool? @MartijnSch / #ConversionJam
- 28. The opportunity is 350-400 tests a year. As ROI per winning test is lower we need to run a lot to have a positive return. Segment Users @MartijnSch / #ConversionJam
- 29. Users are the same @MartijnSch KNOW YOUR SEGMENTS X X X
- 30. Users are the same @MartijnSch KNOW YOUR SEGMENTS X X X
- 31. Google Tag Manager (GTM) is making our lives easier + scalable. What tools we use @MartijnSch / #ConversionJam
- 32. So we need to improve most of our communication around testing. Communicate more @MartijnSch / #ConversionJam
- 33. How did we overcome our fails? @MartijnSch You fail once, you fail twice! #ConversionJam
- 34. We didn’t move to any ‘real’ CRO tool but decided to iterate on our own: GTM <> JS. GTM Download: Github.com/TheNextWeb/CRO @MartijnSch / #ConversionJam
- 35. Building our own solutions for saving testing documentation + analysis of tests. Innovate @MartijnSch / #ConversionJam
- 36. We moved most of our communication towards Slack. Making the teams more involved in the whole process. Always communicate @MartijnSch / #ConversionJam
- 37. We hired two front-end developers, to make sure in the long-run we can test bigger things. Front-end Dev @MartijnSch / #ConversionJam
- 38. 5-7Tests per week 2-4Tests per week 30-40Tests per year 20%More efficient @MartijnSch Improvement What does progress look like?
- 40. Who’s able to deploy winning variants within minutes from their codebase? Continous Improvement @MartijnSch / #ConversionJam
- 41. This will make segmentation + automation even easier. In addition it will decrease the flickering effect. Back-end Testing @MartijnSch / #ConversionJam
- 42. Testing small doesn’t work, so we need to grow even bigger. On average we grow traffic with around 10-20% yearly. At the same rate we can increase our testing program. Big > Biggest @MartijnSch / #ConversionJam
- 43. TL;DR Process 1. Have the right company vision 2. Have the right team structure 3. Have access + support from C level 4. Create partnerships with like minded 5. You’ll have trouble as well, nobody’s perfect 6. Don’t be afraid to invent the wheel again 7. Always focus on innovation @MartijnSch#ConversionJam
- 44. What tests did we learn from @MartijnSch LEARNINGS
- 46. Sharing @MartijnSch#ConversionJam More content on social media drives brand awareness + reach
- 47. Sharing brings in more loyalty, more engagement and hopefully more traffic. Share buttons
- 48. • Variant increase clicks with 16% • Had a significance + power level of > 99% • We ran the test for 1x 7+ days • Targetting over 200.000 users Winner: Variant Original: Variant
- 49. Engagement @MartijnSch#ConversionJam More engagement increase loyalty which provides advertising revenue
- 50. Engagement is important, focus on getting more clicks on the related stories in the sidebar. Rankings!
- 51. • Variant increase clicks with 36% • Had a significance + power level of > 99% • We ran the test for 7+ days • Targetting over 300.000 users Winner: Variant 1 Original: Variant:
- 52. Engagement is important, focus on getting more clicks on the related stories below the article. Related Stories
- 53. • Variant increase clicks with 36% • Had a significance + power level of > 99% • We ran the test for 2x 14+ days • Targetting over 500.000 users Winner: Variant Original:Variant
- 54. So what if we remove the whole sidebar, wouldn’t people be more focused on the stuff that is out there? Big, bigger
- 55. • Variant decreased clicks with 17% + 14% • Had a significance + power level of > 99% • We ran the test for 14+ days • Targetting over 420.000 users Winner: Original
- 56. Now you know how to run 200 A/B Tests a Year? @MartijnSch
- 57. But did we get to 200 tests? @MartijnSch
- 58. “Why didn’t we run 200 tests?” “What went wrong?” “Do I need to start crying?” Boris (CEO) @MartijnSch / #ConversionJam
- 59. “Well, if this guy wouldn’t have stuck his fingers into a lawn mower, YES!” Missing out on ±20 tests in 2015. The Lawn Mower @MartijnSch / #ConversionJam
- 60. Take Aways: 1. Focus on running X tests a year 2. Improve the process 3. Have the right (free) tools 4. Have the right people + support 5. Document/Communicate everything 6. Know what the future looks like 7. Have fun! @MartijnSch#ConversionJam
- 61. Emerce Conversion - @MartijnSch Title
- 63. Do you have any Questions? @MartijnSch#ConversionJam