Opticon 2015-Scaling Your Testing Program for Maximum Impact
0 likes944 views
The document outlines strategies for scaling testing programs effectively, emphasizing intelligent prioritization, managing multiple testing streams, and preventing errors in tests. Presenters from Moz and The Next Web share their experience in increasing the frequency of tests from one per month to five to six per week, discussing key performance indicators and automation for efficiency. Recommendations include understanding KPIs, creating a backlog for ideas, and leveraging data to target optimization opportunities across user paths.
Opticon 2015-Scaling Your Testing Program for Maximum Impact
- 1. #opticon2015 Scaling your Testing Program for Maximum Impact Andy Nelson Director of Growth Marketing, Moz @theandynelson andy.nelson@moz.com Martijn Scheijbeler Lead Growth, SEO & Analytics, The Next Web @martijnsch martijn@thenextweb.co m Khattaab Khan Strategy Consultant, Optimizely @khattaabkhan khattaab@optimizely.com
- 2. Increase Your Impact Without More Resources Andy Nelson Director of Growth Marketing, Moz @theandynelson andy@moz.com #opticon2015
- 5. #1 Go Big
- 11. Elements of good prioritization Core • Potential impact to success metric • Estimated runtime required • Estimated level of effort
- 12. Elements of good prioritization Modifiers • Alignment with strategy • Support from VOC, analytics, prior testing • Urgency Core • Potential impact to success metric • Estimated runtime required • Estimated level of effort
- 14. #3 Swim Lanes
- 19. #4 Multiple Streams of Work
- 21. 1 week 1 day1 day
- 22. Checkout Optimization Product Page Optimization Test Analyze Design/Build Test 1 Test 2 Test 3 Test 4 Test Analyze Design Test Design/Build
- 23. #5 Prevent Busted Tests (or at least catch them early)
- 29. 0 22500 45000 67500 90000 5/3 5/10 5/17 5/24 5/31 6/7 Watch for irregular events
- 30. Test big, bold changes Improve your prioritization Run multiple tests Eliminate dead time Improve your QA Takeaways
- 32. #opticon2015 Thank you! Andy Nelson Director of Growth Marketing, Moz @theandynelson andy@moz.com
- 33. #opticon2015 How The Next Web’s Testing Program Scaled Martijn Scheijbeler Lead Growth, SEO & Analytics - The Next Web @martijnsch martijn@thenextweb.com
- 34. Yearly 5.000 attendees, 75.000 products sold. Monthly 6 million unique users, 12 million pageviews. The Next Web: 2006 > 2015
- 36. What’s happened in the last 12 months? Scaling our testing program
- 37. That clearly wasn’t good enough 1 test a month :(
- 38. Currently 5-6 tests a week, how did we get there? 15-20 tests a month :)
- 39. What did we learn along the way? • Swim lanes, KPIs, Goals. • Segments: traffic sources, etc. • Idea creation: backlog + prioritisation. • Scaling > Analysing results. • Do it live!
- 40. Unique Users, Page Views, Emails & Revenue. Goals: UU, PV, Emails & $ Related stories, pagination, sharing, email signups. KPI: CTR, Engagement, Funnels
- 41. New v.s. Returning - Europe v.s. USA - Facebook v.s. SEO - Mobile v.s. Desktop Type, Regions, Source, Device
- 42. Idea creation v.s. Planning • Different templates + segments. • How much time per test? Run tests for a fixed predefined time. • The maximum of your program. • Take away: create a backlog!
- 43. The part that we can’t scale yet
- 45. Well, this is both the most boring and most exciting part! 1-2 main KPIs, 15 secondary KPIs
- 46. Rails, Google Analytics API Built with love by the TNW Marketing team. TNW Analytics
- 47. TNW Analytics was born!
- 48. Adding a new idea
- 49. Schedule
- 50. Test
- 51. Saving 20% of time per test!
- 52. Take away: Be able to push code!
- 53. What’s next?
- 54. Scaling our testing program to the next level? • Even more templates, across more projects. • 25+ tests a month with current growth of traffic. • Bigger team? We can’t keep up with coding ;-) • More automation? Push test variant to GTM. • Mix it up with personalisation/ behaviour. • Push our experiments to Slack for sharing. • Create a better style guide + Bootstrap guidelines.
- 55. Summary & Take aways • Know your KPI framework, expectations and limitations. • Know your segments, CR% so you know how to prioritise. • Follow your planning, priorities. • Scale = 40% - 50% automation: analyse fast! • Be able to push code live!
- 56. #opticon2015 Have fun & Thank you! Martijn Scheijbeler Lead Growth , SEO & Analytics - TNW @martijnsch martijn@thenextweb.com
- 57. Strategies Essential to Achieve Testing Program Scale #opticon2015 Khattaab Khan Strategy Consultant, Optimizely @khattaabkhan khattaab@optimizely.com
- 58. Focus on Optimization Opportunities that Matter
- 60. 44% 42% 8% 52% 87% 81% 41% 31% 44% 32% 11% 12% 15% 27% 48% 16% 7% Homepage to Category Category to Product Product to Shopping Cart Shopping Cart to Checkout Checkout to Payments Payments to Conversion Continue down funnel Visit different page Exit website X A M P LE Homepage Category Page Product Page Shopping Cart Checkout Page Payment Page With a conversion funnel, we could analyze all steps to see how and where users drop offEvaluate User Path
- 61. 44% 42% 8% 52% 87% 81% 41% 31% 44% 32% 11% 12% 15% 27% 48% 16% 7% Homepage to Category Category to Product Product to Shopping Cart Shopping Cart to Checkout Checkout to Payments Payments to Conversion Continue down funnel Visit different page Exit website X A M P LE Homepage Category Page Product Page Shopping Cart Checkout Page Payment Page With a conversion funnel, we could analyze all steps to see how and where users drop off Focus Testing on Problem Areas that are Drivers of Business Value
- 63. Roadmap
- 64. Roadmap + Resource Planning
- 65. Custom Prioritization Scoring Pauline Marol Lead PM, Site Optimization “I don’t want to be the one making decisions on what experiments I run and don’t run because I am biased.”
- 66. Results Analysis Results Page Best Practices • Evaluate opportunity costs • Analyze segment performance • Shareable results link
- 67. Results Sharing Communicating Results + Roadmap Iteration • Hold team accountable to strategy that inspired variation design • Validate/invalidate hypothesis to evaluate goal effectiveness • Define where results output will live (e.g. database, wiki, etc.)
- 69. Simultaneous Testing Use Cases Experiments on same page Experiments on same flow (e.g. checkout / registration flow) Multivariate experiment
- 70. Beware of Interaction Effects Elements on same page Elements on subsequent pages in a funnel When multiple points of user interaction influence the same goal, interaction effects can taint data
- 71. Evolution of Testing Org Structure
- 72. Core Testing Team MEASURE RESULTS Analytics Dev + QADesign Program Manager CRO Analyst optional
- 73. Evolution into Optimization Center of Excellence MEASURE RESULTS Analytics Dev + QADesign Program Manager Product Manager Product Manager Dedicated optimization resources serving multiple BUs
- 74. Leverage data to focus on goals that directly drive business value Hold your team accountable with effective planning documents Know when to run simultaneous tests Scale your program structure to support multiple business units Takeaways
- 75. #opticon2015 Thank you! Khattaab Khan Strategy Consultant, Optimizely @khattaabkhan khattaab@optimizely.com
- 76. #opticon2015 Andy Nelson Director of Growth Marketing, Moz @theandynelson andy.nelson@moz.com Martijn Scheijbeler Lead Growth, SEO & Analytics, The Next Web @martijnsch martijn@thenextweb.co m Khattaab Khan Strategy Consultant, Optimizely @khattaabkhan khattaab@optimizely.com Now for some Q&A