DV 2016: Building a Culture of Testing and Optimization
Download as PPTX, PDF0 likes518 views
This document discusses building a culture of testing and optimization. It outlines key steps to create an effective test and learn organization, including defining roles like sponsors, ideators, executors and analyzers. It emphasizes the importance of developing a testing plan, conducting regular testing meetings, and standardizing the testing process. The overall message is that organizations should start with basic testing to establish processes and not panic if early tests show no significant results, as the goal is to learn through testing.
![Test Name
Test
Cell Test Description Conversion Test Type Test Location Status
Launch
Date
E
D
Control: Current quote first position
Test:
The IRS is expecting “many filings” to be in error at some level. No one can just blow off the law, but there will be something
of a pass on reporting penalties as long as it can be demonstrated that effort went into compliance. Walter Fagan
Control:
What’s dominating the Employee Benefits agenda in 2016?
Get the answers to the big questions →
Test A:
What’s dominating the Employee Benefits agenda in 2016?
Get expert insights and tools you can use in the year ahead →
Control: current template design
Test: Test: plain text format that will appear personalized including 1. Dear [name] 2. Sincerely [regional EB president]
Control: no nurture email
Test: CMnurture series to those who completed ACA nurture Ems
Control: always on
Test A: second panel of ACA site
Test B: new contact form to be designed for campaign
Control: no button/SMbuttons
Test: remove SMbuttons and add contact us button that goes to contact a rep form
Control: https://www.hubinternational.com/employee-benefits/contact-us/
Test: modal pop-up
Control: Current copy
Test:
Headline: Optimize your Employee Benefits in 2016
Box 1: ACA Reporting & Verification Audits
Box 2: Cost Management Strategies
Q1 Campaign Site
email
Live 9-Feb T
Complete 4-Feb 1EB Always On HP Interstitial Link clicks A/B Q1 ACA Campaign Site
ACAPanel Jack Quote 1 Share button clicks A/B
TBD T
CM nurture to ACAlist
Increase lead
pipeline
A/B email
Live 9-Feb T
CM nurture email #2 template Email CTR A/B On-Hold
Contact Us button location Button clicks A/B Q1 ACA Campaign Site
ACAAd words Landing page test
Form Completion
Rate
A/B/C Ad words
Live 7-Mar
Complete 22-Feb 11
T
Contact Rep FormDesign - sub
pages
Form Completion
Rate
A/B Always On Site Live 9-Mar
L1 page CTA Link clicks A/B Q1 L1 page Complete 9-Mar 6](https://image.slidesharecdn.com/dv2016day1trackcjringvfinal-160515211022/85/DV-2016-Building-a-Culture-of-Testing-and-Optimization-18-320.jpg)
DV 2016: Building a Culture of Testing and Optimization
- 1. © 2016 Tealium Inc. All rights reserved. | 1 Building a Culture of Testing and Optimization Joel Ring, HUB International
- 2. 2000 2006 20152009 Joel Ring Director of Analytics, HUB International 15+ Years in digital analytics Created and managed the Web Analytics department at bestbuy.com Sr. Digital Marketing Consultant working with some of the biggest brands on the web Created the Digital Strategy & Solutions team helping clients with their analytics and digital marketing Manage the Analytics team focusing on digital analytics, data sciences, and testing & optimization
- 3. A lot has changed 2000 2006 20152009
- 4. The great thing about the internet is you can measure anything! We’re ‘gonna fail- fast and break things! Like Facebook! The only way to know what will work best is to test it!
- 5. With all the data generated through testing we will be able to understand our customers like never before and learn new ways to meet their needs.
- 6. Every minute… • Facebook users share nearly 2.5 million pieces of content. • Twitter users tweet nearly 300,000 times. • Instagram users post nearly 220,000 new photos. • YouTube users upload 72 hours of new video content. • Email users send over 200 million messages. • Amazon generates over $80,000 in online sales.
- 7. “Two-thirds of business are unable to extract value from their information.” Source: 2015 Information Value Index from PwC and Iron Mountain
- 8. If we have access to all this data, how do we create an environment that enables us to derive value from that data?
- 9. Definition of an effective test and learn organization look like?
- 10. Implemented roles and procedures to create, execute, and evaluate tests AND are actively communicating results to drive change. “The key to great analysis is not math, it’s communication.”
- 11. Keys to creating a test and learn organization Sponsors Ideators Executors Analyzers The sponsor helps evangelize the testing organization and helps drive changes based on the results. The ideators help develop testing ideas. These can be part of the testing team or stakeholders. This is the group that executes the tests. This will generally encompass multiple departments and can include creative and technical resources. The analyzers tabulate and communicate the results of the tests. They will be the team members who declare the test “winners”. • Individuals can, and will, fall into multiple roles • Include people from multiple departments • Make sure to include stakeholders • How many people in these roles? • Forrester estimates 5-15
- 12. How do I get the initial buy in?
- 13. Demonstrate value using existing channels that already allow for testing Email – Display Advertising – Paid Search How do I get the initial buy in?
- 14. Conduct a kick off meeting
- 15. Conduct a kick off meeting • Bring together people who will be either involved in or interested in the execution and effectiveness of your testing program • Goal of meeting is to educate generate excitement through collaboration • Kick off agenda o Testing 101 • Types of test (A/B, multivariate, etc.) • What makes a good test • What is a testing plan o Intro to our testing platform o Intro to our testing process o Ideation workshop • Used the ideation ideas to start our testing prioritization
- 16. Develop a testing plan
- 17. Develop a testing plan • Guides your testing prioritization, status and results • What is in a testing plan? o List each test idea o Conversion o Status (Live, scheduled, complete, removed, on-hold) o Date launched o Date completed o Results (winner) o Test type (A/B, A/B/C, multivariate) o Test location (site, email, SEM, display advertising) • Keep in a central location accessible by the team
- 18. Test Name Test Cell Test Description Conversion Test Type Test Location Status Launch Date E D Control: Current quote first position Test: The IRS is expecting “many filings” to be in error at some level. No one can just blow off the law, but there will be something of a pass on reporting penalties as long as it can be demonstrated that effort went into compliance. Walter Fagan Control: What’s dominating the Employee Benefits agenda in 2016? Get the answers to the big questions → Test A: What’s dominating the Employee Benefits agenda in 2016? Get expert insights and tools you can use in the year ahead → Control: current template design Test: Test: plain text format that will appear personalized including 1. Dear [name] 2. Sincerely [regional EB president] Control: no nurture email Test: CMnurture series to those who completed ACA nurture Ems Control: always on Test A: second panel of ACA site Test B: new contact form to be designed for campaign Control: no button/SMbuttons Test: remove SMbuttons and add contact us button that goes to contact a rep form Control: https://www.hubinternational.com/employee-benefits/contact-us/ Test: modal pop-up Control: Current copy Test: Headline: Optimize your Employee Benefits in 2016 Box 1: ACA Reporting & Verification Audits Box 2: Cost Management Strategies Q1 Campaign Site email Live 9-Feb T Complete 4-Feb 1EB Always On HP Interstitial Link clicks A/B Q1 ACA Campaign Site ACAPanel Jack Quote 1 Share button clicks A/B TBD T CM nurture to ACAlist Increase lead pipeline A/B email Live 9-Feb T CM nurture email #2 template Email CTR A/B On-Hold Contact Us button location Button clicks A/B Q1 ACA Campaign Site ACAAd words Landing page test Form Completion Rate A/B/C Ad words Live 7-Mar Complete 22-Feb 11 T Contact Rep FormDesign - sub pages Form Completion Rate A/B Always On Site Live 9-Mar L1 page CTA Link clicks A/B Q1 L1 page Complete 9-Mar 6
- 19. Conduct regular testing/analytics meetings
- 20. Conduct regular testing/analytics meetings • Include core team and stakeholders • Use the testing plan as a guideline for the agenda o Review tests in progress • Determine any status changes o Review tests scheduled to launch o Discuss any new tests o Reprioritize based on new tests (if necessary) o Present deep dives from previous meeting questions • Use the meeting to generate new test and analysis ideas
- 21. Standardize the test process and embed it in business as usual
- 22. DK Cool CrayonTestapproved BuildtestAsset(s) needed? Add to testplan N Create asset(s) Y Review test Launchtest Wait 7days
- 23. DK Cool CrayonTestapproved BuildtestAsset(s) needed? Review test Create asset(s) LaunchtestAdd to testplan Y N Wait 7days Why wait 7 days? • Early results show greater variance. • Behavior often varies by day of week, want to ensure all days are represented.
- 24. DK Cool CrayonTestapproved BuildtestAsset(s) needed? Review test Create asset(s) LaunchtestAdd to testplan Y N Wait 7days Declare winner? Y Review meeting N Determinenext steps Updatetest plan End Suspend test DeclareWinner Continue testing Updatetest plan Updatetest plan End Revisetest? N Y Review results
- 25. Final thoughts
- 26. Start Slow
- 27. Start Slow Use basic tests to establish your process, then grow into complexity
- 28. Don’t panic if your tests don’t have significant results
- 29. You still learn from tests with no differences Don’t panic if your tests don’t have significant results
- 30. Implemented roles and procedures to create, execute, and evaluate tests AND are actively communicating results to drive change.
- 31. Questions?
- 32. THANK YOU