Getting testing right
0 likes389 views
This document provides a practical guide to testing in direct marketing. It discusses why testing is important, key concepts like sampling distributions and statistics, and outlines a structured testing lifecycle approach. This includes developing an annual testing strategy to prioritize questions, designing individual tests by calculating appropriate sample sizes and coding treatments, executing the test, evaluating results using statistical analysis to determine significance, and building insights to update strategy. Cautionary tales highlight the importance of testing value, sample size, evaluating multiple variables, and setting appropriate expectations. The overall goal is to make evidence-based marketing decisions through incremental, well-planned testing.
Getting testing right
- 1. Getting Testing Right: A Practical Guide to Testing in Direct Marketing IoF DM and Fundraising - March 2013
- 2. About me Richard Hughes Marketing Data Team Manager Previously: Data Planner at Bluefrog Data Analyst at Good Agency / Cascaid Database Administrator at Crusaid (AIDS/HIV charity)
- 3. Objectives • Talk through some of the finer points of testing • the strategy side and the data techie side • Both are really important! • A brain dump of everything I’ve learnt about testing • Advice on best practice • Show that it can be exciting • Inspire you to think about testing you can do
- 4. Why talk about testing? – Not enough people talk about how to do it – There is little info on the web for DM – I’ve seen it go wrong – Used well, testing can be very powerful, requires some thought and planning!
- 5. Definitions Split testing, or A / B testing, is when an audience is split into two or more groups and given different treatments in order to determine the most effective treatment
- 7. Why Test Anyway? How many communications How much should should we send we ask our throughout the supporters to year? donate? Which creative should we choose? Which Email Subject gets the best open rate?
- 8. Why Test Anyway? What stationery types perform the What’s the best? More money best time to on more expensive send a pack? packs? Who is the best signatory? How do cash appeals affect regular giving attrition rates?
- 9. Marketing Triangle Audience Elements that affect results Timing Message
- 10. The Data Pyramid
- 11. Gut Reaction Versus Evidence • Sometimes as experienced marketeers we know intuitively know the answers to some of these questions • But we want to move to situation with where we make evidence based decisions
- 12. Concepts
- 13. Concepts We are trying to find out if one approach is more likely to get better results than another •Testing is affected by probability –This means there is no guarantee that an approach will always “win” –We can say that it is more likely to win and we can say how confident we are
- 14. Sampling Distribution • When we test we … – measure a sample of our audience and use that to generalise about the rest of the database
- 15. The results can be put into a bell curve If we sample data from our database many times and treat in certain way we get a normal distribution
- 16. Two Curves Mathematical properties of the curve means we can use stats to determine how likely test a and B are different
- 17. Stats Summary • The response rate for each test is a normally distributed • We want to measure the difference in performance between a given treatment and the control. • The difference itself is a normally distributed random variable.
- 18. Structured Approach: Testing Life Cycle Testing Strategy Design Test Insight (Tactics) Evaluate Execute
- 19. Annual Testing Strategy • Good testing starts with careful thinking • Document what you want to find out • Check and reflect on your questions • Ensure that tests will deliver actionable results
- 20. Annual Testing Strategy •Build scenarios to understand where you are going to get the best value •Prioritise – focus on the best outcomes For UNICEF, this means focusing Select tests that will have most on the outcome that brings the impact, e.g. in mail packs, focus best result for children on outers rather than copy buried inside.
- 21. Cautionary Tales 1 • Testing can be expensive – Paying for different creative – Paying for different stationery to be printed – Ring fencing certain supporters from different comms is all expensive • This is an important consideration when thinking about the value of the test
- 22. Designing Tests: Sample Sizes • Think about volume for your test – You need sufficient quantity in your test • The sample needs have enough volume to be able to generalise about the population
- 23. Calculating Sample Sizes Deep Dark Statistics: • www.lucidview.com/sample_size.htm • The most useful online resource that has quite a technical explanation
- 24. Calculating Sample Sizes • Two things determine sample size – Existing Response Rate • Low number of responders means we need a bigger sample – Uplift of test • Small uplift means we need a bigger sample to see if there is a difference
- 25. Sample Sizes – Worked Example Take from http://www.testsignificance.com/
- 26. Testing more than one thing at once • Need to be careful, but can split by more than one test Treatment A Treatment B Totals X & Y Treatment X Segment 1 Segment 2 Treatment Y Segment 3 Segment 4 Totals for A & B
- 27. Cautionary Tales 2 • Be careful about testing too many things in one campaign – They can be difficult to manage – Cause confusion evaluating
- 28. Selected the Data • Once you you’ve decided on the volumes the next task is to make sure you split the data fairly – This means selecting two or more samples, ordering by factors that are important and selecting alternate rows – Do not take top / bottom half of spreadsheet
- 29. Coding • This might be a no brainer but ensuring the coding of A and B is set up right is important
- 30. Evaluating • We need to determin if two different results are significant • This means showing that we are 95% confident there is a significant difference • Quite a few websites that can help
- 31. Evaluating • If we are testing prompt amounts in packs we also need to test to see if the average gift is significantly different • We can use a T-test for this
- 32. Cautionary Tales 3 • Testing sometimes don’t tell us anything interesting • This is a lesson in setting expectations • Don’t say “we’re going to find out which is better” • Instead say “We’re going to find out if there is any difference”
- 33. Don’t forget to focus on Net Income Mailed Cost Response RR Income Net Average RoI 10000 £7,500 800 8% £14,400 £6,900 £18 1.92 10000 £11,000 1100 11% £19,800 £8,800 £18 1.80
- 34. Building Insight • Understanding what your tests means for your programme • Updating your strategy
- 35. Final Thoughts • Testing is about making incremental improvements • If you need more dramatic change then think about your overall fundraising strategy • Make sure you do lots of planning
- 36. Summary • What are your marketing questions? Testing Strategy • What are your priorities? • Calculate testing volume Test Design • Split data fairly, Code data appropriately Execute • Mail, email, phone Evaluate • Evaluate significance of results Build Insight • Update documentation on your audience insights
- 37. Any questions?
- 38. Thank you • Richard Hughes • richardh@unicef.org.uk