Direct mail testing & key metrics nedma 2016
- 1. Direct Mail Testing & Key Metrics Presented by: Brett Jones May 17, 2016 2:30 – 3:20
- 2. Today We’ll Discuss § WHY its important to test § HOW to design a valid test § WHAT to test & how to measure results 1
- 3. WHY IT’S IMPORTANT TO TEST 2
- 4. 3 Ways to Make More Money More Donors • Acquisition Lists / Select criteria for Lapsed • Package/Creative • Lower initial ask • Upfront / back-end premium More Gifts per Donor • RFM segmentation to identify those donors most likely to give an additional gift • Follow up or additional appeals • Less aggressive ask string Higher Average Gift • RFM segmentation to identify those with higher previous gifts • More aggressive ask string 3 #4: Cutting Costs
- 5. The Testing Process Determine your objective Identify an opportunity Test your hypothesis Take action based on results 4 Possible Actions: 1. Retest 2. Rollout 3. Move on to another hypothesis
- 6. HOW TO DESIGN A VALID TEST 5
- 7. HOW to design a valid test What change do you want to create? Develop a hypothesis Isolate a variable & construct a valid test Execute Test Determine key metric Read results 6
- 9. Focus on the things that will make a difference List, 60%Creative, 20% Offer, 20% 8
- 13. Package Test in Acquisition: Action v. Memo QTY # Gifts % Resp. Avg. Gift Gross $ Net $ Net Cost to Acquire “Action” Control 29,905 384 1.28% $35.99 $13,819 ($4,723) ($12.29) “Memo” Test 29,901 208 0.70% $39.19 $8,152 ($9,999) ($48.07) 99% Confidence Level What change did we want to create? What was the hypothesis? How would we determine success? How did we construct a valid test? What won? What might we do next?
- 15. Which format won? Segment QTY # Gifts % Resp. Avg. Gift Gross $ Gross Rev/M Control OE 13,033 229 2.29% $48.09 $14,380 $1,103 Picture Window 13,033 373 2.86% $44.36 $16,547 $1,270 99% Confidence Level What change did we want to create? Our hypothesis. What variable did we isolate? How would we determine success? What we will do next? Why / When to use this metric
- 17. Component Test in Acquisition: RE v. BRE Segment QTY # Gifts % Resp. Avg. Gift Gross $ BRE 9,931 97 0.98% $36.81 $3,571 RE 9,931 69 0.69% $41.09 $2,835 95% Confidence Level What change did we want to create? What was the hypothesis? What won? What we will do next?
- 18. Format Test: Attached vs. Separate Reply
- 19. Format Test: Attached vs. Separate Reply Segment QTY # Gifts % Resp. Avg. Gift Gross $ Cost/Piece Separate Reply 63,239 1,034 1.60% $66.87 $69,141 $0.36 Attached Reply 63,773 1,176 1.84% $75.64 $88,962 $0.30 99% Confidence Level What change did we want to create? What was the hypothesis? What about the increase in the average gift? What won? What we will do next?
- 20. OFFER TESTS 19
- 21. Longwood Gardens Can an offer be too rich? 20 Test Offer: 2 Guest Passes Control Offer: 2 Additional Guest Passes
- 22. Segment QTY # Gifts % Resp. Avg. Gift Gross $ Gross Rev/M Control: MORE passes 56,763 484 0.85% $75.63 $36,605 $645 Test : fewer passes 56,761 558 0.98% $74.98 $41,840 $737 What change did we want to create? What was the hypothesis? What would we measure to determine success? What variable did we isolate? What won? Why? 95% Confidence Level Longwood Gardens Offer Test
- 23. Any Amount Makes You a Member 22 Sometimes you need to evaluate results over a l-o-n-g period of time
- 24. Open Ask Test Results § Response Rate increased by 122% § Average Gift fell from $55 to $28 § Net Cost to Acquire a member cut in half § Minimal reduction in retention rate Year 2 23
- 25. LIST TESTS 24
- 26. Segment Test Segment QTY # Gifts % Resp. Avg. Gift Gross $ Net Rev. Recently Lapsed 9,682 139 1.44% $48.10 $6,686 $1,861 Longer Lapsed 21,681 197 0.91% $52.56 $10,354 ($451) Would you mail this group again? How would you decide?
- 27. Segment Test Segment QTY # Gifts % Resp. Avg. Gift Gross $ Net Rev. Net Cost to Acquire Acquisition 63,521 381 0.60% $46.49 $17,713 ($13,984) ($36.70) Recently Lapsed 9,682 139 1.44% $48.10 $6,686 $1,861 + Longer Lapsed 21,681 197 0.91% $52.56 $10,354 ($451) ($2.29) Would you mail this group again? Goal: More New Donors Key Metric: Net Cost to Acquire Longer Lapsed donor vs. Acquisition
- 28. Every acquisition mailing is an opportunity to tests new lists Performance Risk
- 29. A Balanced List Plan Test Lists 10% max Retests Approx. 10% Core Lists 80% of names
- 30. § List testing opportunity #1: Identify lists that are mediocre performers whose response may be increased with a tighter select List Segment Historical Resp. % Avg. Gift Habitat For Humanity/24 Mos. .87% $48.09 List Segment % Resp. Avg. Gift Habitat For Humanity/12 Mos. 1.01% $48.07 Current Performance Sample Test Results If the available universe doesn’t shrink too much, you may be better served by tightening the select criteria
- 31. § List testing opportunity #2: The other side of the coin Identify lists that are very strong performers that could generate additional members with an expanded select (larger universe) and comparable (or acceptable) response List Segment Mailing Universe Historical Resp. % Members World Wildlife Fund/12 Mos. 4,000 1.03% 41 World Wildlife Fund/12 Mos. Mailing Universe % Resp. Members World Wildlife Fund/24 Mos. 7,000 .91% 63 Current Performance Sample Test Results Sometimes the greater good is served by a lower response rate on a larger universe of prospects – ultimately delivering you more new donors.
- 32. Is an UNsuccessful test one that doesn’t give you the answer you expected?
- 33. What if a test loses? What if I’m “too small” to test?
- 34. 33 Remember…
- 35. Brett Jones Vice President, Fundraising DMW Fundraising bjones@dmwfundraising.com Thank you!