MRS Speaker Evening- hosted by the ADA Network
Download as PPTX, PDF0 likes2,895 views
The document provides information about using propensity score matching to evaluate the effect of advertising on a client's brand image. It discusses: - Using propensity scoring methodology to create a matched sample of "controls" (non-viewers) for "cases" (viewers) to help assess the impact of seeing an advertisement on having a positive brand image, while controlling for other factors. - Other potential applications of propensity score matching. - In conclusion, propensity score matching allows comparing ad viewers to non-viewers who have been matched on characteristics to help isolate the effect of viewing the ad from other influences on brand perceptions.
MRS Speaker Evening- hosted by the ADA Network
- 1. supported by MRS Speaker Evening hosted by the ADA Network
- 2. •CPD and recognition •Mentoring 10 Speaker events 6 Webinars 50 e-books 100s of journals CPD •Accreditation Specialist briefings Bespoke training Government rosters TPS exemption •Annual Conference (Impact) •MRS •Code of •Conduct •Sector insight and knowledge •MRS Awards, Excellence and Oppies•Standards and regulation •Standards, •Policy, thought •leadership
- 3. For 60 years the MRS Code of Conduct has been the foundation of good research. MRS Code of Conduct The Code helps to protect providers, buyers and respondents Research activities are rooted into ethical considerations. Researchers look at two major elements • doing right and • preventing harm, through protecting privacy, obtaining consent. Generally speaking, an understanding of ethics is rooted and should be seen as an essential part of acknowledging the researcher's identity, their own position in the research and as a means of justifying their research approach Ethics
- 4. Evidence-based resource allocation 27 June 2019 Alex Maksimov, Analytical Consultant, Google UK alexmaksimov@google.com Application in Paid Search marketing
- 5. Growing accountability in digital marketing
- 6. Growing accountability in digital marketing Evidence-based resource allocation
- 7. How do I define return on investment? How do I model return on investment? How do I allocate investment optimally? Three key questions
- 8. How do I define return on investment? How do I model return on investment? How do I allocate investment optimally? Three key questions
- 9. Marketing return on investment is poorly defined
- 10. ROI Best practice from finance? ROA NPV IRR ROS EVA GM OM EBIT EBITDA NOPAT ROE Return on Investment Return on Assets Return on Sales Return on Equity Net Present Value Internal Rate of Return Gross Margin Operating Margin Economic Value Added Earnings Before Interest and Taxes Earnings Before Interest, Taxes, Depreciation and Amortisation Net Operating Profit After Tax
- 11. Should marketing strive for this level of precision?
- 12. Probably not... Should marketing strive for this level of precision?
- 13. Probably not... Should marketing strive for this level of precision? ...but a healthy mix of metrics is essential Absolute vs Relative Revenue vs Profit Average vs Marginal
- 14. $2K$10K Absolute vs Relative ROAS 5x Investment Revenue $50K Campaign A 10x Campaign B $20K
- 15. $2K$10K Absolute vs Relative ROAS 5x Investment Revenue $20K Campaign A 10x $20K Campaign B Extra revenue $30K
- 16. Revenue vs Profit Campaign A Campaign B $10K$10K $5K $5K $5K $5K Revenue Marketing Investment ROAS 2x 2x
- 17. Revenue vs Profit Campaign A Campaign B $10K$10K $5K $5K $1K $4K $4K $1K Revenue Profit Product Cost Marketing Investment ROAS 2x 2x
- 18. Revenue vs Profit Campaign A Campaign B $10K$10K $5K $5K $1K $4K $4K $1K Revenue Profit Marketing Investment ROAS 2x 2x Profit Difference $3K Product Cost
- 19. Average vs Marginal Campaign A Campaign B Average ROAS 5x 4x Average ROAS 3x 6x
- 20. Average vs Marginal Campaign A Campaign B Average ROAS 5x Marginal ROAS 4x Average ROAS Marginal ROAS 3x 6x
- 21. 1x Average vs Marginal Campaign A Campaign B Average ROAS 5x Marginal ROAS 4x Average ROAS Marginal ROAS 3x 6x ROAS Difference
- 22. A healthy mix of investment metrics for better decision making Common sense... ...but often not present in marketing reports Absolute vs Relative Revenue vs Profit Average vs Marginal
- 23. How do I define return on investment? How do I model return on investment? How do I allocate investment optimally? Three key questions
- 24. Start by gathering evidence
- 25. We are used to seeing historical data in a time series format
- 26. Supplement time series with relationships We are more interested in the relationship between marketing investment and business outcomes than trend over time
- 27. Start by modelling traffic as a function of marketing spend Investment Historical weekly observations Observation
- 28. Start by modelling traffic as a function of marketing spend Investment Historical weekly observations Observation Model
- 29. Start by modelling traffic as a function of marketing spend Decreasing returns to scale ● Non-linear relationship ● Concave ● Often power or logarithmic functions Investment Historical weekly observations Observation Model
- 31. Overlay business metrics INVESTMENT INVESTMENT x CONVERSI ON RATE
- 32. Overlay business metrics INVESTMENT INVESTMENT INVESTMENT x CONVERSI ON RATE x ORDER VALUE
- 33. Ability to anticipate business outcomes A flexible model to estimate business outcomes at ● Different levels of spend ● Different resource allocations
- 34. Ability to make better decisions Investment A flexible model to estimate business outcomes at ● Different levels of spend ● Different resource allocations
- 35. Ability to make better decisions Investment +11 +5 +200 +200 A flexible model to estimate business outcomes at ● Different levels of spend ● Different resource allocations
- 36. Ability to allocate resources optimally Investment A flexible model to estimate business outcomes at ● Different levels of spend ● Different resource allocations Campaign A Campaign B
- 37. Ability to allocate resources optimally Investment A flexible model to estimate business outcomes at ● Different levels of spend ● Different resource allocations Campaign A Campaign B Current Spend Current Spend
- 38. Ability to allocate resources optimally Investment A flexible model to estimate business outcomes at ● Different levels of spend ● Different resource allocations Campaign A Campaign B Current Spend Current Spend Marginal Return Marginal Return
- 39. How do I define return on investment? How do I model return on investment? How do I allocate investment optimally? Three key questions
- 40. If allocation is even remotely optimised, the next marketing dollar should result in fewer sales than the previous Optimally allocating a finite resource is at the heart of business strategy
- 41. Optimisation is achieved by making trade-offs between alternatives Investment Campaign A Campaign B Current Spend Current Spend Optimal Spend Optimal Spend
- 42. But most advertisers are faced with too many choices Response curves for different campaigns
- 43. Linear programming definition Linear programming in its most basic form
- 44. Linear programming definition Linear programming in its most basic form A method to achieve the best outcome in a model
- 45. Linear programming definition Linear programming in its most basic form Objective function What do we want the algorithm to do? A method to achieve the best outcome in a model
- 46. Linear programming definition Linear programming in its most basic form Objective function Constraints What do we want the algorithm to do? Are there any limits or threshold that need to be taken into account? A method to achieve the best outcome in a model
- 47. Linear programming application in digital marketing Maximise Conversions Maximise Revenue Maximise Profit Objective function Maximise Traffic Allocate investment to
- 48. Linear programming application in digital marketing Maximise Conversions Maximise Revenue Maximise Profit Budget < $XM ROAS > X CPC < $X.XX Constraints Objective function CPA < $X.XX Maximise Traffic Subject to Allocate investment to
- 50. Maximise Traffic Maximise Conversions Maximise Revenue Maximise Profit Investment ($M) 10.00 10.00 10.00 10.00 Objective functions An example Input
- 51. Maximise Traffic Maximise Conversions Maximise Revenue Maximise Profit Traffic (M) Conversions (K) Revenue ($M) Profit ($M) Investment ($M) 10.00 10.00 10.00 10.00 ROAS (x) CPC ($) CPA ($) ROI (x) Objective functions An example Constraints Outcomes Input
- 52. Maximise Traffic Maximise Conversions Maximise Revenue Maximise Profit Traffic (M) 28.25 27.63 25.83 26.47 Conversions (K) 56.01 63.42 57.36 56.66 Revenue ($M) 35.92 36.43 38.13 35.47 Profit ($M) 14.44 15.02 14.59 15.16 Investment ($M) 10.00 10.00 10.00 10.00 ROAS (x) 3.59 3.64 3.81 3.55 CPC ($) 0.35 0.36 0.39 0.38 CPA ($) 178.53 157.69 174.35 176.5 ROI (x) 1.44 1.50 1.46 1.52 Objective functions Constraints Outcomes Input You’re already making trade offs, even if you don’t realise it
- 53. Maximise Traffic Maximise Conversions Maximise Revenue Maximise Profit Traffic (M) 28.25 27.63 25.83 26.47 Conversions (K) 56.01 63.42 57.36 56.66 Revenue ($M) 35.92 36.43 38.13 35.47 Profit ($M) 14.44 15.02 14.59 15.16 Investment ($M) 10.00 10.00 10.00 10.00 ROAS (x) 3.59 3.64 3.81 3.55 CPC ($) 0.35 0.36 0.39 0.38 CPA ($) 178.53 157.69 174.35 176.5 ROI (x) 1.44 1.50 1.46 1.52 Objective functions Constraints Outcomes Input You’re already making trade offs, even if you don’t realise it
- 54. Maximise Traffic Maximise Conversions Maximise Revenue Maximise Profit Revenue ($M) 38.13 35.47 Profit ($M) 14.59 15.16 Investment ($M) 10.00 10.00 ROAS (x) 3.81 3.55 ROI (x) 1.46 1.52 Objective functions Constraints Outcomes Input You’re already making trade offs, even if you don’t realise it
- 55. How do I define return on investment? Recap and recommended actions ❏ Audit your mix of metrics ❏ Include absolute, profitability and marginal metrics
- 56. How do I define return on investment? How do I model return on investment? Recap and recommended actions ❏ Audit your mix of metrics ❏ Include absolute, profitability and marginal metrics ❏ Look beyond time series ❏ Identify relationships between variables ❏ Construct flexible models for decision making
- 57. How do I define return on investment? How do I model return on investment? How do I allocate investment optimally? Recap and recommended actions ❏ Audit your mix of metrics ❏ Include absolute, profitability and marginal metrics ❏ Look beyond time series ❏ Identify relationships between variables ❏ Construct flexible models for decision making ❏ Utilise linear programming ❏ Make the right trade-offs for your business
- 58. Thank you
- 59. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only Using Propensity Score Matching to evaluate the effect of advertising on our client’s brand image Presentation to the MRS Andrew Zelin, Ipsos MORI 27 June 2019
- 60. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 60 What we will cover • Background to the Problem • Propensity Scoring Methodology • Using the scores to create a matched sample of controls for the cases • Other applications • Conclusion
- 61. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 61 Existing Relationships II A leading digital advertising company needed to assess whether seeing one of the their advertisements is associated with having a positive image of the brand Are those who have seen one of the advertisements, more likely to have a positive image of the brand than those that have not?
- 62. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 62 Overview I BUT…people who have higher levels of skills in digital areas and / or use this as part of their careers (ie are more “tecky”) • are more likely to see the ad • and are also more positive towards the brand than people who are not
- 63. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 63 Existing Relationships I Seen ad +ve Image of Brand “Teckyness” of respondent Effect of interest Confounder Confounder
- 64. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 64 Existing Relationships II Seen ad +ve Image of Brand “Teckyness” of respondent So from this, we cannot tell easily how much of the positive image is due to seeing the ad v being a user of digital technology.
- 65. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 65 Overview II “Propensity Score” Matching A Matched Evaluation to compare the ad viewers (cases) with the non-viewers (controls). • through the creation of a “Propensity Score” (PS) for each respondent in terms of how likely they are to have seen the ad, • followed by appropriate matching of the cases and controls on this PS and hence levels of “Teckyness”. Ad viewers (cases): Non-viewers (controls): Brand Image
- 66. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 66 Overview III • By doing this, we were able to delineate how much of the increase in perceived brand image for those who viewed the ad over those that did not, was genuinely down to viewing the ad, rather than simply being more “tecky”. • ….other methods tested – eg at the analytical stage / by bringing these into the model using covariates, but that has not been successful – eg variance inflation Brand Image Ad viewers (cases): ? Non-viewers (controls): ?
- 67. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 67 Creating a Propensity Score: Overview • Set up an “Evaluation” methodology using Propensity Scores (PS) • “Propensity Score Matching” - first documented by Robin in 1983. https://en.wikipedia.org/wiki/Propensity_score_matching https://academic.oup.com/biomet/article/70/1/41/240879 STEPS: 1. Calculate a PS for each respondent (viewers and non- viewers); 2. Find the closest match non-viewer(s) on PS for each viewer; 3. Derive a weight for each non-viewer based on how well it generally matches to all viewers; 4. Create a fresh sample of non-viewers from the original viewers with selection probability prop. to this weight this fresh sample will be matched to the viewers on the basis of “teckyness”.
- 68. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 68 Creating a Propensity Score: Step 1 0.00 Propensity (to be an ad-viewer) Score 1.00 0.50 Definition of Propensity Score (PS): • An overall “teckyness” score based on a composite of related variables, • These variables have different relative weights / influences on this, such that this PS correlates as strongly as possible with viewing the ad. Logistic regression modelling was carried out: • To determine which “techiness” questions are correlated with being an ad viewer v not an ad viewer • to derive their coefficients / influences; • and to calculate a PS for every respondent; PS is driven by ones responses to: • (L_T1) Digital skills; • (GQ2) Attitudes to technology; Understanding how to use technology is important for your career; PS = 1/(1+exp(3.294-1.05*L_T1-1.982*GQ2)
- 69. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 69 Ad viewers (cases): Non-viewers (controls): Creating a Propensity Score: Step 2 (Closest match) • 500 cases • 900 controls • 1:1 match of 500:500 means losing 400 controls • How can we still use these 400?
- 70. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 70 Ad viewers (cases): Non-viewers (controls): • What was actually done was: • Take each case / ad viewer in turn; • Create a selection weight for each control, based on how similar each non- viewer Propensity Score (PSnv) is to the viewer PSv - ie (1/PSv-PSnv)^2) • If the difference between PSv and PSnv exceeds 0.05, set weight to zero. • Repeat for each viewer; • After that, take each non-viewer, and summate all of the weights received from all of its matches from each viewer; ….and this total is the “selection weight” needed for each non-viewer 0.05 Creating a Propensity Score: Step 3 (Weights)
- 71. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 71 • Control respondents (non-viewers) with high selection weights are those which are well matched / were under-represented when aligning the cases and controls for “teckyness”, and hence need to be weighted up. • Controls with low selection weights are those which are poorly matched / were over-represented and need to be weighted down. The final step involves a fresh sample of controls being selected through “Bootstrapping” from the current pool of controls, …matching the cases in terms of “teckyness” / PS. To achieve a balanced sample such that comparisons on brand image scores can be made by directly comparing all of the cases with the re-sampled controls To select a balanced sample of non-viewers to match the viewers Creating a Propensity Score: Step 4 (Re-selection)
- 72. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 72 Comparing the Propensity Scores of the samples 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 0.036 0.057 0.059 0.091 0.093 0.096 0.141 0.145 0.148 0.212 0.217 0.222 0.313 0.319 0.435 Percentofsample Propensity score Comparing the distributions of PS for the not-seen ad sample before and after PW, vs PS of seen ad group Not_seen_Raw_sample Not_seen_matched_sample Seen_ad
- 73. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 73 Comparing the Propensity Scores of the samples 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 0.036 0.057 0.059 0.091 0.093 0.096 0.141 0.145 0.148 0.212 0.217 0.222 0.313 0.319 0.435 Percentofsample Propensity score Comparing the distributions of PS for the not-seen ad sample before and after PW, vs PS of seen ad group Not_seen_Raw_sample Not_seen_matched_sample Seen_ad
- 74. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 74 Comparing the Propensity Scores of the samples 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 0.036 0.057 0.059 0.091 0.093 0.096 0.141 0.145 0.148 0.212 0.217 0.222 0.313 0.319 0.435 Percentofsample Propensity score Comparing the distributions of PS for the not-seen ad sample before and after PW, vs PS of seen ad group Not_seen_Raw_sample Not_seen_matched_sample Seen_ad Note that in terms of PS, the profile of the matched sample of controls (blue) aligns much better to that of the cases (green), than does the unmatched controls (red)
- 75. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 75 Comparing Brand images before vs after matching 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% Mean PS Brand favourability <Google> Brand Advocacy <Google> WOM Advocacy <Google> Closeness to <Google> How well does Google offer P&S Do you use Google offer P&S Percentofsample Propensity score Comparing Brand images of those who have seen the ad against those who have not; before and after matching (n=2,000) Not_seen_ad - Without_PW Not_seen_ad - With_PW Seen_ad
- 76. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 76 Comparing Brand images before vs after matching 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% Mean PS Brand favourability <Google> Brand Advocacy <Google> WOM Advocacy <Google> Closeness to <Google> How well does Google offer P&S Do you use Google offer P&S Percentofsample Propensity score Comparing Brand images of those who have seen the ad against those who have not; before and after matching (n=2,000) Not_seen_ad - Without_PW Not_seen_ad - With_PW Seen_ad Note that the “matching” closes the gap in brand image between the views and the non- viewers, but even adjusting for this – the ad still has a positive effect on image overall
- 77. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 77 Comments and learnings What will be done in future analyses: • Use and test a longer list of variables in the PS model – eg age, gender, ethnicity; • Validate to compare matches of PS, using single variables used in PS (eg digital skils), as well as the overall PS; • The value of the final “Bootstrap” re-sampling is questionable. Is it wasteful on respondents? Are we better to use 900 weighted controls?
- 78. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 78 Other Applications Data Fusion – Merging large databases Clinical Trials / Evaluating Patient Pathways Mixed mode surveys
- 79. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only 79 Conclusions and Impact on the end-Client • Propensity Scoring and Weighting methods may be effective in evaluations to eliminate confounding effects; • These scores can be used to create matches, which in turn can generate selection weights needed to produce balanced samples • Propensity Scoring and Weighting constitute robust and objective methodologies to be developed using evaluations to assess the true impact of advertisement exposure on brand image. • A number of standard Key Drivers analyses had been carried out on brand image taking in all of the possible confounding factors, although none really got to the heart of the matter as to whether seeing the ad is associated with a positive image, adjusting for respondents’ levels of “teckyness • The method is Relatively easy to implement; • Has a number of other implications and possible uses.
- 80. Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only Thank you for your attention Andrew Zelin Lead Statistician Healthcare, Ipsos
- 81. QUANTIFYING THE QUALITATIVE MRS ADVANCED ANALYTICS
- 82. QUANTIFYING THE QUALITATIVE WHO ARE WE ▸Rob Bramwell, Consultant from CDO Partners ▸Brighton-based data and analytics organisation ▸Specialise in helping companies improve through the use of data ▸Work with organisations from startups to global enterprises
- 83. QUANTIFYING THE QUALITATIVE WHY ARE WE HERE ▸Talk about our experiences working with customers to help them programmatically analyse and process qualitative data ▸Share some frameworks and approaches we’ve used ▸Give some examples of customer projects ▸Hopefully help you see how you can quantify qualitative data
- 85. QUANTIFYING THE QUALITATIVE ANALYTICAL PATTERNS ▸What ▸Usually a simple measure - what have we sold ▸How ▸Adding dimensions to those measures ▸Trends, timelines ▸Why ▸Statistical analysis ▸Qualitative analysis
- 86. QUANTIFYING THE QUALITATIVE ANALYTICAL PATTERNS ▸Useful in a wide range of applications ▸HR and performance - why are people leaving ▸Research and policy - what should we be driving ▸Customer interaction - why are people complaining ▸Sales and marketing - why are customers churning
- 88. QUANTIFYING THE QUALITATIVE MAKING DATA WORK IN HEALTHCARE▸Recently been working with a range of government organisations and NGO using qualitative and quantitative data ▸Specifically in healthcare where a lot of the data is in a range of formats ▸Very difficult to automate the capture, ingestion and analysis of this data ▸Makes it incredibly time consuming
- 89. WE DON’T HAVE ENOUGH ANALYSTS, THE DATA IS TOO MUCH TO ANALYSE AND WE’RE BEHIND ON RESEARCH Head of Intelligence - Healthcare NGO QUANTIFYING THE QUALITATIVE
- 90. QUANTIFYING THE QUALITATIVE ANALYTICAL PATTERNS ▸What is happening ▸We can see raw numbers but not in context easily ▸How is it happening ▸We can’t see trends or dimensionality as none of the data is joined together and comes in snapshots ▸Why is it happening ▸Interviews, notes, personal opinion, diaries…not sustainable
- 92. QUANTIFYING THE QUALITATIVE QUANTITATIVE WHAT AND HOW FIRST▸ Provides significant value ▸ Quantitative metrics ▸ Effort is involved in the integration of data sources and modelling of dimensions ▸ Data storytelling ▸ Gives you a baseline ▸ Points them at key areas to explore next
- 93. QUANTIFYING THE QUALITATIVE QUALITATIVE ▸ Identify data sources ▸ Patient reviews ▸ Published reports ▸ Government reports ▸ Regulatory ▸ News feeds ▸ Social media
- 94. QUANTIFYING THE QUALITATIVE QUALITATIVE ▸ Define your key performance questions ▸ What do we need to know ▸ Define your key analytical questions ▸ What would we like to know ▸ What hypotheses do we have ▸ Define a taxonomy ▸ Helps structure the filters and searches
- 95. QUANTIFYING THE QUALITATIVE QUALITATIVE ▸ We used a platform called Squirro ▸ Faceted search application ▸ Runs on Elastic and Python ▸ Considered other platforms, NLP and RPA style processing we have used elsewhere for similar use cases ▸ Healthwatch England ▸ EDF
- 96. QUANTIFYING THE QUALITATIVE BUILD APPLICATION S▸ Data pipelines ▸ Build automated pipelines ▸ Smart filters ▸ Build filters based on taxonomy to automate searching ▸ Discovery dashboards ▸ Define user stories and key questions
- 101. QUANTIFYING THE QUALITATIVE OUTCOMES ▸30% analyst time saved through automation ▸Backlog of 50% cases reduced to 0% ▸Volumes of processing increased ▸Research and policy analysis can be done within days not months
- 102. QUANTIFYING THE QUALITATIVE NEXT STEPS ▸Additional sources of data ▸More automation in data review and taxonomy ▸Predictive modelling on topics and clusters
- 104. QUANTIFYING THE QUALITATIVE GET IN TOUCH!▸ rob@cdo-partners.com ▸ www.cdo-partners.com ▸ @cdopartners
- 105. Questions to the panel