Where the Wild Bots are OPSNY June 2016
Download as PPTX, PDF5 likes3,656 views
The document discusses the challenges of ad fraud and ad blocking in digital media, highlighting how both negatively impact measurement and advertising dollars. Ad fraud results in significant financial losses for advertisers and publishers, with bots generating fake traffic that complicates analytics. Key insights include the need to measure ad blocking, bot activity, and viewability together to understand their collective impact on digital advertising efficacy.
![Where the Wild
Bots Are
June 2016
Augustine Fou, PhD.
acfou [at] mktsci.com
212. 203 .7239](https://image.slidesharecdn.com/wherethewildbotsareopsnyjune2016-160608114133/85/Where-the-Wild-Bots-are-OPSNY-June-2016-1-320.jpg)
![June 2016 / Page 8marketing.scienceconsulting group, inc.
linkedin.com/in/augustinefou
It is SO extremely profitable, bad guys won’t stop doing it
Source: https://hbr.org/2015/10/why-fraudulent-ad-
networks-continue-to-thrive
“the profit margin is 99% … [especially
with pay-for-use cloud services ]…”
Source: Digital Citizens Alliance Study, Feb 2014
“highly lucrative, and profitable… with
margins from 80% to as high as 94%…”](https://image.slidesharecdn.com/wherethewildbotsareopsnyjune2016-160608114133/85/Where-the-Wild-Bots-are-OPSNY-June-2016-9-320.jpg)
Where the Wild Bots are OPSNY June 2016
- 1. Where the Wild Bots Are June 2016 Augustine Fou, PhD. acfou [at] mktsci.com 212. 203 .7239
- 2. June 2016 / Page 1marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Brief Overview Ad fraud and ad blocking lower the effectiveness of digital media and messes up measurement. • Ad Fraud – quick review ‐ fraud bot activity (fake traffic, fake clicks) wastes ad dollars and messes up measurement • Ad Blocking – new, original data (AdMonsters study) ‐ bots don’t use ad blocking; ad blocking must be measured together with bots and viewability • Actions – looking ahead
- 4. June 2016 / Page 3marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Fraud continues upward as digital ad spend goes up Digital ad fraud Digital ad spend Source: IAB 2015 FY Report $ billions E High / Low Estimates
- 5. June 2016 / Page 4marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Ad fraud is a “double-whammy” for advertisers Messed Up AnalyticsWasted Ad Dollars Ad shown to bots are wasted Fake traffic, impressions, clicks are all recorded by analytics
- 6. June 2016 / Page 5marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Ad fraud is a “QUAD-whammy” for good publishers 2. “Bottom line” profitability squeezed 1. “Top line” ad revenue stolen 4. Reputations ruined by bad guys covering tracks 3. Ad blockers further reduce ad revenue
- 7. June 2016 / Page 6marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Fraud siphons 1/2 of dollars out of ad ecosystem Advertisers “ad spend” in digital is $60B in FY2015 Publishers are left with only 1/2 of the dollars Bad Guys siphon 1/2 of ad spend OUT of the ecosystem Ad dollars are being siphoned OUT of the ecosystem into the pockets of the bad guys 1/2 1/2 Users use ad blocking and need to protect privacy
- 8. June 2016 / Page 7marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Bad guys follow the money – CPM, CPC fraud Impressions (CPM/CPV) Clicks (CPC) Search 32% 91% digital spend Display 12% Video 7% Mobile 40% Leads (CPL) Sales (CPA) Lead Gen $2.0B Other $5.0B • classifieds • sponsorship • rich media (86% in FY2014) Source: IAB 2015 FY Report (83% in FY2013)
- 9. June 2016 / Page 8marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou It is SO extremely profitable, bad guys won’t stop doing it Source: https://hbr.org/2015/10/why-fraudulent-ad- networks-continue-to-thrive “the profit margin is 99% … [especially with pay-for-use cloud services ]…” Source: Digital Citizens Alliance Study, Feb 2014 “highly lucrative, and profitable… with margins from 80% to as high as 94%…”
- 10. June 2016 / Page 9marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Two main types of fraud and how each is generated Impression (CPM) Fraud (includes mobile display, video ads) 1. Put up fake websites and load tons of ads on the pages Search Click (CPC) Fraud (includes mobile search ads) 2. Use bots to repeatedly load pages to generate fake ad impressions (hide the true origins to avoid detection) 1. Put up fake websites and participate in search networks 2. Use bots to type keywords to cause search ads to load and then to click on the ad to generate the CPC revenue
- 11. June 2016 / Page 10marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Bots are the cause of all automated ad fraud Headless Browsers Selenium PhantomJS Zombie.js SlimerJS Mobile Simulators 35 listed Bots are made from malware compromised PCs or headless browsers (no screen) in datacenters.
- 12. June 2016 / Page 11marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Any device with chip/connectivity can be used as a bot Traffic cameras turned into botnet (Engadget, Oct 2015) mobile devices webcams connected traffic lights connected cars thermostat connected fridge Security cams used as 400 Gbps DDoS botnet (Engadget, Jun 2016)
- 13. June 2016 / Page 12marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou What I heard (at Publishers Forum) “Ad fraud doesn’t affect us” “I wasn’t really aware of bots and fraud” “Our SSP has an anti-fraud vendor” “we checked, we have very low bots”
- 14. Bots and Bad Guys
- 15. June 2016 / Page 14marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Websites – spectrum from bad to good Ad Fraud Sites Click Fraud Sites 100% bot mostly human longtail mid-tail mainstream Sites w/ Sourced Traffic Piracy Sites “cash-out sites” “sites w/ questionable practices” Premium Publishers “good guys”
- 16. June 2016 / Page 15marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Bots – from easy-to-detect to advanced bots 10,000 bots observed in the wild user-agents.org bad guys’ bots 3% Dstillery, Oct 9, 2014_ “findings from two independent third parties, Integral Ad Science and White Ops” 3.7% Rocket Fuel, Sep 22, 2014 “Forensiq results confirmed that ... only 3.72% of impressions categorized as high risk.” 2 - 3% comScore, Sep 26, 2014 “most campaigns have far less; more in the 2% to 3% range.” bot list-matching “not on any list” disguised as normal browsers – Internet Explorer; constantly adapting to avoid detection
- 17. June 2016 / Page 16marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Premium publishers have lots of humans
- 18. June 2016 / Page 17marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Programmatic impressions look much different
- 19. June 2016 / Page 18marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Humans (blue) on ad networks vs good publishers Ad Networks Publishers
- 20. June 2016 / Page 19marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou End of month traffic and impressions fulfillment Traffic surge Impressions surge volume bars (green) Stacked percent Blue (human) Red (bots) red vs blue trendlines
- 21. June 2016 / Page 20marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Real traffic surges, human visits due to news Traffic surgesvolume bars (green) Stacked percent Blue (human) Red (bots) red v blue trendlines
- 22. June 2016 / Page 21marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Fraud Activities Mess Up Measurement
- 23. June 2016 / Page 22marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou http://www.olay.co m/skin-care- products/OlayPro- X?utm_source=msn &utm_medium=cpc &utm_campaign=Ol ay_Search_Desktop Bad guys easily hide fraud by passing fake parameters Click thru URL passes fake source “utm_source=msn” buy eye cream online (expensive CPC keyword) 1. Fake site that carries search ads Olay.com ad in #1 position 2. search ad served, fake click Destination page fake source declared 3. Click through to destination page
- 24. June 2016 / Page 23marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Bad guys fake KPIs, trick measurement systems Bad guys have higher CTR Bad guys have higher viewability AD Bad guys stack ads above the fold to fake 100% viewability Good guys have to array ads on the page – e.g. lower average viewability.
- 25. June 2016 / Page 24marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Bad guys’ bots can fake most quantity metrics click on links load webpages tune bounce rate tune pages/visit
- 26. June 2016 / Page 25marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Recognizing human vs bot traffic patterns Bot traffic is “programmed” so the amount of traffic is the same (red line, flat across) Human visit websites during waking hours, using search
- 27. June 2016 / Page 26marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Google analytics view of traffic from fraud source Despite cutting off the traffic from the fraud site, there was no change to the number of pledges and downloads, during the same period of time. 102,231 sessions 0 sessions goal event – no change “ … because bots don’t make donations!”
- 28. June 2016 / Page 27marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou AppNexus example – cleaned up 92% of impressions Increased CPM prices by 800% Decreased impression volume by 92% Source: http://adexchanger.com/ad-exchange-news/6-months-after-fraud-cleanup-appnexus-shares-effect-on-its-exchange/ 260 billion 20 billion > $1.60 < 20 cents “good for them; good for advertisers who buy from them”
- 29. June 2016 / Page 28marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Bad guys’ bots earn more money, more efficiently Higher bots in retargetingBots collect cookies to look attractive Source: DataXu/DoubleVerify Webinar, April 2015 Source: White Ops / ANA 2014 Bot Baseline
- 30. June 2016 / Page 29marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Fraud operations are massively scalable Cash out sites are massively scalableAuto create fraud sites with algos 131 ads on page X 100 iframes = 13,100 ads /page Stacked redirects (e.g. dozens) Known blackhat technique to hide real referrer and replace with faked referrer. Example how-to: http://www.blackhatworld.co m/blackhat-seo/cloaking- content-generators/36830- cloaking-redirect-referer.html Thousands of requests per page
- 31. The Connection to Ad Blocking
- 32. June 2016 / Page 31marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Humans block ads; fraud bots don’t High human samples High bot samples 17% blocked 42% blocked 1% blocked 3% blocked
- 33. June 2016 / Page 32marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Humans use ad block; ads served to non-blocking bots Total Internet Users – 285 millionNon-Human Traffic adblocking humans Total Human Users – 120 million Adblock Users (humans) – 50 million U.S. Only Source: eMarketer 2016 estimate Source: Distil Networks 2015 170 million 50 million 70 million non-adblocking humans Source: PageFair / Adobe 2015 “subtracting adblocking humans, your programmatic ads are served to a population that is disproportionally (71%) non-human.”
- 34. June 2016 / Page 33marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Blocking, bots, viewability must be measured together bots (White Ops) viewability (Moat) adblocking (PageFair) “fraud sites with lots of bots also have very high viewability” “sites with lots of bots have abnormally low adblocking” (bots don’t block ads) “sites that cheat have abnormally high viewability and low ad blocking”
- 35. June 2016 / Page 34marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Change perspective to focus on positive/reliable human visible loaded
- 36. June 2016 / Page 35marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou AdMonsters Publishers Study
- 37. June 2016 / Page 36marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Desperately seeking high “LVH” ad inventory human visible loaded
- 38. June 2016 / Page 37marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Publishers participating in study - examples
- 39. June 2016 / Page 38marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou More great publishers who participated in study • 5+2 pattern visible; lower traffic overnight too • humans (blue) much higher than bots (red)
- 40. June 2016 / Page 39marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Publisher site with great content and humans |A| ad loaded 64% |B| visible 86% |C| human 89% 57%
- 41. June 2016 / Page 40marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou By contrast, impressions served on ad networks |A| ad loaded 23% |B| visible 19% |C| human 39% 4%
- 42. June 2016 / Page 41marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Examples of widely varying LVH measurements |A| ad loaded 81% |B| visible 92% |C| human 91% 77% |A| ad loaded 58% |B| visible 66% |C| human 71% 27% Publisher (High LVH) Publisher (Low LVH) |A| ad loaded 55% |B| visible 60% |C| human 44% |A| ad loaded 35% |B| visible 48% |C| human 38% Ad Network (High) Ad Network (Low) 6% 12%
- 43. June 2016 / Page 42marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Current industry level view – can be more accurate Source: Terence Kawaja @tkawaja – Digital Media Summit, May 2016
- 44. June 2016 / Page 43marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Programmatic traffic – bots, ad blocking, viewability Non-Human Traffic (NHT) HUMAN VISITORS List-matchbotdetection Adblockedbyhumanuser simplebots,crawlers advanced bots (mouse, scroll, click) humans tricked • invisible ads • domain spoofing • site bundling • ad injection • pixel stuffing • cookie cloning • clickjacking • sourced traffic • arbitrage • click bait • ad carousel ad loaded, visible, human (LVH) ads served advertiser VALUE advertiser WASTE “cash-out sites” “sites w/ questionable practices” “good guys”
- 45. June 2016 / Page 44marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Premium publishers – LVH (loaded, visible, human) (NHT) HUMAN VISITORS List-matchbotdetection Adblockedbyhumanuser simplebots,crawlers advancedbots ads served advertiser VALUE ad loaded, visible, human (LVH)
- 46. June 2016 / Page 45marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou AdMonsters Publishers Study – Class of May 2016 AdMonsters Publishers Study • 30 days, directly measured • 30 publishers/sites • 1 billion pageviews • ocean of blue
- 47. June 2016 / Page 46marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Take Action Now
- 48. June 2016 / Page 47marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Challenge all assumptions • mobile ad blocking is lower – perhaps, but it is also possible that it is due to more incomplete measurement in mobile • desktop ad blocking is low – but this may be due to more bots visiting (bots don’t use ad block) • programmatic ads have higher CTR – this may be due to bots creating fake clicks to trick you into sending them more money • fraud is in the lowest cost inventory – no, in fact there is much more fraud in the highest CPM ads like video ads • ads are not served if ad block is on – some ad blockers now call the ad to be served, then suppress it from displaying • viewability vendor takes care of it – viewability is supposed to mean no IVT and no ad blocking; it doesn’t actually, ask about it.
- 49. June 2016 / Page 48marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Top Advertiser Concerns
- 50. June 2016 / Page 49marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou About the Author
- 51. June 2016 / Page 50marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Dr. Augustine Fou – Recognized Expert on Ad Fraud 2013 2014 2015 SPEAKING ENGAGEMENTS / PANELS 4A’s Webinar on Ad Fraud AdCouncil Webinar on Ad Fraud TelX Marketplace Live ARF Audience Measurement / ReThink IAB Webinar on Ad Fraud / Botnets AdMonsters Publishers Forum / OPS
- 52. June 2016 / Page 51marketing.scienceconsulting group, inc. linkedin.com/in/augustinefou Harvard Business Review – October 2015 Excerpt: Hunting the Bots Fou, a prodigy who earned a Ph.D. from MIT at 23, belongs to the generation that witnessed the rise of digital marketers, having crafted his trade at American Express, one of the most successful American consumer brands, and at Omnicom, one of the largest global advertising agencies. Eventually stepping away from corporate life, Fou started his own practice, focusing on digital marketing fraud investigation. Fou’s experiment proved that fake traffic is unproductive traffic. The fake visitors inflated the traffic statistics but contributed nothing to conversions, which stayed steady even after the traffic plummeted (bottom chart). Fake traffic is generated by “bad-guy bots.” A bot is computer code that runs automated tasks.