How TrafficGuard uses Druid to Fight Ad Fraud and Bots
0 likes159 views
TrafficGuard specializes in measurement verification and fraud prevention in digital advertising, significantly invalidating over 13 billion impressions, clicks, and events. The partnership with Druid and Imply enhances their operational efficiency by providing real-time operational data and analytics, while minimizing management overhead. This collaboration has led to improved performance metrics, including page load times under five seconds during high concurrency conditions.
How TrafficGuard uses Druid to Fight Ad Fraud and Bots
- 1. Real time intelligence to combat ad fraud TrafficGuard’s winning partnership with Druid & Imply April 2020 Raigon Jolly, Head of Analytics & Data Science, TrafficGuard 1
- 2. Virtual Druid Summit - September 2020 222 What is TrafficGuard? Measurement Verification Measurement, verification and fraud prevention for digital advertising. Invalid Traffic Prevention Reporting
- 3. Virtual Druid Summit - September 2020 333 What is TrafficGuard? 55% of the team comprises data scientists, analysts and engineers 13+ billion* impressions, clicks and events invalidated 5+ years of fighting ad fraud Listed on the Australian Securities Exchange (ASX: AV1) Global business headquartered in Perth, Australia
- 4. Virtual Druid Summit - September 2020 The challenging digital advertising landscape 1. Journey 2. Scale 3. Real time nature
- 5. Virtual Druid Summit - September 2020 The challenging digital advertising landscape Impression Click Conversion Event Post-Conversion Event 1. Journey 2. Scale 3. Real time nature
- 6. Virtual Druid Summit - September 2020 1. Journey 2. Scale 3. Real time nature The challenging digital advertising landscape
- 7. Virtual Druid Summit - September 2020 The challenging digital advertising landscape 1. Journey 2. Scale 3. Real time nature
- 8. Virtual Druid Summit - September 2020 The challenging digital advertising landscape 1. Journey 2. Scale 3. Real time nature
- 9. Virtual Druid Summit - September 2020 Customer Reporting Analytical layer ● Granularity of detail and providing log level insight helps us differentiate from competitors who only provide aggregated data to clients ● 14+ Reporting Subsections in product. ● Some having 18 different queries with comparisons on time period ● Concurrency: Peak Concurrency 1000 queries a second ● Low latency ● Streamlined development with SQL - helping is faster time to market
- 10. Virtual Druid Summit - September 2020 Query Optimisation ● Separate datasets for larger client, operational reporting, unified reporting ● Govern KPI in PIVOT, have role based access control, versioning of definition ● Precalculate into one field ● Define difference between NULL(eg “(not set)”) and empty fields ● Define data retention and multi tenancy early on ● Remove case statements/TIME EXTRACT/ REGEX in the group by clause Learnings
- 11. Virtual Druid Summit - September 2020 Query Optimisation ● Have sound Data dictionary and governance ● Keep name of field descriptive and maintaining lineage ● Adjustment process for handling immutability ● Clarity as proactive monitor system + AWS EC2 cloudwatch ● If the dimensions are only used for COUNT DISTINCT calculations, use theta sketches measures and remove the dimensions. ● Maintain performance history of common queries Learnings
- 12. Virtual Druid Summit - September 2020 Some useful slices ● SQL query ID - Point to product area ● Identity - who ? what type? ● Dimension - Common Used ● Num metric - Common Used ● Num complex metric - Precalculate at ingestion/upstream ● Num dimension - Precalculate at ingestion/upstream ● Duration ● Type ● P98 Latency - Cluster issues, Proactive monitoring, Leading Indicator Know thy queries using Clarity Learnings
- 13. Virtual Druid Summit - September 2020 Operational Data Store Real time detection operations layer Real-time operational data store provides us: ● Operational analytics visibility across the business ● Ability to quickly identify data quality issues ● Data profiling (Null, Missing, Unique, Spike and drops) ● Ability to test integrations in real time to ensure TrafficGuard is receiving correct and complete signal
- 14. Virtual Druid Summit - September 2020 Log, Threat hunting & Prototyping Analytical layer
- 15. Virtual Druid Summit - September 2020 TrafficGuard Feature Store with Druid aggregated data Streaming Layer
- 16. Virtual Druid Summit - September 2020 Source: VoltDB Ease of management, more economical One solution across the data lifecycle
- 17. Virtual Druid Summit - September 2020 TrafficGuard Partnership with Imply and Druid ● Multi year engagement ● Team can focus on last mile ● Imply takes care of rolling updates on Druid, Clarity and Pivot ● The consultation, guidance and proactive support from imply team is keeping our operation overhead to minimal
- 18. Virtual Druid Summit - September 2020 With Druid, TrafficGuard has achieved... Page load <5 secs @ high concurrency 70% analytics powered by Druid High dimensionality & cardinality an asset, not challenge Immediate and significant cost savings
- 19. Time for questions @TrafficGuardAI 19 Thank you! Apache Druid is an independent project of The Apache Software Foundation. More information can be found at https://druid.apache.org. Apache Druid, Druid, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.
- 20. November 2-4, 2020 San Francisco, CA druidsummit.org 20 Register Now for Druid Summit