What does Netflix, NTT and Rubicon Project have in common? Apache Druid.
3 likes266 views
The document discusses the use cases and architecture of Apache Druid, a new operational data store designed to address analytics challenges such as scale, speed, and complexity. It highlights how Druid has replaced Redshift for various analytics tasks across major telecommunications and advertising companies, showcasing significant performance improvements and capabilities to handle vast amounts of data. Key figures emphasize Druid's capacity for managing trillions of events and rapid data processing across numerous global users.
What does Netflix, NTT and Rubicon Project have in common? Apache Druid.
- 1. NETFLIX + NTT + RUBICON PROJECT USE CASES
- 2. WHO AM I ▸ Dir. of Solutions Engineering, Imply ▸ Author: Virtualizing Hadoop ▸ 11 years of experience in distributed systems, big data platforms, cloud computing
- 3. AGENDA ▸ The analytics challenges it solves ▸ Use Cases ▸ Architecture
- 5. APACHE DRUID ▸ New class of Operational Data Store ▸ Solves the following analytics challenges ▸ scale ▸ speed ▸ grain complexity ▸ high dimensionality ▸ concurrency ▸ freshness
- 6. USE CASES
- 8. SOME NUMBERS ▸ 160 Billion events per day ▸ 190 countries ▸ 300 million devices ▸ Thousands of Druid users ▸ 100s of Druid nodes
- 9. CHALLENGES ▸ RedShift was used as the backend for their ad-hoc aggregated analytics dashboard ▸ It’s slow ▸ Cannot support longer data retention ▸ Cannot support a lot of dimensions ▸ Apache Druid replaced RedShift
- 10. USE CASES ▸ AWS capacity planning ▸ Payment analysis ▸ Algorithm comparison ▸ Security ▸ Client performance / Quality of Experience(QoE)
- 13. HIGHLIGHTS ▸ 4th largest telecommunication in the world ▸ Provides high speed, high capacity IP communication services for ▸ Europe ▸ North and South America ▸ Asia ▸ Oceana
- 15. CHALLENGES ▸ Legacy netflow analytics system ▸ was a blackbox, difficult to troubleshoot and extend on both frontend and backend ▸ not scaling in a cost performant manner ▸ limited ad-hoc analysis
- 16. USE CASES ▸ Netflow analysis ▸ Capacity planning ▸ Traffic matrix analysis ▸ Inter-domain traffic analysis
- 19. SOME NUMBERS ▸ Thousands of external customers, publishers, DSPs across the globe ▸ Trillions of ad and bid requests quarterly ▸ 1K header bidding connections ▸ 40% growth in mobile ad spend Q2 2018 vs Q2 2017 ▸ 70% growth in video ad spend 1st half 2018 vs 1st half 2017 ▸ 300% growth of audio ad spend in Q2 2018
- 20. CHALLENGES ▸ Advertising traffic exponentially grew ▸ MySQL can only store 10% of data volume daily ▸ Scaling interactive analytics to a wide base of users was tough ▸ Cost, performance, timeliness
- 21. DRUID NUMBERS ▸ >2TB data per hour to Druid ▸ <500ms average response time ▸ >1 Trillion events per day ▸ Thousands of users across the globe
- 22. ARCHITECTURE
- 24. SEGMENT ▸ Highly optimized storage unit ▸ Highly compressed bitmap indexes ▸ 150MB - 700MB size ▸ Determines parallelism ▸ Read in memory ▸ No contentions between read and writes ▸ 10x - 75x storage space savings
- 25. DATA MODEL ▸ Roll ups ▸ Approximation algorithms ▸ Segment granularity ▸ Query granularity ▸ Metrics ▸ Bitmap type (concise vs roaring)
- 26. QUERY OPTIMIZATION ▸ Threads ▸ Heap ▸ Horizontal scaling ▸ topN vs groupby ▸ datasketches ▸ splitting data sources for targeted queries
- 27. JOIN THE COMMUNITY Druid community site (current): http://druid.io/ Druid community site (new): https://druid.apache.org/ Imply distribution: https://imply.io/get-started
- 28. TRY THIS AT HOME