Deep dive into druid
- 1. Deep Dive Into Druid Sep 2017
- 2. Agenda ➔ Where is druid?? ➔ Columnar Databases / Distributed Database ➔ Druid Concepts ➔ Indexing Process ➔ Querying Process ➔ Some Benchmarks ➔ Monitoring / Logging 2
- 4. Row Store 4
- 5. Row Store - On Disk 5
- 8. Column Store - On Disk 8
- 10. Distributed Database - Sharding 10
- 11. Distributed Database - Sharding 11
- 13. Druid Concepts What is Druid? • OLAP • Columnar • Real-Time • Distributed • Optimized for Aggregations • Optimized for query performance • Horizontally Scale 13
- 14. Druid Concepts What it’s not : • Relational - (no joins!) – lookup feature as partial solution • Good at ad-hoc row retrieval • Fully ACID – no transactions – eventually consisted • Easy to change/delete data • Simple 14
- 15. Data Model • Cluster - Instance of Druid system • Data Source - collection of data, equivalent for Table • Segment - Immutable Timestamped data part of data-source (file) • Shard / Partition - For big segments, enabling partitions split the segment 15 Data Source “Tracker_Stats_Zynga” Segment May-17 Segment June-17 Partition P0 Partition P1 Partition P0 Partition P1
- 16. Druid Concepts 16 S3 / HDFS Segments Store Deep Storage Batch / RT Overlord MiddleManagers Indexing Service Brokers Historicals Query Layer Coordinator ZooKeeper Metadata-DB Management
- 18. Indexing Process - Batch Ingestion Task 18 Overlord Singular Load-Launcher API Pending Waiting Running Completed PeonPeon PeonPeon MiddleManager MiddleManager EMR Indexing To S3 Singular Load-Monitor ZooKeeper Metadata-DB
- 22. Metadata-DB Loading Segments For Query 22 Deep Storage Coordinator Historicals Historicals Historicals Segments Disk RAM Segments
- 24. Query Process 24 Singular WebApp / API Historicals Historicals Historicals Broker Broker Aggregations Lookups Segments Disk RAM Cache
- 25. 25 Numbers
- 26. Numbers ➔ Druid Servers (Prod): 20 ➔ EMR Indexing Servers: 32 ➔ AVG Indexing Task Time: 4 Minutes ➔ Daily Indexing Tasks: 15,000 ➔ Total Cluster Size: 3 TB ➔ Time to deploy a new cluster: 5 Minutes 26