Introduction to Druid and Druidry
Download as PPTX, PDF0 likes119 views
The document discusses Druid, a real-time analytics database, detailing its architecture, data processing abilities, and performance metrics. It covers various functionalities such as query capabilities, and the trade-offs in using Druid, including handling high-frequency data and limitations on data uniqueness. It also mentions tools for visualization and querying data stored in Druid, along with examples of SQL queries used in operation.
![Inverted Index
Timestamp Page Language Country Added Deleted
00:00:00 Justin Bieber en USA 25 127
00:01:00 Justin Bieber en USA 32 45
00:01:00 Kesha en CA 60 186
00:02:00 Kesha en CA 12 53
00:02:00 Kesha en IND 12 53
Justin Bieber [1,1,0,0,0]
Kesha [0,0,1,1,1]
IND [0,0,0,0,1]](https://image.slidesharecdn.com/introductiontodruidanddruidry-180528041413/85/Introduction-to-Druid-and-Druidry-13-320.jpg)
![Inverted Index
Justin Bieber [1,1,0,0,0]
Kesha [0,0,1,1,1]
IND [0,0,0,0,1]
Char Added of Justin OR Kesha from India
(11000 OR 00111) AND 00001
= 00001](https://image.slidesharecdn.com/introductiontodruidanddruidry-180528041413/85/Introduction-to-Druid-and-Druidry-14-320.jpg)
![{
"queryType": "topN",
"dataSource": "sample_data",
"dimension": "sample_dim",
"threshold": 5,
"metric": "count",
"granularity": "all",
"aggregations": [
{
"type": "longSum",
"name": "count",
"fieldName": "count"
}
],
"intervals": [
"2013-08-31T00:00:00.000/2013-09-
03T00:00:00.000"
]](https://image.slidesharecdn.com/introductiontodruidanddruidry-180528041413/85/Introduction-to-Druid-and-Druidry-26-320.jpg)
![{
"queryType": "topN",
"dataSource": "sample_data",
"dimension": "sample_dim",
"threshold": 5,
"metric": "count",
"granularity": "all",
"aggregations": [
{
"type": "longSum",
"name": "count",
"fieldName": "count"
}
],
"intervals": [
"2013-08-31T00:00:00.000/2013-09-
03T00:00:00.000"
]](https://image.slidesharecdn.com/introductiontodruidanddruidry-180528041413/85/Introduction-to-Druid-and-Druidry-30-320.jpg)
Introduction to Druid and Druidry
- 1. Introduction to Druid Gagan Gupta
- 2. How do we do? ● What channels were viewed in prime-time in Karnataka during weekend? ● How many ad requests did we get from a particular Android OS Version? ● Given a region, on how many unique users we are showing ad per device? ● What are our sales this quarter? ● What was the trend this week as compared to last week?
- 9. Dictionary Encoding Device City Page Time stayed Pixel 2 Bengaluru Landing 60 Pixel 2 Gurugram Landing 30 iPhone X Bengaluru Landing 10 Bengaluru : 0 Gurugram : 1
- 10. Roll up Timestamp Page Language Country Added Deleted 00:01:35 Justin Bieber en USA 10 65 00:01:63 Justin Bieber en USA 15 62 00:02:51 Justin Bieber en USA 32 45 00:01:11 Kesha en CA 17 87 00:02:24 Kesha en CA 43 99 00:02:03 Kesha en CA 12 53
- 11. Roll up Timestamp Page Language Country Added Deleted 00:01:35 Justin Bieber en USA 10 65 00:01:63 Justin Bieber en USA 15 62 00:02:51 Justin Bieber en USA 32 45 00:01:11 Kesha en CA 17 87 00:02:24 Kesha en CA 43 99 00:02:03 Kesha en CA 12 53 Minute Level Rollup
- 12. Roll up Timestamp Page Language Country Added Deleted 00:00:00 Justin Bieber en USA 25 127 00:01:00 Justin Bieber en USA 32 45 00:01:00 Kesha en CA 60 186 00:02:00 Kesha en CA 12 53 Minute Level Rollup
- 13. Inverted Index Timestamp Page Language Country Added Deleted 00:00:00 Justin Bieber en USA 25 127 00:01:00 Justin Bieber en USA 32 45 00:01:00 Kesha en CA 60 186 00:02:00 Kesha en CA 12 53 00:02:00 Kesha en IND 12 53 Justin Bieber [1,1,0,0,0] Kesha [0,0,1,1,1] IND [0,0,0,0,1]
- 14. Inverted Index Justin Bieber [1,1,0,0,0] Kesha [0,0,1,1,1] IND [0,0,0,0,1] Char Added of Justin OR Kesha from India (11000 OR 00111) AND 00001 = 00001
- 15. Column Oriented
- 16. Architecture
- 18. Real-time node Responsible for ingesting data in real-time
- 19. Historical node Where the data actually resides
- 20. Coordinator Node Decides what data (segment) should be on which node
- 21. Broker Node Routes query to correct nodes and merges them
- 22. External Dependencies 1. Deep Storage (HDFS/S3) 2. Zookeeper 3. Metadata store (MySQL)
- 23. Trade-offs No joins Poor for high-cardinality dimensions Cannot go below specified granularity Exact unique not available
- 24. Druid @ Production ● 3+ trillion events/month ● 3M+ events/sec ● 100+ PB of raw data ● 50+ trillion events ● 1000 queries/sec
- 26. { "queryType": "topN", "dataSource": "sample_data", "dimension": "sample_dim", "threshold": 5, "metric": "count", "granularity": "all", "aggregations": [ { "type": "longSum", "name": "count", "fieldName": "count" } ], "intervals": [ "2013-08-31T00:00:00.000/2013-09- 03T00:00:00.000" ]
- 27. Dashboards Pivot (Not opensource anymore) Turnilio (Opensourced version of Pivot) Superset by AirBNB Metabase http://druid.io/libraries.html
- 28. Messy stuff Spelling mistake Type safety Correct Query Readability
- 29. Druidry An Open-Source Java Client for Druid
- 30. { "queryType": "topN", "dataSource": "sample_data", "dimension": "sample_dim", "threshold": 5, "metric": "count", "granularity": "all", "aggregations": [ { "type": "longSum", "name": "count", "fieldName": "count" } ], "intervals": [ "2013-08-31T00:00:00.000/2013-09- 03T00:00:00.000" ]
- 31. DruidAggregator aggregator = new LongSumAggregator("count", "count"); Interval interval = new Interval(startTime, endTime); DruidTopNQuery query = DruidTopNQuery.builder() .dataSource("sample_data") .dimension(new SimpleDimension("sample_dim")) .threshold(5) .granularity(new SimpleGranularity(PredefinedGranularity.ALL)) .aggregators(Collections.singletonList(aggregator)) .intervals(Collections.singletonList(interval)) .build();
Editor's Notes
- #2: Introduction 2 parts to big data problem Data Gathering Sense of that data
- #3: To make sense of what is happening when you don’t know what you are looking for. BI queries. Make sense in milliseconds. (Exploratory nature)
- #4: Traditional DB solutions are not meant for such huge data (millions of event stream per day). Meant for metadata and aggregates
- #5: We would need to precompute for all combinations of dimensions Querying is lightning fast Storage is slightly better as it is aggregated Storage Grows Exponentially Meant for caches
- #6: Compression available at storage
- #7: (Demo here)
- #9: Diff between real time and interactive.
- #11: Actual Data
- #12: Actual Data
- #13: Minute Level Rollup 6->4
- #16: Less disk scans Bad for fetching/updating data for a particular user.
- #18: For each function, there is a node. If external dependency fails, it should not fail.
- #19: Stores some real-time data for query purposes also.
- #21: Recent data is spreaded out
- #22: Which historical node to query and realtime node if necessary
- #23: Deep Storage - persistent Zookeeper - Coordination between node, availability + which node have which data Metadata - Tier rules/Segments
- #24: Another tool in kit. No Replacement
- #27: Not feasible.
- #28: http://druid.io/libraries.html What if programmatic is there? Programmatic can be painful.
- #33: Officially listed
- #34: Thanks. Building on your contributions.