SlideShare a Scribd company logo

Dok Talks #124 - Intro to Druid on Kubernetes

DoKC, profile picture
DoKC

Apache Druid is a real-time OLAP database that offers scalable batch and real-time data processing with low latency for analytical queries. It can be deployed on Kubernetes using Helm charts, allowing for easy management of configurations, scaling, and high availability. While it supports various ingestion methods and data management features, improvements like enhanced metrics collection and multi-tier configurations are still in development.

Software
1 of 39
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
on
Apache Druid on Kubernetes Apache Druid Database Overview Kubernetes & Helm Charts Apache Druid’s Helm Chart Overview Scaling Up and Down Auto-Scaling Ingestion What it Doesn’t (yet) Do
It is a database that is: Fully scalable Batch and real-time data Ad-hoc statistical queries Low latency delivery What is Apache Druid?
log search real-time ingest flexible schema text search Fully scalable Batch and real-time data Ad-hoc statistical queries Low latency delivery
log search real-time ingest flexible schema text search timeseries low latency ingest time-based storage time functions Fully scalable Batch and real-time data Ad-hoc statistical queries Low latency delivery
columnar efficient storage fast analytic queries data distribution log search real-time ingest flexible schema text search timeseries low latency ingest time-based storage time functions Fully scalable Batch and real-time data Ad-hoc statistical queries Low latency delivery
columnar efficient storage fast analytic queries data distribution log search real-time ingest flexible schema text search timeseries low latency ingest time-based storage time functions High Performance Real-time Analytics
Apache®, Apache Druid®, Druid®, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. The Architecture
The Druid Architecture Overview & High Availability Query Services Data Services Master Services broker middle- manager historical middle- manager historical middle- manager historical broker broker Deep Storage: - HDFS - S3, GCP, Azure - local ( test only) router overlord overlord coordinator coordinator middle- manager historical middle- manager historical middle- manager historical
The Druid Architecture Data Ingestion Processing Query Services Data Services Master Services broker historical historical middle- manager historical broker broker Deep Storage: router overlord overlord coordinator coordinator historical historical middle- manager historical
The Druid Architecture Data Ingestion Processing Query Services Data Services Master Services broker historical historical middle- manager historical broker broker Deep Storage: router overlord overlord coordinator coordinator historical historical middle- manager historical REST API
The Druid Architecture Data Ingestion Processing Query Services Data Services Master Services broker historical historical middle- manager historical broker broker Deep Storage: router overlord overlord coordinator coordinator historical historical middle- manager historical Streaming data Batch data
The Druid Architecture Data Ingestion Processing Query Services Data Services Master Services broker historical historical middle- manager historical broker broker Deep Storage: router overlord overlord coordinator coordinator historical historical middle- manager historical middle- manager middle- manager middle- manager middle- manager Streaming data Batch data Streaming data Streaming data
Deep Storage: The Druid Architecture Data Management Processing Query Services Data Services Master Services broker middle- manager historical broker broker router overlord overlord coordinator coordinator middle- manager historical Streaming data Batch data
Deep Storage: The Druid Architecture Query Processing Query Services Data Services Master Services broker middle- manager historical broker broker router overlord overlord coordinator coordinator middle- manager historical Streaming data Batch data middle- manager middle- manager REST API
Dok Talks #124 - Intro to Druid on Kubernetes
A Little History first… in their own words Kubernetes
Kubernetes Cluster High Level Functions Kubernetes Control Plane ➔ Acquire/manage Nodes and Storage ➔ Accept new object requests ➔ Schedule and manage containers on Nodes ➔ Instantiate containers for object deployment ➔ Monitor object state ➔ Apply application policies ◆ Restart policy ◆ Upgrade ◆ Fault tolerance namespace my-dev Dev Node Operating System Container Runtime Container zookeeper 2.1.4 zookeeper Container druid 0.22.1 coordinator Container druid 0.22.1 historical Container druid 0.22.1 overlord Container druid 0.22.1 broker Container druid 0.22.1 router Container druid 0.22.1 middle-manager Container postgresql 8.6.4 postgresql namespace qa-test Master Node Operating System Container Runtime Container zookeeper 2.1.4 zookeeper Container druid 0.22.1 coordinator Container postgresql 8.6.4 postgresql Master Operating System Container Runtime Container druid 0.22.1 overlord Master Node Operating System Container Runtime Container zookeeper 2.1.4 zookeeper Container druid 0.22.1 coordinator Master Operating System Container Runtime Container zookeeper 2.1.4 zookeeper Container druid 0.22.1 overlord Container druid 0.22.1 broker Query Node Operating System Container Runtime Container druid 0.22.1 router Container druid 0.22.1 broker Query Operating System Container Runtime Container druid 0.22.1 router Data Node Operating System Container Runtime Container druid 0.22.1 historical Data Operating System Container Runtime Container druid 0.22.1 historical Data Node Operating System Container Runtime Container druid 0.22.1 middle-manager Realtime Operating System Container Runtime Container druid 0.22.1 middle-manager
Kubernetes provides Orchestration at Scale ● High Availability - ○ Recovery - Actively monitors and restarts pods if appropriate ○ AntiAffinity - Insures no single point of failure by placing services on separate nodes ○ Persistent storage enables fast Historical recovery ● Scalability ○ Manage individual components’ scale by changing one property ○ Autoscaling based on resource utilization ● Security - ○ Encryption ○ Ingress control & network Isolation ● Upgrades - ○ Roll out changes automatically and with controlled disruption Why Apache Druid on Kubernetes
In general, it is a set of templates that describe Kubernetes objects that, in turn, provide services & applications. Apache Druid ® helm chart @ https://github.com/apache/druid/tree/master/helm/druid - Dependencies - zookeeper, postgresql or mysql - Templates for each microservice (historical, broker, middlemanager, etc.) - Default values.yaml - these are the parameters for an installation. Users override values to create different deployments with their own values.yaml: A Parameterization of Complex Deployments Helm Charts historical: replicaCount: 10 # scale of historical data middleManager: replicaCount: 6 # scale of real-time ingestion
Template Objects Apache Druid Helm Chart Query Services Data Services Master Services broker middle- manager historical middle- manager historical middle- manager historical broker broker router overlord overlord coordinator coordinator middle- manager historical middle- manager historical middle- manager historical ● Deployment - manages a set of stateless pods and keeps them running ● Ingress - outside access ● Service - Logical persistent network access, HTTP(S) port
Template Objects Apache Druid Helm Chart Query Services Data Services Master Services broker middle- manager historical middle- manager historical middle- manager historical broker broker router overlord overlord coordinator coordinator middle- manager historical middle- manager historical middle- manager historical ● StatefulSet - local files hold intermediate ingestion files, so stateful helps jobs pick up where they left off.. ● PodDisruptionBudget - determines how many pods can be offline at a time -> upgrades ● Service - Logical persistent access, HTTP(S) port ● Horizontal Pod Autoscaler - controls autoscaling
Template Objects Apache Druid Helm Chart Query Services Data Services Master Services broker middle- manager historical middle- manager historical middle- manager historical broker broker router overlord overlord coordinator coordinator middle- manager historical middle- manager historical middle- manager historical ● StatefulSet - persistent storage is extremely important at recovery time => very fast recovery. ● PodDisruptionBudget - determines how many pods can be offline at a time -> upgrades ● Service - Logical persistent access, HTTP(S) port ● No HPA - Autoscaling is not a good idea here.
A very simple example How to Use - Druid Helm Chart An example helps, so if we deploy vanilla: > git clone https://github.com/apache/druid > cd druid > helm dependency update helm/druid > helm install helm/druid a_druid -n a_space --create-namespace > kubectl get pods -n a_space NAME READY STATUS RESTARTS AGE druid-broker-744c5f46b7-5l4r7 1/1 Running 0 8m12s druid-coordinator-7c79f9c6c9-9wlqk 1/1 Running 0 8m12s druid-historical-0 1/1 Running 0 8m12s druid-middle-manager-0 1/1 Running 0 8m12s druid-postgresql-0 1/1 Running 0 8m12s druid-router-84d7cc6d87-w546r 1/1 Running 0 8m12s druid-zookeeper-0 1/1 Running 0 8m12s druid-zookeeper-1 1/1 Running 0 7m41s druid-zookeeper-2 1/1 Running 0 7m10s
A very simple example How to Use - Druid Helm Chart Create a change file like values_2_historicals.yaml: historical: replicaCount: 2 # scale of historical data Best Practice ( requires helm diff add-on ) : > helm diff upgrade -C 2 a_druid helm/druid -n a_space -f values_2_historicals.yaml reading three way merge from env default, druid-historical, StatefulSet (apps) has changed: ... spec: serviceName: druid-historical - replicas: 1 + replicas: 2 selector: matchLabels:
A very simple example How to Use - Druid Helm Chart Apply the change: > helm upgrade helm/druid a_druid -n a_space -f values_2_historicals.yaml > kubectl get pods -n a_space NAME READY STATUS RESTARTS AGE druid-broker-744c5f46b7-5l4r7 1/1 Running 0 13m druid-coordinator-7c79f9c6c9-9wlqk 1/1 Running 0 13m druid-historical-0 1/1 Running 0 13m druid-historical-1 0/1 Running 0 23s druid-middle-manager-0 1/1 Running 0 13m druid-postgresql-0 1/1 Running 0 13m druid-router-84d7cc6d87-w546r 1/1 Running 0 13m druid-zookeeper-0 1/1 Running 0 13m druid-zookeeper-1 1/1 Running 0 13m druid-zookeeper-2 1/1 Running 0 12m
Configuration with Helm Chart Query Services Data Services Master Services broker middle- manager historical middle- manager historical middle- manager historical broker broker Deep Storage: s3 local hdfs router overlord overlord coordinator coordinator middle- manager historical middle- manager historical middle- manager historical Metadata DB: postgresql mysql my_values.yaml: configVars: druid_storage_type ( See also more properties @ https://druid.apache.org/doc s/latest/configuration/index. html#deep-storage )
Configuration with Helm Chart Query Services Data Services Master Services broker middle- manager historical middle- manager historical middle- manager historical broker broker Deep Storage: s3 local hdfs router overlord overlord coordinator coordinator middle- manager historical middle- manager historical middle- manager historical Metadata DB: postgresql mysql my_values.yaml: configVars: druid_metadata_storage_type …connector_connectURI …connector_user …connector_password my_values.yaml: configVars: druid_storage_type ( See also more properties @ https://druid.apache.org/doc s/latest/configuration/index. html#deep-storage )
Configuration with Helm Chart Query Services Data Services Master Services broker middle- manager historical middle- manager historical middle- manager historical broker broker Deep Storage: s3 router overlord overlord coordinator coordinator middle- manager historical middle- manager historical middle- manager historical Metadata DB: postgresql my_values.yaml: <service>: resources: requests: cpu: 250m memory: 1Gi limits: cpu: 1000m memory: 2Gi ( A great resource to determine good values is @https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html )
Data Ingestion and Helm Chart Data Ingestion Processing Query Services Data Services Master Services broker historical historical middle- manager historical broker broker Deep Storage: router overlord overlord coordinator coordinator historical historical middle- manager historical REST API my_values.yaml: router: replicaCount: 2 ingress: enabled: True my_values.yaml: overlord: replicaCount: 2 coordinator: replicaCount: 2
Data Ingestion and Helm Chart Data Ingestion Processing Query Services Data Services Master Services broker historical historical middle- manager historical broker broker Deep Storage: router overlord overlord coordinator coordinator historical historical middle- manager historical REST API my_values.yaml: middleManager: replicaCount: 2 antiaffinity nodeSelector config: druid_indexer_runn… druid_indexer_fork…
Highly Available Data Ingestion Data Ingestion Processing Query Services Data Services Master Services broker historical historical middle- manager historical broker broker Deep Storage: router overlord overlord coordinator coordinator historical historical middle- manager historical Streaming data Batch data kafka_ingestion.json: { … “ioConfig”:{ “taskCount”: 2, “replicas”: 2, “taskDuration”:”PT1H” } }
Data Ingestion and Helm Chart Data Ingestion Processing Query Services Data Services Master Services broker historical historical middle- manager historical broker broker Deep Storage: router overlord overlord coordinator coordinator historical historical middle- manager historical middle- manager middle- manager middle- manager middle- manager Streaming data Batch data Streaming data Streaming data my_values.yaml: middleManager: replicaCount: 6 my_values.yaml: middleManager: autoscaling: enabled: true minReplicas: 2 maxReplicas: 6 metrics: memory and cpu thresholds
Deep Storage: Historicals and Helm Chart Data Management Processing Query Services Data Services Master Services broker middle- manager historical broker broker router overlord overlord coordinator coordinator middle- manager historical Streaming data Batch data my_values.yaml: historical: replicaCount: 2 antiaffinity nodeSelector
Deep Storage: Query Processing & Helm Chart Query Services Data Services Master Services broker middle- manager historical broker broker router overlord overlord coordinator coordinator middle- manager historical Streaming data Batch data middle- manager middle- manager REST API my_values.yaml: broker: replicaCount: 3 antiaffinity nodeSelector
Summarizing ● Apache Druid is a real-time OLAP database ● Kubernetes makes deploying and managing the database easier ○ Increased availability (monitored, auto-recovered, persistent) ○ Better RTO and RPO ○ Autoscaled components for ingestion and real-time query ● helm install makes it easy to deploy many different configurations: ○ Create and manage different values.yaml for each config: ■ dev-min-cluster.yaml ■ qa-ha-cluster.yaml ■ prod-ha-cluster-autoscaling.yaml ● Changes to the configs can be applied live ■ helm diff and helm upgrade ● Not just scaling ● Rolling upgrades too
What can you do to help What doesn’t it do? ● Metrics configuration - enable metrics collection and display ○ Metrics are part of Apache Druid ○ Metric-emitters have been contributed by the community ■ Influxdb-metrics-emitter, prometheus-emitter, kafka-emitter… and many more ○ Helm chart could use a set of options to turn on metrics and enable specific emitters. ● Multi-tier configurations are not yet enabled ○ Apache Druid support multiple temperature levels, i.e. ■ High speed SSDs vs High volume HDDs ○ Helm chart could use a dynamic tier configuration mechanism ● The Apache Druid Community : ○ You are invited! ○ Fork the repo at https://github.com/apache/druid ○ Make your changes ○ Submit a PR!
ASF Slack #druid Google Groups https://groups.google.com/forum/#!forum/druid-user Druid Meetups https://www.meetup.com/pro/apache-druid/ Druid News & Info @druidio #apachedruid @implydata Druid Professionals Group https://www.linkedin.com/groups/8791983/ Druid User Forum by Imply https://www.druidforum.org Imply Community Team community@imply.io & Imply Training Program https://learn.imply.io
Apache®, Apache Druid®, Druid®, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. Thank you

More Related Content

PDF
Apache Druid Auto Scale-out/in for Streaming Data Ingestion on Kubernetes
DataWorks Summit
 
PDF
Container orchestration from theory to practice
Docker, Inc.
 
PDF
Don’t Forget About Your Past—Optimizing Apache Druid Performance With Neil Bu...
HostedbyConfluent
 
PPTX
Virtualized Big Data Platform at VMware Corp IT @ VMWorld 2015
Rajit Saha
 
PPTX
Architectures, Frameworks and Infrastructure
harendra_pathak
 
PDF
DoK Talks #91- Leveraging Druid Operator to manage Apache Druid on Kubernetes
DoKC
 
PDF
Splunk: Druid on Kubernetes with Druid-operator
Imply
 
PDF
Serverless Machine Learning on Modern Hardware Using Apache Spark with Patric...
Databricks
 
Apache Druid Auto Scale-out/in for Streaming Data Ingestion on Kubernetes
DataWorks Summit
 
Container orchestration from theory to practice
Docker, Inc.
 
Don’t Forget About Your Past—Optimizing Apache Druid Performance With Neil Bu...
HostedbyConfluent
 
Virtualized Big Data Platform at VMware Corp IT @ VMWorld 2015
Rajit Saha
 
Architectures, Frameworks and Infrastructure
harendra_pathak
 
DoK Talks #91- Leveraging Druid Operator to manage Apache Druid on Kubernetes
DoKC
 
Splunk: Druid on Kubernetes with Druid-operator
Imply
 
Serverless Machine Learning on Modern Hardware Using Apache Spark with Patric...
Databricks
 

Similar to Dok Talks #124 - Intro to Druid on Kubernetes (20)

PDF
Apache Eagle: Secure Hadoop in Real Time
DataWorks Summit/Hadoop Summit
 
PDF
Apache Eagle at Hadoop Summit 2016 San Jose
Hao Chen
 
PDF
Scaling PHP apps
Matteo Moretti
 
PDF
MariaDB on Docker
MariaDB plc
 
PDF
PaaSTA: Autoscaling at Yelp
Nathan Handler
 
PDF
Create a Varnish cluster in Kubernetes for Drupal caching - DrupalCon North A...
Ovadiah Myrgorod
 
PDF
How to Puppetize Google Cloud Platform - PuppetConf 2014
Puppet
 
PDF
20150704 benchmark and user experience in sahara weiting
Wei Ting Chen
 
PPTX
HadoopCon- Trend Micro SPN Hadoop Overview
Yafang Chang
 
PPT
Hw09 Production Deep Dive With High Availability
Cloudera, Inc.
 
PDF
GE Predix 新手入门 赵锴 物联网_IoT
Kai Zhao
 
PPTX
Private Cloud with Open Stack, Docker
Davinder Kohli
 
PPTX
1. beyond mission critical virtualizing big data and hadoop
Chiou-Nan Chen
 
PPT
Champion Fas Deduplication
Michael Hudak
 
PDF
Hadoop and OpenStack - Hadoop Summit San Jose 2014
spinningmatt
 
PDF
Hadoop and OpenStack
DataWorks Summit
 
PDF
Orchestrating Redis & K8s Operators
DoiT International
 
PDF
Prometheus and Docker (Docker Galway, November 2015)
Brian Brazil
 
PPTX
What's the Hadoop-la about Kubernetes?
DataWorks Summit
 
PPTX
SF Big Analytics 20190612: Building highly efficient data lakes using Apache ...
Chester Chen
 
Apache Eagle: Secure Hadoop in Real Time
DataWorks Summit/Hadoop Summit
 
Apache Eagle at Hadoop Summit 2016 San Jose
Hao Chen
 
Scaling PHP apps
Matteo Moretti
 
MariaDB on Docker
MariaDB plc
 
PaaSTA: Autoscaling at Yelp
Nathan Handler
 
Create a Varnish cluster in Kubernetes for Drupal caching - DrupalCon North A...
Ovadiah Myrgorod
 
How to Puppetize Google Cloud Platform - PuppetConf 2014
Puppet
 
20150704 benchmark and user experience in sahara weiting
Wei Ting Chen
 
HadoopCon- Trend Micro SPN Hadoop Overview
Yafang Chang
 
Hw09 Production Deep Dive With High Availability
Cloudera, Inc.
 
GE Predix 新手入门 赵锴 物联网_IoT
Kai Zhao
 
Private Cloud with Open Stack, Docker
Davinder Kohli
 
1. beyond mission critical virtualizing big data and hadoop
Chiou-Nan Chen
 
Champion Fas Deduplication
Michael Hudak
 
Hadoop and OpenStack - Hadoop Summit San Jose 2014
spinningmatt
 
Hadoop and OpenStack
DataWorks Summit
 
Orchestrating Redis & K8s Operators
DoiT International
 
Prometheus and Docker (Docker Galway, November 2015)
Brian Brazil
 
What's the Hadoop-la about Kubernetes?
DataWorks Summit
 
SF Big Analytics 20190612: Building highly efficient data lakes using Apache ...
Chester Chen
 
Ad

More from DoKC (20)

PDF
Distributed Vector Databases - What, Why, and How
DoKC
 
PDF
Is It Safe? Security Hardening for Databases Using Kubernetes Operators
DoKC
 
PDF
Stop Worrying and Keep Querying, Using Automated Multi-Region Disaster Recovery
DoKC
 
PDF
Transforming Data Processing with Kubernetes: Journey Towards a Self-Serve Da...
DoKC
 
PDF
The State of Stateful on Kubernetes
DoKC
 
PDF
Colocating Data Workloads and Web Services on Kubernetes to Improve Resource ...
DoKC
 
PDF
Make Your Kafka Cluster Production-Ready
DoKC
 
PDF
Dynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo W...
DoKC
 
PDF
Run PostgreSQL in Warp Speed Using NVMe/TCP in the Cloud
DoKC
 
PDF
The Kubernetes Native Database
DoKC
 
PDF
ING Data Services hosted on ICHP DoK Amsterdam 2023
DoKC
 
PDF
Implementing data and databases on K8s within the Dutch government
DoKC
 
PDF
StatefulSets in K8s - DoK Talks #154
DoKC
 
PDF
Running PostgreSQL in Kubernetes: from day 0 to day 2 with CloudNativePG - Do...
DoKC
 
PDF
Analytics with Apache Superset and ClickHouse - DoK Talks #151
DoKC
 
PPTX
Overcoming challenges with protecting and migrating data in multi-cloud K8s e...
DoKC
 
PDF
Evaluating Cloud Native Storage Vendors - DoK Talks #147
DoKC
 
PDF
Kubernetes Cluster Upgrade Strategies and Data: Best Practices for your State...
DoKC
 
PDF
We will Dok You! - The journey to adopt stateful workloads on k8s
DoKC
 
PPTX
Mastering MongoDB on Kubernetes, the power of operators
DoKC
 
Distributed Vector Databases - What, Why, and How
DoKC
 
Is It Safe? Security Hardening for Databases Using Kubernetes Operators
DoKC
 
Stop Worrying and Keep Querying, Using Automated Multi-Region Disaster Recovery
DoKC
 
Transforming Data Processing with Kubernetes: Journey Towards a Self-Serve Da...
DoKC
 
The State of Stateful on Kubernetes
DoKC
 
Colocating Data Workloads and Web Services on Kubernetes to Improve Resource ...
DoKC
 
Make Your Kafka Cluster Production-Ready
DoKC
 
Dynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo W...
DoKC
 
Run PostgreSQL in Warp Speed Using NVMe/TCP in the Cloud
DoKC
 
The Kubernetes Native Database
DoKC
 
ING Data Services hosted on ICHP DoK Amsterdam 2023
DoKC
 
Implementing data and databases on K8s within the Dutch government
DoKC
 
StatefulSets in K8s - DoK Talks #154
DoKC
 
Running PostgreSQL in Kubernetes: from day 0 to day 2 with CloudNativePG - Do...
DoKC
 
Analytics with Apache Superset and ClickHouse - DoK Talks #151
DoKC
 
Overcoming challenges with protecting and migrating data in multi-cloud K8s e...
DoKC
 
Evaluating Cloud Native Storage Vendors - DoK Talks #147
DoKC
 
Kubernetes Cluster Upgrade Strategies and Data: Best Practices for your State...
DoKC
 
We will Dok You! - The journey to adopt stateful workloads on k8s
DoKC
 
Mastering MongoDB on Kubernetes, the power of operators
DoKC
 
Ad

Recently uploaded (20)

PDF
Flood Susceptibility Mapping Using Image-Based 2D-CNN Deep Learnin. Overview ...
lawrenceomai2
 
PDF
Understanding Forklifts - TECH EHS Solution
TECH EHS Solution
 
PDF
Complete React Javascript Course Syllabus.pdf
sachinpatil536675
 
PDF
How to Migrate SBCGlobal Email to Yahoo Easily
raymondjones7273
 
PPTX
L1 - Introduction to python Backend.pptx
MoizAhmed480282
 
PPTX
Essential Infomation Tech presentation.pptx
pradeepjava2016
 
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 41
ghrom2211g
 
PDF
Why TechBuilder is the Future of Pickup and Delivery App Development (1).pdf
TechBuilder
 
PPT
Introduction Database Management System for Course Database
ReinertYosua
 
PPT
JAVA ppt tutorial basics to learn java programming
pradeepjava2016
 
PPTX
Transform Your Business with a Software ERP System
Canada Software Company
 
PDF
PTS Company Brochure 2025 (1).pdf.......
pts464036
 
PDF
Which alternative to Crystal Reports is best for small or large businesses.pdf
Varsha Nayak
 
PPTX
history of c programming in notes for students .pptx
Vijayakumari829030
 
PPTX
ManageIQ - Sprint 268 Review - Slide Deck
ManageIQ
 
PPTX
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
PDF
How Creative Agencies Leverage Project Management Software.pdf
Orangescrum
 
PDF
Adobe Illustrator 28.6 Crack My Vision of Vector Design
ghrom2211g
 
PPTX
Operating system designcfffgfgggggggvggggggggg
rmskumara09
 
PDF
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
Andre Kraus
 
Flood Susceptibility Mapping Using Image-Based 2D-CNN Deep Learnin. Overview ...
lawrenceomai2
 
Understanding Forklifts - TECH EHS Solution
TECH EHS Solution
 
Complete React Javascript Course Syllabus.pdf
sachinpatil536675
 
How to Migrate SBCGlobal Email to Yahoo Easily
raymondjones7273
 
L1 - Introduction to python Backend.pptx
MoizAhmed480282
 
Essential Infomation Tech presentation.pptx
pradeepjava2016
 
Internet Downloader Manager (IDM) Crack 6.42 Build 41
ghrom2211g
 
Why TechBuilder is the Future of Pickup and Delivery App Development (1).pdf
TechBuilder
 
Introduction Database Management System for Course Database
ReinertYosua
 
JAVA ppt tutorial basics to learn java programming
pradeepjava2016
 
Transform Your Business with a Software ERP System
Canada Software Company
 
PTS Company Brochure 2025 (1).pdf.......
pts464036
 
Which alternative to Crystal Reports is best for small or large businesses.pdf
Varsha Nayak
 
history of c programming in notes for students .pptx
Vijayakumari829030
 
ManageIQ - Sprint 268 Review - Slide Deck
ManageIQ
 
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
How Creative Agencies Leverage Project Management Software.pdf
Orangescrum
 
Adobe Illustrator 28.6 Crack My Vision of Vector Design
ghrom2211g
 
Operating system designcfffgfgggggggvggggggggg
rmskumara09
 
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
Andre Kraus
 

Dok Talks #124 - Intro to Druid on Kubernetes

  • 1. on
  • 2. Apache Druid on Kubernetes Apache Druid Database Overview Kubernetes & Helm Charts Apache Druid’s Helm Chart Overview Scaling Up and Down Auto-Scaling Ingestion What it Doesn’t (yet) Do
  • 3. It is a database that is: Fully scalable Batch and real-time data Ad-hoc statistical queries Low latency delivery What is Apache Druid?
  • 4. log search real-time ingest flexible schema text search Fully scalable Batch and real-time data Ad-hoc statistical queries Low latency delivery
  • 5. log search real-time ingest flexible schema text search timeseries low latency ingest time-based storage time functions Fully scalable Batch and real-time data Ad-hoc statistical queries Low latency delivery
  • 6. columnar efficient storage fast analytic queries data distribution log search real-time ingest flexible schema text search timeseries low latency ingest time-based storage time functions Fully scalable Batch and real-time data Ad-hoc statistical queries Low latency delivery
  • 7. columnar efficient storage fast analytic queries data distribution log search real-time ingest flexible schema text search timeseries low latency ingest time-based storage time functions High Performance Real-time Analytics
  • 8. Apache®, Apache Druid®, Druid®, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. The Architecture
  • 9. The Druid Architecture Overview & High Availability Query Services Data Services Master Services broker middle- manager historical middle- manager historical middle- manager historical broker broker Deep Storage: - HDFS - S3, GCP, Azure - local ( test only) router overlord overlord coordinator coordinator middle- manager historical middle- manager historical middle- manager historical
  • 10. The Druid Architecture Data Ingestion Processing Query Services Data Services Master Services broker historical historical middle- manager historical broker broker Deep Storage: router overlord overlord coordinator coordinator historical historical middle- manager historical
  • 11. The Druid Architecture Data Ingestion Processing Query Services Data Services Master Services broker historical historical middle- manager historical broker broker Deep Storage: router overlord overlord coordinator coordinator historical historical middle- manager historical REST API
  • 12. The Druid Architecture Data Ingestion Processing Query Services Data Services Master Services broker historical historical middle- manager historical broker broker Deep Storage: router overlord overlord coordinator coordinator historical historical middle- manager historical Streaming data Batch data
  • 13. The Druid Architecture Data Ingestion Processing Query Services Data Services Master Services broker historical historical middle- manager historical broker broker Deep Storage: router overlord overlord coordinator coordinator historical historical middle- manager historical middle- manager middle- manager middle- manager middle- manager Streaming data Batch data Streaming data Streaming data
  • 14. Deep Storage: The Druid Architecture Data Management Processing Query Services Data Services Master Services broker middle- manager historical broker broker router overlord overlord coordinator coordinator middle- manager historical Streaming data Batch data
  • 15. Deep Storage: The Druid Architecture Query Processing Query Services Data Services Master Services broker middle- manager historical broker broker router overlord overlord coordinator coordinator middle- manager historical Streaming data Batch data middle- manager middle- manager REST API
  • 17. A Little History first… in their own words Kubernetes
  • 18. Kubernetes Cluster High Level Functions Kubernetes Control Plane ➔ Acquire/manage Nodes and Storage ➔ Accept new object requests ➔ Schedule and manage containers on Nodes ➔ Instantiate containers for object deployment ➔ Monitor object state ➔ Apply application policies ◆ Restart policy ◆ Upgrade ◆ Fault tolerance namespace my-dev Dev Node Operating System Container Runtime Container zookeeper 2.1.4 zookeeper Container druid 0.22.1 coordinator Container druid 0.22.1 historical Container druid 0.22.1 overlord Container druid 0.22.1 broker Container druid 0.22.1 router Container druid 0.22.1 middle-manager Container postgresql 8.6.4 postgresql namespace qa-test Master Node Operating System Container Runtime Container zookeeper 2.1.4 zookeeper Container druid 0.22.1 coordinator Container postgresql 8.6.4 postgresql Master Operating System Container Runtime Container druid 0.22.1 overlord Master Node Operating System Container Runtime Container zookeeper 2.1.4 zookeeper Container druid 0.22.1 coordinator Master Operating System Container Runtime Container zookeeper 2.1.4 zookeeper Container druid 0.22.1 overlord Container druid 0.22.1 broker Query Node Operating System Container Runtime Container druid 0.22.1 router Container druid 0.22.1 broker Query Operating System Container Runtime Container druid 0.22.1 router Data Node Operating System Container Runtime Container druid 0.22.1 historical Data Operating System Container Runtime Container druid 0.22.1 historical Data Node Operating System Container Runtime Container druid 0.22.1 middle-manager Realtime Operating System Container Runtime Container druid 0.22.1 middle-manager
  • 19. Kubernetes provides Orchestration at Scale ● High Availability - ○ Recovery - Actively monitors and restarts pods if appropriate ○ AntiAffinity - Insures no single point of failure by placing services on separate nodes ○ Persistent storage enables fast Historical recovery ● Scalability ○ Manage individual components’ scale by changing one property ○ Autoscaling based on resource utilization ● Security - ○ Encryption ○ Ingress control & network Isolation ● Upgrades - ○ Roll out changes automatically and with controlled disruption Why Apache Druid on Kubernetes
  • 20. In general, it is a set of templates that describe Kubernetes objects that, in turn, provide services & applications. Apache Druid ® helm chart @ https://github.com/apache/druid/tree/master/helm/druid - Dependencies - zookeeper, postgresql or mysql - Templates for each microservice (historical, broker, middlemanager, etc.) - Default values.yaml - these are the parameters for an installation. Users override values to create different deployments with their own values.yaml: A Parameterization of Complex Deployments Helm Charts historical: replicaCount: 10 # scale of historical data middleManager: replicaCount: 6 # scale of real-time ingestion
  • 21. Template Objects Apache Druid Helm Chart Query Services Data Services Master Services broker middle- manager historical middle- manager historical middle- manager historical broker broker router overlord overlord coordinator coordinator middle- manager historical middle- manager historical middle- manager historical ● Deployment - manages a set of stateless pods and keeps them running ● Ingress - outside access ● Service - Logical persistent network access, HTTP(S) port
  • 22. Template Objects Apache Druid Helm Chart Query Services Data Services Master Services broker middle- manager historical middle- manager historical middle- manager historical broker broker router overlord overlord coordinator coordinator middle- manager historical middle- manager historical middle- manager historical ● StatefulSet - local files hold intermediate ingestion files, so stateful helps jobs pick up where they left off.. ● PodDisruptionBudget - determines how many pods can be offline at a time -> upgrades ● Service - Logical persistent access, HTTP(S) port ● Horizontal Pod Autoscaler - controls autoscaling
  • 23. Template Objects Apache Druid Helm Chart Query Services Data Services Master Services broker middle- manager historical middle- manager historical middle- manager historical broker broker router overlord overlord coordinator coordinator middle- manager historical middle- manager historical middle- manager historical ● StatefulSet - persistent storage is extremely important at recovery time => very fast recovery. ● PodDisruptionBudget - determines how many pods can be offline at a time -> upgrades ● Service - Logical persistent access, HTTP(S) port ● No HPA - Autoscaling is not a good idea here.
  • 24. A very simple example How to Use - Druid Helm Chart An example helps, so if we deploy vanilla: > git clone https://github.com/apache/druid > cd druid > helm dependency update helm/druid > helm install helm/druid a_druid -n a_space --create-namespace > kubectl get pods -n a_space NAME READY STATUS RESTARTS AGE druid-broker-744c5f46b7-5l4r7 1/1 Running 0 8m12s druid-coordinator-7c79f9c6c9-9wlqk 1/1 Running 0 8m12s druid-historical-0 1/1 Running 0 8m12s druid-middle-manager-0 1/1 Running 0 8m12s druid-postgresql-0 1/1 Running 0 8m12s druid-router-84d7cc6d87-w546r 1/1 Running 0 8m12s druid-zookeeper-0 1/1 Running 0 8m12s druid-zookeeper-1 1/1 Running 0 7m41s druid-zookeeper-2 1/1 Running 0 7m10s
  • 25. A very simple example How to Use - Druid Helm Chart Create a change file like values_2_historicals.yaml: historical: replicaCount: 2 # scale of historical data Best Practice ( requires helm diff add-on ) : > helm diff upgrade -C 2 a_druid helm/druid -n a_space -f values_2_historicals.yaml reading three way merge from env default, druid-historical, StatefulSet (apps) has changed: ... spec: serviceName: druid-historical - replicas: 1 + replicas: 2 selector: matchLabels:
  • 26. A very simple example How to Use - Druid Helm Chart Apply the change: > helm upgrade helm/druid a_druid -n a_space -f values_2_historicals.yaml > kubectl get pods -n a_space NAME READY STATUS RESTARTS AGE druid-broker-744c5f46b7-5l4r7 1/1 Running 0 13m druid-coordinator-7c79f9c6c9-9wlqk 1/1 Running 0 13m druid-historical-0 1/1 Running 0 13m druid-historical-1 0/1 Running 0 23s druid-middle-manager-0 1/1 Running 0 13m druid-postgresql-0 1/1 Running 0 13m druid-router-84d7cc6d87-w546r 1/1 Running 0 13m druid-zookeeper-0 1/1 Running 0 13m druid-zookeeper-1 1/1 Running 0 13m druid-zookeeper-2 1/1 Running 0 12m
  • 27. Configuration with Helm Chart Query Services Data Services Master Services broker middle- manager historical middle- manager historical middle- manager historical broker broker Deep Storage: s3 local hdfs router overlord overlord coordinator coordinator middle- manager historical middle- manager historical middle- manager historical Metadata DB: postgresql mysql my_values.yaml: configVars: druid_storage_type ( See also more properties @ https://druid.apache.org/doc s/latest/configuration/index. html#deep-storage )
  • 28. Configuration with Helm Chart Query Services Data Services Master Services broker middle- manager historical middle- manager historical middle- manager historical broker broker Deep Storage: s3 local hdfs router overlord overlord coordinator coordinator middle- manager historical middle- manager historical middle- manager historical Metadata DB: postgresql mysql my_values.yaml: configVars: druid_metadata_storage_type …connector_connectURI …connector_user …connector_password my_values.yaml: configVars: druid_storage_type ( See also more properties @ https://druid.apache.org/doc s/latest/configuration/index. html#deep-storage )
  • 29. Configuration with Helm Chart Query Services Data Services Master Services broker middle- manager historical middle- manager historical middle- manager historical broker broker Deep Storage: s3 router overlord overlord coordinator coordinator middle- manager historical middle- manager historical middle- manager historical Metadata DB: postgresql my_values.yaml: <service>: resources: requests: cpu: 250m memory: 1Gi limits: cpu: 1000m memory: 2Gi ( A great resource to determine good values is @https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html )
  • 30. Data Ingestion and Helm Chart Data Ingestion Processing Query Services Data Services Master Services broker historical historical middle- manager historical broker broker Deep Storage: router overlord overlord coordinator coordinator historical historical middle- manager historical REST API my_values.yaml: router: replicaCount: 2 ingress: enabled: True my_values.yaml: overlord: replicaCount: 2 coordinator: replicaCount: 2
  • 31. Data Ingestion and Helm Chart Data Ingestion Processing Query Services Data Services Master Services broker historical historical middle- manager historical broker broker Deep Storage: router overlord overlord coordinator coordinator historical historical middle- manager historical REST API my_values.yaml: middleManager: replicaCount: 2 antiaffinity nodeSelector config: druid_indexer_runn… druid_indexer_fork…
  • 32. Highly Available Data Ingestion Data Ingestion Processing Query Services Data Services Master Services broker historical historical middle- manager historical broker broker Deep Storage: router overlord overlord coordinator coordinator historical historical middle- manager historical Streaming data Batch data kafka_ingestion.json: { … “ioConfig”:{ “taskCount”: 2, “replicas”: 2, “taskDuration”:”PT1H” } }
  • 33. Data Ingestion and Helm Chart Data Ingestion Processing Query Services Data Services Master Services broker historical historical middle- manager historical broker broker Deep Storage: router overlord overlord coordinator coordinator historical historical middle- manager historical middle- manager middle- manager middle- manager middle- manager Streaming data Batch data Streaming data Streaming data my_values.yaml: middleManager: replicaCount: 6 my_values.yaml: middleManager: autoscaling: enabled: true minReplicas: 2 maxReplicas: 6 metrics: memory and cpu thresholds
  • 34. Deep Storage: Historicals and Helm Chart Data Management Processing Query Services Data Services Master Services broker middle- manager historical broker broker router overlord overlord coordinator coordinator middle- manager historical Streaming data Batch data my_values.yaml: historical: replicaCount: 2 antiaffinity nodeSelector
  • 35. Deep Storage: Query Processing & Helm Chart Query Services Data Services Master Services broker middle- manager historical broker broker router overlord overlord coordinator coordinator middle- manager historical Streaming data Batch data middle- manager middle- manager REST API my_values.yaml: broker: replicaCount: 3 antiaffinity nodeSelector
  • 36. Summarizing ● Apache Druid is a real-time OLAP database ● Kubernetes makes deploying and managing the database easier ○ Increased availability (monitored, auto-recovered, persistent) ○ Better RTO and RPO ○ Autoscaled components for ingestion and real-time query ● helm install makes it easy to deploy many different configurations: ○ Create and manage different values.yaml for each config: ■ dev-min-cluster.yaml ■ qa-ha-cluster.yaml ■ prod-ha-cluster-autoscaling.yaml ● Changes to the configs can be applied live ■ helm diff and helm upgrade ● Not just scaling ● Rolling upgrades too
  • 37. What can you do to help What doesn’t it do? ● Metrics configuration - enable metrics collection and display ○ Metrics are part of Apache Druid ○ Metric-emitters have been contributed by the community ■ Influxdb-metrics-emitter, prometheus-emitter, kafka-emitter… and many more ○ Helm chart could use a set of options to turn on metrics and enable specific emitters. ● Multi-tier configurations are not yet enabled ○ Apache Druid support multiple temperature levels, i.e. ■ High speed SSDs vs High volume HDDs ○ Helm chart could use a dynamic tier configuration mechanism ● The Apache Druid Community : ○ You are invited! ○ Fork the repo at https://github.com/apache/druid ○ Make your changes ○ Submit a PR!
  • 38. ASF Slack #druid Google Groups https://groups.google.com/forum/#!forum/druid-user Druid Meetups https://www.meetup.com/pro/apache-druid/ Druid News & Info @druidio #apachedruid @implydata Druid Professionals Group https://www.linkedin.com/groups/8791983/ Druid User Forum by Imply https://www.druidforum.org Imply Community Team community@imply.io & Imply Training Program https://learn.imply.io
  • 39. Apache®, Apache Druid®, Druid®, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. Thank you