Data Warehouse on Kubernetes: lessons from Clickhouse Operator
5 likes2,441 views
1) Kubernetes operators can simplify deploying data warehouses like ClickHouse from a single specification and provide portability, fast deployment, and flexible resource management. 2) While the Kubernetes operator model reduces overhead, ensuring data safety remains a challenge as data warehouses are complex and losing data is problematic. 3) Future work includes developing data warehouses as a service on Kubernetes with multi-tenancy, high availability, security, and optimized resource utilization, as well as enhancing ClickHouse to better match Kubernetes' cloud-native execution model.
Data Warehouse on Kubernetes: lessons from Clickhouse Operator
- 1. DATA WAREHOUSE AND KUBERNETES: LESSONS FROM THE CLICKHOUSE OPERATOR Robert Hodges -- August 2019
- 2. Brief Intros www.altinity.com Leading software and services provider for ClickHouse Major committer and community sponsor in US and Western Europe Robert Hodges - Altinity CEO 30+ years on DBMS plus virtualization and security. ClickHouse is DBMS #20
- 3. Why run data warehouse on Kubernetes? 1. Same environment as other cloud native services 2. Portability 3. Fast deployment cycles 4. Flexible mapping to resources
- 5. Introduction to ClickHouse Understands SQL Runs on bare metal to cloud Shared nothing architecture Stores data in columns Parallel and vectorized execution Scales to many petabytes Is Open source (Apache 2.0) a b c d a b c d a b c d a b c d And it’s really fast!
- 6. ClickHouse structure is optimized for speed Table Part Index Columns Indexed Sorted Compressed Part Index Columns Part
- 7. ClickHouse has built-in sharding & replication ClickHouse event_loc ClickHouse event event_loc ClickHouse event_loc ClickHouse event_loc ClickHouse event_loc ClickHouse event_loc SELECT ... FROM event GROUP BY ... Result Set Zookeeper ZNodes Zookeeper ZNodes Zookeeper ZNodes
- 8. Kubernetes
- 9. “Kubernetes is the new Linux” Actually it’s an open-source platform to: ● manage container-based systems ● build distributed applications declaratively ● allocate machine resources efficiently ● automate application deployment
- 10. A typical distributed service Load Balancer Service #1 Service #3 Service #2 Storage Storage Storage Traffic
- 11. Defined using Kubernetes resources Pod “svc-1” Persistent Volume Service “svc” Stateful Set Persistent Volume Claim Persistent Volume Persistent Volume Pod “svc-2” Pod “svc-2” Persistent Volume Claim Persistent Volume Claim Config Maps SecretsConfig Maps Secrets
- 12. Kubernetes NodeKubernetes NodeKubernetes Node Mapped to proxies, containers, and storage Container “svc-1” NVMe SSD NGINX “svc” Container “svc-2” Container “svc-3” NVMe SSD NVMe SSD
- 14. ClickHouse on Kubernetes is complex! Zookeeper Services Zookeeper-0 Zookeeper-2 Zookeeper-1Shard 1 Replica 1 Replica Service Load Balancer Service Shard 1 Replica 2 Shard 2 Replica 1 Shard 2 Replica 2 Replica Service Replica Service Replica Service User Config Map Common Config Map Stateful Set Pod Persistent Volume Claim Persistent Volume Per-replica Config Map
- 15. Operators encapsulate complex deployments kube-system namespace ClickHouse Operator your-favorite namespace Apache 2.0 source, distributed as Docker imageSingle specification Best practice deployment ClickHouse Resource Definition
- 16. Basic data warehouse topology apiVersion: "clickhouse.altinity.com/v1" kind: "ClickHouseInstallation" metadata: name: "ch01" spec: configuration: clusters: - name: replicated layout: shardsCount: 2 replicasCount: 2 zookeeper: nodes: - host: zookeeper.zk Name used to identify all resources Definition of cluster Location of service we depend on
- 17. Simplicity requires defaults defaults: templates: volumeClaimTemplate: persistent podTemplate: clickhouse:19.6 serviceTemplate: minikube templates: volumeClaimTemplates: - name: persistent spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi Name of template Storage misconfigurations lead to insidious errors
- 18. Templates can be simple, too defaults: templates: volumeClaimTemplate: persistent podTemplate: clickhouse:19.6 serviceTemplate: minikube templates: podTemplates: - name: clickhouse:19.6 spec: containers: - name: clickhouse-pod image: yandex/clickhouse-server:19.6.2.11 Name of template Most values take defaults
- 19. templates: podTemplates: - name: clickhouse-in-zone-us-east-1b spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "failure-domain.beta.kubernetes.io/zone" operator: In values: - "us-east-1b" containers: - name: clickhouse-pod image: yandex/clickhouse-server:19.3.7 Or specify complex configuration Set availability zone affinity More container properties
- 20. Versatile mapping to different deployments ClickHouse Resource Definition Pod Load Balance PodPod Pod Pod Load BalanceLoad Balance Load BalanceLoad Balance Pod Pod Load BalanceLoad Balance Pod Pod Minikube Multi-AZ Deployment (Differences mostly in templates)
- 21. Changes are recognized automatically defaults: templates: volumeClaimTemplate: persistent podTemplate: clickhouse:19.11 serviceTemplate: minikube templates: podTemplates: - name: clickhouse:19.11 spec: containers: - name: clickhouse-pod image: yandex/clickhouse-server:19.11.3.11 Make new version the default Define template for new version
- 22. Upgrade runs while service is online Pod chi-0-0 Update resource definition ClickHouse Operator Apply Pod chi-0-1 Pod chi-1-1 Pod chi-1-0 Compare resource to actual state Upgrade pods sequentially ClickHouse Resource Definition
- 23. Grafana ClickHouse monitoring with prometheus ClickHouse Operator (ServiceMonitor) ClickHouse Installations Prometheus
- 24. Demo Time Fast data warehouse deployment on Kubernetes
- 26. Pod chi-0-1 Surprise! DNS is different in Kubernetes Pod chi-1-1 Pod chi-0-1 Pod chi-1-0 Pod chi-0-0 DNS DNS DNS Restart Pod restart invalidates cluster DNS mappings Core DNS Server Name resolution deadlock at startup Must resolve host name to start up Won’t resolve host until pod starts
- 27. Kubernetes overhead is minimal (whew!) Cluster deploy and load Query Comparison Redshift dc2.large vs. Kubernetes EC2 r5.xlarge with EBS (st1)
- 28. No surprise: error handling is complicated ClickHouse Operator ClickHouse Resource Definition Complex specification Kubernetes Storage Provider Asynchronous execution Local semantics
- 29. Biggest challenge Data warehouses are not cattle Losing/compromising data is bad Safety is paramount Security, migration, availability require logic above level of the operator
- 30. Biggest gain Kubernetes democratizes data warehouse access Set up complex configurations in minutes Map data warehouse flexibly to resources Integrate easily with other services
- 31. Wrap-up
- 32. Conclusions ● Kubernetes operators set up DW from single specification ● ClickHouse experience validates Kubernetes value: ○ Every application can have a data warehouse! ○ Portable ○ Fast deployment ○ Flexible resource management ● Kubernetes operator alone is not enough for all use cases
- 33. Future Work ● Data warehouse as a service on Kubernetes ○ Multi-tenancy ○ Data availability ○ Security ○ Optimized resource utilization ● Extend ClickHouse to match cloud native execution model ○ Decouple storage and compute ○ Rebalance data on scale-up/down
- 34. Thank you! We’re hiring! Presenter: rhodges@altinity.com ClickHouse Operator: https://github.com/Altinity/clickhouse-operator ClickHouse: https://github.com/yandex/ClickHouse Altinity: https://www.altinity.com