ClickHouse Monitoring 101: What to monitor and how
7 likes2,862 views
The document discusses ClickHouse monitoring, led by experienced presenters Ned McClain and Robert Hodges. It covers key metrics for monitoring ClickHouse performance, including query performance, replication issues, and provides various approaches for monitoring using tools like Prometheus and Grafana. Additionally, it emphasizes the importance of building comprehensive monitoring for the entire application and includes resources for further reading.
![Which metrics can reveal root causes?
ClickHouse Server
Table: events
Part: 201801_1_1_0
Part: 201801_2_2_0
Part: 201801_1_2_1
INSERT
SELECT
Merge
ALTER
TABLE
events.InsertQuery
events.InsertRows
events.RejectedInserts
events.DelayedInserts
asychronous_events.MaxPartCountForPartition
events.MergedRows
metrics.Merge
metrics.BackgroundPoolTask
Total number
of parts [per
table]
metrics.PartMutations](https://image.slidesharecdn.com/clickhousemonitoring101-2020-04-01-final-200401213943/85/ClickHouse-Monitoring-101-What-to-monitor-and-how-8-320.jpg)
![Zookeeper
ZNodes
Zookeeper
ZNodes
Which metrics diagnose replication issues?
INSERT
Zookeeper
ZNodes
ClickHouse
Table:
events
ClickHouse
Table:
events
asynchronous_events.
ReplicasMaxAbsoluteDelay
ReplicasMaxInsertsInQueue
ReplicasMaxMergesInQueue
metrics.
LeaderReplica
ReadonlyReplica
ZookeeperTransactions
ZookeeperHardwareExceptions
zookeeper_NumAliveConnections
zookeeper_Max[Avg]RequestLatency
zookeeper_Max[Min]ClientResponseSize
zookeeper_OutstandingRequests
zookeeper_PacketsReceived[Sent]
zookeeper_ElectionTimeTaken
zookeeper_Log[Data]DirSize
zookeeper_InMemoryDataTree_NodeCount](https://image.slidesharecdn.com/clickhousemonitoring101-2020-04-01-final-200401213943/85/ClickHouse-Monitoring-101-What-to-monitor-and-how-15-320.jpg)
ClickHouse Monitoring 101: What to monitor and how
- 2. Presenting the Presenters Ned McClain Data Warehouse Engineer 20+ years architecting Internet-scale infrastructure with an emphasis on performance and security. Robert Hodges CEO 30+ years on DBMS including 20 different DBMS types. Working on Kubernetes since 2018.
- 3. ...And Altinity Leading software and services provider for ClickHouse #2 committer on ClickHouse Lead maintainer for Kubernetes, Grafana, ODBC, ... Sponsor of US and EU ClickHouse events
- 4. What to Monitor on ClickHouse
- 5. Useful information from monitoring Determining current state Analyzing long-term trends Alerts and reporting Comparing over time or experiment groups Conducting ad hoc retrospective analysis (i.e., debugging) Security incident detection and response Cost modeling and analysis
- 6. Case #1: When INSERTs go bad Too many parts (655). Merges are processing significantly slower than inserts
- 7. How does ClickHouse add or change data? ClickHouse Server Table: events Part: 201801_1_1_0 Part: 201801_2_2_0 Part: 201801_1_2_1 INSERT SELECT Merge ALTER TABLE
- 8. Which metrics can reveal root causes? ClickHouse Server Table: events Part: 201801_1_1_0 Part: 201801_2_2_0 Part: 201801_1_2_1 INSERT SELECT Merge ALTER TABLE events.InsertQuery events.InsertRows events.RejectedInserts events.DelayedInserts asychronous_events.MaxPartCountForPartition events.MergedRows metrics.Merge metrics.BackgroundPoolTask Total number of parts [per table] metrics.PartMutations
- 9. ClickHouse has great operational metrics Always-available system tables query_log (Query history) part_log (Part merges) text_log (System log messages) async… _metrics (Background metrics) events (Cumulative event counters) metrics (Current counters) parts (All table parts) More system tables you can enable replicas (Replicated tables) metric_log (Query history)
- 10. Case #2: Query memory questions kernel: Out of memory: Kill process 16959 (AsyncBlockInput) score 947 or sacrifice child
- 11. How does ClickHouse process queries? ClickHouse Server Table: events Parts primary.idx .mrk & .bin Table: devices Parts primary.idx .mrk & .bin SELECT … FROM events JOIN devices ON ...
- 12. Which metrics show query performance? ClickHouse Server Table: events Parts primary.idx .mrk & .bin Table: devices Parts primary.idx .mrk & .bin SELECT … FROM events JOIN devices ON ... events. Query SelectQuery asynchronous_events.MarkCacheFiles events. SelectedMarks/SelectedParts MarkCacheHits/MarkCacheMisses UncompressedCacheHits UncompressedCacheMisses CompressedReadBufferBytes ReadCompressedBytes metrics. MemoryTracking
- 13. Case #3: Replication regrets <Error> . . . Table is in readonly mode
- 14. Zookeeper ZNodes Zookeeper ZNodes How does ClickHouse replicate data? INSERT Zookeeper ZNodes ClickHouse Table: events ClickHouse Table: events
- 15. Zookeeper ZNodes Zookeeper ZNodes Which metrics diagnose replication issues? INSERT Zookeeper ZNodes ClickHouse Table: events ClickHouse Table: events asynchronous_events. ReplicasMaxAbsoluteDelay ReplicasMaxInsertsInQueue ReplicasMaxMergesInQueue metrics. LeaderReplica ReadonlyReplica ZookeeperTransactions ZookeeperHardwareExceptions zookeeper_NumAliveConnections zookeeper_Max[Avg]RequestLatency zookeeper_Max[Min]ClientResponseSize zookeeper_OutstandingRequests zookeeper_PacketsReceived[Sent] zookeeper_ElectionTimeTaken zookeeper_Log[Data]DirSize zookeeper_InMemoryDataTree_NodeCount
- 17. Standard ClickHouse monitoring platforms Sematext / Datadog / Instana Nagios/Icinga/Zabbix InfluxDB / Graphite… Prometheus + Grafana Prometheus + ClickHouse Kubernetes Operator + Grafana
- 18. Three high-level approaches to monitoring 1. Proprietary Agents 2. Application-aware Polling 3. Metrics Exposition
- 19. Metrics collection with Prometheus Alerts to: messaging, ticketing, and webhooks. Dashboards and trend analysis. Prometheus Alertmanager Prometheus Exposed Metrics
- 20. Monitoring a ClickHouse node Prometheus Exposed MetricsClickHouse
- 21. Monitoring a ClickHouse node $ curl clickhouse:9363/metrics
- 22. Monitoring a ClickHouse node /etc/clickhouse-server/config.xml
- 23. Monitoring a ClickHouse Cluster Prometheus ClickHouse Replica ClickHouse Replica ClickHouse Shard ClickHouse Replica ClickHouse Replica ClickHouse Shard
- 24. Monitoring Linux and cloud dependencies ● node_exporter: operating system metrics ○ CPU, memory, load, I/O, filesystem, network, and more. ● cloudwatch_exporter: AWS metrics ○ EC2 instances, ELBs, hosted databases, and more. ● stackdriver_exporter: GCP metrics ● azure_metrics_exporter: Azure metrics
- 27. Google's Four Golden Signals ● Latency ● Traffic ● Errors ● Saturation
- 29. Demo Time! ClickHouse Monitoring on Linux
- 31. ClickHouse operator exports to Prometheus ClickHouse Operator your-favorite namespace ClickHouse Resource Definition z z K8s API Monitoring data Monitoringdata Prometheus Grafana
- 32. System dynamism complicates monitoring Load Balancer Service Shard 1 Replica 1 Stateful Set Pod Persistent Volume Claim Persistent Volume Per-replica Config Map Replica Service ClickHouse Operator Monitoring data from a single replica
- 33. Operator exports new pods automatically Load Balancer Service Shard 1 Replica 1 Stateful Set Pod Persistent Volume Claim Persistent Volume Per-replica Config Map Shard 1 Replica 2 Replica Service Replica Service ClickHouse Operator Monitoring data from two replicas
- 34. Demo Time! ClickHouse Monitoring on Kubernetes with Grafana Dashboard
- 36. Summary ● ClickHouse has a wide range of accessible monitoring data ● Export relevant measurements from system database tables ○ Metrics, events, and asynchronous events ○ Query_log ● Prometheus and Grafana offer a great open source solution ○ But there are many other tools as well ● ClickHouse Operator exports monitoring data automatically to Prometheus ● Build monitoring for the whole application, not just ClickHouse
- 37. Where to look next... ● Google SRE Book: Monitoring Distributed Systems ○ https://landing.google.com/sre/sre-book ● ClickHouse monitoring documentation ○ https://clickhouse.tech/docs/en/operations/monitoring/ ● Grafana Labs Dashboards ○ https://grafana.com/grafana/dashboards ● Altinity Blog ○ https://www.altinity.com/blog ● ClickHouse Kubernetes Operator ○ https://github.com/Altinity/clickhouse-operator
- 38. Thank you! Special Offer: Contact us for a free 1-hour consultation on monitoring Contacts: info@altinity.com Visit us at: https://www.altinity.com Free Consultation: https://blog.altinity.com/offer