Gluster Metrics: why they are crucial for running stable deployments of all sizes
2 likes393 views
The document discusses the Gluster metrics for monitoring stable deployments, outlining the collection, aggregation, and reporting of various performance metrics. It emphasizes the importance of data for identifying issues such as request latency and error rates, and explores future improvements in metric capturing and analysis. Key metrics include system load, network I/O, and client-side statistics for better reliability and performance evaluation.
Gluster Metrics: why they are crucial for running stable deployments of all sizes
- 2. Gluster Metrics Running stable deployments of all sizes David Hasson Production Engineering
- 3. • Collect data locally on each host • Use a transport system on each host to a central location • Aggregate at central location • Interact with and write detectors by querying the central system Reporting and monitoring at web-scale (and you can too!)
- 4. How we identify issues • Alarms • Request latency • Error rates for specific errors • Space usage alarms • Customers • Also, our customers tell us about real problems • Key customer metrics
- 5. Using Data to find problems • Log data • Useful for finding errors on the various daemons • Categorized by a tailing system • The iostats translator • Key metric calculations • I/O sampling • Changes in GlusterFS to export data • Periodic exports of timeseries data, columned records
- 6. Gluster Metrics 1. What we look at today 2. In the pipeline: next steps 3. What we’d like to see in the future
- 7. Top line metrics • SLA unweighted average • SLA error rate / sec
- 8. Full system level stats 1. System load 2. Network I/O
- 9. Aggregate vs. Host Level
- 10. Aggregate vs. Host Level
- 11. Disk stats on the system • IO Scheduler / Block Layer • Await • Queue depths • IO sizes • read / write sec • capacity of the filesystem • Overall I/O operations/sec
- 12. System's view of gluster • CPU, memory on: • Management Daemon (glusterd) • Brick Daemons (glusterfsd) • Protocol Daemons (glusterd / nfs, glusterd / gfproxy)
- 14. Gluster IOstats (timeseries) • Major statistics piece, available upstream • Outputs important metrics • Mean / Top latency • Broken down by op type (WRITE, READ, etc) • Includes aggregate (“unweighted”) average of operations • Useful for top line / SLA alarms
- 16. Gluster IOstats (sampling) • Implemented out of io-stats • Dumped to a file for ingestion by infrastructure • Controlled with volume options • diagnostics.fop-sample-interval: 10
- 17. 1490120858.99,NORMAL,CREATE,33727.0,...host01.fb.com,777,Unknown,/neat/path/name,0,No Error 1490120860.52,LEAST,STAT,67.0,...host03.fb.com,980,/groot/namespace1,<gfid:xyz>,0,No Error 1490120865.51,LEAST,STAT,279033.0,...host18.fb.com,980,/groot/namespace1,<gfid:928>,0,No Error 1490120870.54,HIGH,STAT,70.0,...host21.fb.com,911,/groot/namespace5,<gfid:842>,0,No Error 1490120880.55,HIGH,STATFS,31.0,...host29.fb.com,870,Unknown,<gfid:842>,0,No Error 1490120895.53,LEAST,READDIRP,867.0,...host28.fb.com,874,/groot/namespace1,<gfid:6b3>,0,Success Sampling data NFS Daemon samples
- 18. Gluster IOstats (sampling) • Data can be brought into a centralized datastore • Derived metrics/diagnostics • Fleetwide error rate “drilldowns” • Information per-client, per-datacenter, etc. • Captures outliers
- 19. Gluster Metrics 1. What we look at today 2. In the pipeline: next steps 3. What we’d like to see in the future
- 20. p90/p95 data from iostats • p95 is recognized as a better indicator of “overall badness” in system metrics • Implemented in patches to io-stats
- 21. Client stats (nfusr) • Better “empathy” when looking at client side metrics • Implemented in nfusr code, however designed to same output specifications as Gluster
- 22. Measuring Reliability Where we measure it Capture Metrics NFS Server Kernel NFS Mount Traffic User Application I/O
- 23. Measuring Reliability Where we measure it Capture Metrics NFS Server nfusr Traffic User Application I/O
- 24. Per namespace stats • Currently done in 1st version of throttling changes. • New stats done in the new QoS code (see our other talk) • Allows customer-based reporting, and data on how well QoS is working
- 25. Gluster Metrics 1. What we look at today 2. In the pipeline: next steps 3. What we’d like to see in the future
- 26. Future Ideas • Per-translator metrics • Loose (accounting only) quotas