Overprov a tool for cluster overprovisioning detection
Download as PPTX, PDF0 likes130 views
This document describes a tool called Overprov for detecting cluster overprovisioning. The tool analyzes metrics from clusters to check for signs of overprovisioning based on simple rules. It is designed to save costs in the long run by identifying when clusters have extra resources that are not being used. The tool can analyze multiple metrics for different cluster types and products. It has a modular structure so new analyzers can be added to check additional metrics. The tool generates daily reports but is not meant for instantaneous autoscaling decisions. It includes examples of using the tool to analyze Cassandra and Elasticsearch clusters.
![Code Structure
● run()
for cluster_name in clusters:
dt = detector.ClusterOverprovDetector(
product,
ecosystem,
cluster_name,
metric_list,
start,
stop,
signalfx_auth_token
dt.execute()
● metric_list
metric_list_cass = [
ModuleClass('overprov.analyzers.cpu_idle_analyzer', 'CpuIdleAnalyzer'),
ModuleClass('overprov.analyzers.cass_gc_count_analyzer', 'CassGcCountAnalyzer'),
ModuleClass('overprov.analyzers.cass_disk_free_analyzer', 'CassDiskFreeAnalyzer'),
]](https://image.slidesharecdn.com/overprov-atoolforclusteroverprovisioningdetection-161030084422/85/Overprov-a-tool-for-cluster-overprovisioning-detection-7-320.jpg)
Overprov a tool for cluster overprovisioning detection
- 1. Overprov: A Tool for Cluster Overprovisioning Detection Del Bao
- 3. Problem (2) bizfeed oldgen gc count a day
- 4. Problem (3) generic cassandra byte_percentfree
- 5. what does the tool do?
- 6. Design Goals • save cost in the long run • based on simple rules • eliminate false positive • extensible
- 7. Code Structure ● run() for cluster_name in clusters: dt = detector.ClusterOverprovDetector( product, ecosystem, cluster_name, metric_list, start, stop, signalfx_auth_token dt.execute() ● metric_list metric_list_cass = [ ModuleClass('overprov.analyzers.cpu_idle_analyzer', 'CpuIdleAnalyzer'), ModuleClass('overprov.analyzers.cass_gc_count_analyzer', 'CassGcCountAnalyzer'), ModuleClass('overprov.analyzers.cass_disk_free_analyzer', 'CassDiskFreeAnalyzer'), ]
- 8. You can extend it • create your own analyzer • pass in your start, stop day
- 9. Assumptions • static check, so the daily/hourly resolution, e.g., p95 is fine. • cluster is almost well balanced, so take max/min across cluster hosts in a region represents the entire cluster
- 10. What it’s Not • Fleetmiser – Instantaneous autoscale spot fleet for seagull clusters – a signal of 10 min interval • Paasta – similar to above, only for paasta service
- 11. Demo • virtualenv_run/bin/overprov -p cassandra -c ad_backend --start 60 --stop 30 -e prod -k ./api_token • virtualenv_run/bin/overprov -p cassandra -c ad_backend --start 60 --stop 30 -e prod -k ./api_token -- debug • virtualenv_run/bin/overprov -p elasticsearch -c ads144 - -start 60 --stop 30 -e prod -k ./api_token
- 12. Questions
Editor's Notes
- #7: 来张图配design
- #8: def execute(self): """Run through all registered analyzers and print result tables each analyzer may generate data for only some regions due to lack of data, e.g., outage, signalflow issue, etc. to make it robust, we create region list on the fly and then assemble table data in another loop. if we are missing some region, the result will be None, which is also not overprovisioned """