From Cardinal(ity) Sins to Cost-Efficient Metrics Aggregation
0 likes15 views
The document discusses the rising costs of cloud native observability and metrics aggregation, highlighting the unexpected expenses associated with monitoring Kubernetes environments. It explores factors affecting metrics bills, such as the number of containers, data retention, and metric granularity. Strategies for cost efficiency, including dropping unnecessary metrics and auditing, are emphasized to help organizations manage observability costs effectively.
![chronosphere.io
Most recently [vendor] was looked at to help monitor a small Kubernetes test cluster. 3
nodes.
Now the base rate of $18/mo is fine…except now they charge $1 per container per month
past 10 containers per host.
Since K8s (depending on how you install it) runs a bunch of little containers handling various
back end things, you might not deploy anything to the cluster and still be WAY over that 10
container limit.
In our case it came out to like $200/mo to monitor 3 nodes - that were nowhere fully
loaded.
- Hacker News thread](https://image.slidesharecdn.com/fromcardinalitysinstocost-efficientmetricsaggregationcloudnativepdx-231121000114-1e5bde1f/85/From-Cardinal-ity-Sins-to-Cost-Efficient-Metrics-Aggregation-8-320.jpg)
From Cardinal(ity) Sins to Cost-Efficient Metrics Aggregation
- 1. chronosphere.io From Cardinal(ity) Sins to Cost Efficient Metrics Aggregation Paige Cruz, retired SRE open source observability advocate
- 2. chronosphere.io CFO looking at the o11y bill
- 6. Cloud Native Observability bills are outrageous
- 7. Cloud Native Data Growth 7 Cloud (IaaS, VM-based) 2008 - 2018 Cloud Native (Microservices and Containers) 2018 - ? On-Premises (Data center) 1998 - 2008 Business Increase in Scale Observability Data Increase in Scale *Source: ESG Distributed Cloud Series: Observability, Feb 2022, Scott Sinclair and Rob Strechay
- 8. chronosphere.io Most recently [vendor] was looked at to help monitor a small Kubernetes test cluster. 3 nodes. Now the base rate of $18/mo is fine…except now they charge $1 per container per month past 10 containers per host. Since K8s (depending on how you install it) runs a bunch of little containers handling various back end things, you might not deploy anything to the cluster and still be WAY over that 10 container limit. In our case it came out to like $200/mo to monitor 3 nodes - that were nowhere fully loaded. - Hacker News thread
- 9. chronosphere.io Data volume Experiment: - Hello World app on 4 node Kubernetes cluster with Tracing, End User Metrics (EUM), Logs, Metrics (containers / nodes) - 30 days == +450 GB
- 10. Mighty Metrics
- 11. “ 1 in 10 metrics are actually directly queried - ServiceNow
- 12. Contributing Factors to the Metrics Bill 12 How many things you’re monitoring # of containers and infra components How often each metric is scraped Metric Granularity How long you keep the data Retention Window How many unique combos of dimensions on metrics Cardinality 12
- 13. 13
- 14. 14 Cost of monitoring can be a factor in determining how quickly to deprecate or sunset features/services/environments # of containers and infra components 14
- 15. 15 Emission time = adjust scrape_interval (from 10s samples -> 30s samples) Ingest time = aggregate Over time post-storage = downsampling Metric Granularity 15
- 17. 17 For operational metrics……most (99.9%) of queries do not pass 7 days but average retention at original granularity ranges from 2-4 weeks Retention Window 17
- 18. 18 Low value tags or entire metrics should be dropped as early as possible Dropping Data 18
- 21. “ What is the value of this metric? - You, when auditing metrics
- 22. Auditing Your Metrics 22 22 ● Scope what your team is responsible for ○ filter queries with team:YOURS ● Identify easy wins. Metrics that aren’t ○ In a monitor definition ○ Directly queried by end users ○ Powering charts for visited dashboards ● Identify labels that are unnecessary ○ e.g. prometheus instance label or instance_type ● Share your successes!
- 23. 23 23
- 24. 24 24 CFO looking at the cost efficiency of metrics
- 25. Resources - How Gloo uses the OTel Collector to drop metrics/labels and provide the Minimum Metrics Set - How to drop and delete metrics in Prometheus - How can recording and data roll-up rules help your metrics? - Observability is Too Damn Expensive - DevOpsDays London
- 26. Catch up with me: - Rescuing On-Call Engineers (send your manager) - KubeCon OTel 101: Let’s Instrument! (tracing) workshop - There’s No Place Like Production Conf42 Incident Management paigerduty@ chronosphere.io hachyderm.io LinkedIn
- 27. Q&A