Choose Your Own Adventure - Cloud Native Observability Pitfalls
Download as PPTX, PDF0 likes48 views
The document discusses intrinsic pitfalls in cloud-native observability, emphasizing the importance of recognizing existing landscapes, managing data volume, and controlling costs. It identifies six key pitfalls organizations should watch for: ignoring the landscape, focusing too narrowly on pillars, dealing with excessive tool sprawl, cost management, navigating protocol complexities, and underestimating cardinality. The document encourages a systematic approach to observability to enhance efficiency, minimize costs, and support better outcomes.
Choose Your Own Adventure - Cloud Native Observability Pitfalls
- 1. chronosphere.io chronosphere.io Choose Your Own Adventure Eric D. Schabell Director Evangelism @ericschabell{@fosstodon.org} Cloud Native Observability Pitfalls
- 4. chronosphere.io Data volume Experiment: - Hello World app on 4 node Kubernetes cluster with Tracing, End User Metrics (EUM), Logs, Metrics (containers / nodes) - 30 days == +450 GB
- 6. chronosphere.io Cloud Native at Scale
- 9. chronosphere.io O11y at Scale (need)
- 10. chronosphere.io Picking Your Pitfalls 1. Ignoring existing landscape 2. Focusing on The Pillars 3. Sneaky sprawling mess 4. Controlling costs 5. The protocol jungle 6. Underestimating cardinality (Click on a pitfall to jump to that section)
- 12. chronosphere.io If they can’t see me… they can’t hurt me...
- 13. chronosphere.io
- 14. chronosphere.io Prometheus for metrics, alerting, queries
- 16. chronosphere.io Manual instrumentation (java client lib)
- 18. chronosphere.io Applications (Java) OTel Auto Instrumentation (libraries) OTel API OTel SDK OTel Collector OTLP OTLP OTLP OpenTelemetry (Auto) instrumentation
- 19. chronosphere.io Host Observability Backend (Prometheus, Jaeger, Fluent Bit, etc.), Applications OTel Auto Instrumentation OTel API OTel SDK OTel Collector Agent OTLP OTLP OTLP OTLP OTLP OpenTelemetry Collector (agent)
- 20. chronosphere.io Host Host Host Observability Backend (Prometheus, Jaeger, Fluent Bit, etc.), Applications OTel Auto Instrumentation OTel API OTel SDK OTel Collector Agent OTLP OTLP OTLP OTLP Collector (gateway) OTel Collector Gateway
- 22. chronosphere.io 1. Ignoring existing landscape 2. Focusing on The Pillars 3. Sneaky sprawling mess 4. Controlling costs 5. The protocol jungle 6. Underestimating cardinality (Click on a pitfall to jump to that section, or jump to end) Picking Your Next Pitfall
- 23. chronosphere.io 2. Focusing on The Pillars
- 27. chronosphere.io Car is on fire…
- 28. chronosphere.io Better outcomes… Faster remediation… Easier detection… Happier customers…
- 29. chronosphere.io Phase 1 Know something is happening as fast as possible…
- 30. chronosphere.io Phase 2 Triage with specific information…
- 31. chronosphere.io Phase 3 Understand to ensure never happens again…
- 32. chronosphere.io
- 33. chronosphere.io
- 34. chronosphere.io 1. Ignoring existing landscape 2. Focusing on The Pillars 3. Sneaky sprawling mess 4. Controlling costs 5. The protocol jungle 6. Underestimating cardinality (Click on a pitfall to jump to that section, or jump to end) Picking Your Next Pitfall
- 35. chronosphere.io 3. Sneaky sprawling mess
- 36. chronosphere.io Over 66% of organizations use more than 10 different observability tools – ESG report over exploding data volumes
- 37. chronosphere.io
- 39. chronosphere.io
- 40. chronosphere.io 1. Ignoring existing landscape 2. Focusing on The Pillars 3. Sneaky sprawling mess 4. Controlling costs 5. The protocol jungle 6. Underestimating cardinality (Click on a pitfall to jump to that section, or jump to end) Picking Your Next Pitfall
- 42. chronosphere.io “It’s remarkable how common this situation is, where an organization is paying more for their observability data, than they do for their production infrastructure.”
- 43. chronosphere.io O11y data storage costs are broken. Keeping everything model?
- 44. chronosphere.io Know the cost of observability metrics data?
- 45. chronosphere.io DATA COLLECTION CONTROL PLANE PURPOSE-BUILT DATA STORES PER TELEMETRY TYPE CHRONOSPHERE LENS Align cost to value Single Tenanted Architecture w/ 99.99% Reliability Turns raw data into generated insights for each user Customer Environment Chronosphere SaaS Platform METRICS | LOGS | TRACES | EVENTS Ingest all your data from any source
- 46. chronosphere.io
- 47. chronosphere.io 1. Ignoring existing landscape 2. Focusing on The Pillars 3. Sneaky sprawling mess 4. Controlling costs 5. The protocol jungle 6. Underestimating cardinality (Click on a pitfall to jump to that section, or jump to end) Picking Your Next Pitfall
- 48. chronosphere.io 5. The protocol jungle
- 49. chronosphere.io Without open standards, you’ll not find a way back…
- 50. chronosphere.io
- 51. chronosphere.io Host Observability Backend (Prometheus, Jaeger, Fluent Bit, etc.), Applications OTel Auto Instrumentation OTel API OTel SDK OTel Collector Agent OTLP OTLP OTLP OTLP OTLP OpenTelemetry Collector (agent)
- 52. chronosphere.io Prometheus for metrics, alerting, queries
- 53. chronosphere.io 1. Ignoring existing landscape 2. Focusing on The Pillars 3. Sneaky sprawling mess 4. Controlling costs 5. The protocol jungle 6. Underestimating cardinality (Click on a pitfall to jump to that section, or jump to end) Picking Your Next Pitfall
- 55. chronosphere.io The struggle is real “I don't yet collect spans/traces because I can hardly get our devs to care about basic metrics, let alone traces.” “This is a large enterprise with approx. 1000 developers. Cultivating a culture of engineering that cares about availability is a challenge that we need to solve alongside any technical implementations.”
- 56. chronosphere.io 10 hours on average, per week, trying to triage and understand incidents - a quarter of a 40 hour work week
- 57. chronosphere.io 33% said those issues disrupted their personal life 39% admitting they are frequently stressed out
- 59. chronosphere.io DATA COLLECTION CONTROL PLANE PURPOSE-BUILT DATA STORES PER TELEMETRY TYPE CHRONOSPHERE LENS Align cost to value Single Tenanted Architecture w/ 99.99% Reliability Turns raw data into generated insights for each user Customer Environment Chronosphere SaaS Platform METRICS | LOGS | TRACES | EVENTS Ingest all your data from any source
- 60. chronosphere.io 1. Ignoring existing landscape 2. Focusing on The Pillars 3. Sneaky sprawling mess 4. Controlling costs 5. The protocol jungle 6. Underestimating cardinality (Click on a pitfall to jump to that section, or jump to end) Picking Your Next Pitfall
- 61. chronosphere.io What should be the #1 item on cloud wishlist? What should be #1 item on your cloud native observability wishlist?
- 62. chronosphere.io chronosphere.io Questions? Eric D. Schabell Director Evangelism @ericschabell{@fosstodon.org}