stackconf 2022: Open Source for Better Observability
0 likes59 views
The document discusses the critical role of open source in enhancing observability for cloud-native systems. It highlights the importance of metrics, logs, and traces, as well as the prevalent use of open-source tools within organizations. The author emphasizes a shift towards open-source instrumentation for improved interoperability and the need to address challenges like tool sprawl in observability practices.
stackconf 2022: Open Source for Better Observability
- 1. 1 Open Source for Better Observability @horovits Dotan Horovits
- 3. Dotan Horovits @horovits • Principal Developer Advocate at • 20 years in the hi-tech industry • Developer, architect, product • Co-organizer of DevOpsDays & CNCF Tel Aviv • Run OpenObservability Talks podcast
- 4. 4 • What’s observability • Role of open source in observability • The leading open source tools • OpenTelemetry and the unified vision Agenda Useful links ahead 📷
- 5. 5 Monitoring Cloud Native Systems is Hard
- 7. 7 Cloud Native Monitoring Requires Observability “... loosely coupled systems that are resilient, manageable, and observable.” in Control Theory: “Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.” Metrics Logs Traces Detect Diagnose Isolate & improve Observability Cloud Native Definition
- 8. 8 Cloud Native Monitoring Requires Observability “... loosely coupled systems that are resilient, manageable, and observable.” Metrics Logs Traces Detect Diagnose Isolate & improve Observability In plain English for software systems: "capability to ask and answer questions based on telemetry data" Cloud Native Definition InsideBIGDATA: https://bit.ly/o11y-data-analytics
- 9. The “What”, the “Why”, the “Where” Metrics, Logs, Traces
- 10. 10 Metrics — the “What” • Detect issues, tell what happened • Numerical (counters, gauges, histograms) • Time series storage and query • Typically combined with alerting on events
- 11. 11 Logs — the “Why” • Diagnose issues, tell why it happened • Textual • Parse, index, store, query, analyze
- 12. 12 Traces — the “Where” • Isolate issues, tell where it happened • Improve performance • Construct the trace based on causality
- 13. 13 3 Pillars of Observability The ability to ask and answer questions about our system • Metrics — the “What” • Logs — the “Why” • Traces — the “Where”
- 14. Role in Observability Open Source
- 15. 15 By 2025, 70% of new cloud-native application monitoring will use open-source instrumentation, rather than vendor-specific agents for improved interoperability. Open Source is the preferred choice Open source monitoring tools in use at nearly 60% of surveyed organizations Dec 2, 2020 April, 2021 Sep 2, 2020 The most commonly adopted tools are open source
- 16. 16 There’s no consolidation ● >¼ use 10-20 tools ● Greatest challenge with observability: Engineers and teams using multiple tools (50%) Cloud Native Observability Microsurvey 2022 https://bit.ly/cncf-o11y-2022
- 17. 17 Relicensing is changing OSS landscape Is “vendor owned open source” an oxymoron? https://horovits.medium.com/b5486a4de1c6 Source: opensource.google
- 18. 18 Open Source in Observability • OSS is key • Tool sprawl • Relicensed OSS
- 20. 20 Metrics — the “What” Open Source * Update Apr 2021: relicensed from Apache2 to AGPLv3 by Grafana Labs. *
- 21. 21 Logs — the “Why” Open Source * Update Feb 2021: relicensed from Apache2 to non-OSS (SSPL) by Elastic BV. * * ** ** Update Apr 2021: relicensed from Apache2 to AGPLv3 by Grafana Labs.
- 22. 22 Traces — the “Where” Open Source
- 23. 23 Observability is more than Logs+Metrics+Traces Metrics Logs Traces Detect Diagnose Isolate & improve Observability Unstructured and ad-hoc Structured, machine-formatted, correlated Reactive monitoring Proactive data analysis and insights Logging/metrics/tracing silos Unified data collection and analytics
- 24. 24 How do we generate and capture Telemetry? API SDK Daemon Collector Protocol Agent
- 25. 25 OpenTelemetry “OpenTelemetry is an observability framework - software and tools that assist in generating and capturing telemetry data from cloud-native software.” Across Traces, Metrics, Logs
- 26. 26 A unified set of vendor-agnostic APIs, SDKs and tools for generating and collecting telemetry data, and then exporting it to a variety of analysis tools. Source: https://logz.io/learn/opentelemetry-guide/
- 27. 27 Modern systems need to be observable Open source is key to Observability Summary Open source for better observability Tool sprawl and data silos are inhibitors Single pane of glass to unleash data analytics
- 28. 28 Open source for better observability: https://horovits.medium.com/8c65b5630561 Observability as a data analytics problem: https://bit.ly/o11y-data-analytics Cloud Native Observability Microsurvey 2022 https://bit.ly/cncf-o11y-2022 Is “vendor owned open source” an oxymoron? https://horovits.medium.com/b5486a4de1c6 Quick Dictionary to Open<X> Projects https://horovits.medium.com/e9e48f0113de OpenTelemetry Guide: https://bit.ly/otel-kubecon OpenObservability Talks: https://anchor.fm/openobservability Questions & Resources @horovits