Why Distributed Tracing is Essential for Performance and Reliability
0 likes131 views
This document discusses how distributed tracing is essential for improving performance, reliability, developer velocity, and managing costs. It explains that distributed tracing provides end-to-end visibility into requests as they move between services, and allows teams to understand dependencies, pinpoint issues, and optimize software performance. The document provides examples of how distributed tracing can accelerate root cause analysis, generate more actionable alerts, inform architectural decisions, and help reduce infrastructure costs by focusing logging and storage. It concludes by recommending steps to implement distributed tracing, such as starting with critical user experiences, establishing conventions, and integrating tracing into existing workflows.
Why Distributed Tracing is Essential for Performance and Reliability
- 1. Austin Parker, Principal Developer Advocate Why Distributed Tracing is Essential for Performance and Reliability or, How to Get Actual Business Value From Distributed Tracing!
- 2. Who Am I? Austin Parker Principal Developer Advocate 2 @austinlparker austin@lightstep.com✉
- 4. 4 More autonomy… but less visibility!
- 5. Observe kustomize ? ?? Control Team-by- team ok Must be org-wide! Distributed tracing!
- 6. What you can control What you are responsible for Stress (n): responsibility without control 6
- 7. Closing the gap between control and responsibility Responsibility for delivering performance and reliability Like many problems, the solution requires: - Having the right data - Setting the right goals - Giving teams ownership Distributed tracing is essential to closing this gap 7
- 8. 1. (n) the ability to navigate from effect to cause 2. (adj) related to supporting that ability (such as a tool or process) "used an observability tool to understand what caused the change” For example, being able to navigate from… Spike in errors → misconfiguration Increased latency → new customer behavior User complaints → upstream service deployed Observability əb-ˈzər-və-bi-lə-tē 8
- 9. Getting Actual Business Value From Distributed tracing 9 Developer velocity Software performance Managing costs Fundamentals Deploying tracing
- 12. Traces are a form of telemetry based on spans with structure - Span = timed event describing work done by a single service Tracing is a diagnostic tool that reveals… … how a set of services coordinate to handle individual user requests … from mobile or browser to backends to databases (end-to-end) … including metadata like events (logs) and annotations (tags) Provides a request-centric view of application performance Distributed tracing, defined 12
- 13. Relationships matter 13 Traces encode causal relationships between callers and callees calls returns
- 14. Traces are the raw material, not the finished product Distributed traces – basically just structs Distributed tracing – the art and science of deriving value from traces 14
- 16. Increasing developer velocity - Make (common) tasks faster - Reduce interruptions - Improve communication - Prioritize high impact work 16 Verify deployments Root cause analysis Better alerts Understand dependencies Define and track SLOs
- 17. Accelerate root cause analysis 17
- 18. More actionable alerts 18 “Are We All on the Same Page? Let’s Fix That”Luis Mineiro, SREcon EMEA 2019 Search for “same page usenix”)
- 19. Understanding dependencies… without tracing 19 A B C E D C B B D A B E D 8% error rate avg. response size up 31% request rate up 4%
- 20. Understanding dependencies Without tracing... - Each connection in isolation - “A talks to B” - No way to narrow scope - No way to meaningfully tie in other metrics 20 With tracing... - End-to-end context - Request graph - Can refine based on any property of the request - Metrics linked to current scope
- 21. Use traces and service dependencies - Enhance training for new team members - Facilitate operational review meetings - Inform architectural design decisions - Set SLOs for internal services 21 Use SLOs to…- Measure reliability- Set error budgets- Hold teams accountable
- 23. Improving software performance Performance means “performance as experienced by end users” Tracing can help by… - Better distribution of computation - Focusing optimization where it matters 23
- 24. Defining the critical path 24 waiting for blue… A (part of a) span is on the critical path if: - reducing its duration speeds up overall request therefore, blue is on the critical path here
- 26. Given a choice between speeding up A and B… 1. 50% improvement in B is better than an 50% improvement in A 2. No improvement in A will ever improve overall performance by >15% Obvious… once you have the data :) A B A B A B Amdahl’s Law 26 OR
- 28. Types of costs Operational costs - Developer time (failed deployments, oncall, meeting overhead) Revenue and reputational costs - Missed SLOs, failed conversions, unhappy users Infrastructure costs - Compute, network, storage, API usage Monitoring costs 28 Take aggregated logs as an example
- 29. Calculating logging costs Initial Factors ‐ Aggregating and indexing logs per service: ‐ Storage ‐ Compute ‐ Network ‐ Peak instance count ‐ Retention period ‐ Services involved in a request 29 Initial Values Assuming 50GB of log data a day, 14 day retention, high availability (no cold storage) 1 Primary (L Compute Optimized) $89 2 Data (XL Memory Optimized) $426 3 SSDs (General Purpose) $201
- 30. $716 Cloud spend @ 50GB/logs (monthly) 30
- 31. $3,386 Total after setup, maintenance (monthly) 31
- 32. Reducing logging spend with tracing Annotate spans with logs! It’s as easy as: span.addEvent(“illegal base64 data at input byte 7”) Leverage traces to determine which logs to store 32 Monthly logging spend $3,571 $712 Logging data is more valuable in context!
- 34. On your tracing migration Tracing is not an all-or-nothing endeavour - How to deliver incremental value for the org - How to use that value to inform next steps of the journey Value to developers should be your (meta-)metric of success journey 34
- 35. Step 1 Start w/ customer-critical experiences Look at the edge and build an MVP - As close as you can (reasonably) get to users - Often an API gateway or proxy Map incoming operations → dependencies - Identify next steps - Build a case for others to adopt tracing 35
- 36. Step 2 Playbook for service owners Establish conventions for tags, etc. - What matters to your business? - What would explain failures? Instrument frameworks, libraries, shared services - Accelerate adoption by reusing code - Enforce conventions programmatically 36
- 37. Step 3 Integrate with existing workflows Where do engineers work today? - IDEs, testing frameworks, CI/CD - Dashboards - Notification and alerting - … 37
- 38. Building observable services Use open standards like OpenTelemetry for instrumenting service code. OpenTelemetry provides a single set of APIs, SDKs, and tools for generating distributed traces and metrics from your services. 38
- 39. In summary, distributed tracing provides... Faster RCA Better alerts Up-to-date dependency maps Improved compute fan-out Targeted optimization Integrated telemetry 39 Improved developer velocity Faster software performance Better cost management Distributed tracing puts application behavior in context to help answer the primary question of observability: “What caused that change?”
- 41. Q&A