![USE LOGGER
private int myMethod(int x) {
LOGGER.info("running my great code with x={}", x);
int y = methodX(x);
LOGGER.info("my great code finished with y={}", y);
return y;
}
2023-03-21 19:24:43,829 [main] INFO UserService - running my gre
2023-03-21 19:24:43,829 [main] INFO UserService - my great code](https://image.slidesharecdn.com/observabilitycookbookdevoxxpl24-240620115551-ebbf7ef3/85/Beginner-s-Guide-to-Observability-Devoxx-PL-2024-12-320.jpg)
Beginner's Guide to Observability@Devoxx PL 2024
- 1. BEGINNER'S GUIDE TO BEGINNER'S GUIDE TO BEGINNER'S GUIDE TO BEGINNER'S GUIDE TO BEGINNER'S GUIDE TO BEGINNER'S GUIDE TO BEGINNER'S GUIDE TO BEGINNER'S GUIDE TO BEGINNER'S GUIDE TO BEGINNER'S GUIDE TO BEGINNER'S GUIDE TO BEGINNER'S GUIDE TO OBSERVABILITY OBSERVABILITY OBSERVABILITY OBSERVABILITY OBSERVABILITY OBSERVABILITY OBSERVABILITY OBSERVABILITY OBSERVABILITY OBSERVABILITY OBSERVABILITY OBSERVABILITY Michał Niczyporuk @mihn
- 2. ABOUT ME
- 3. WHAT IS WHAT IS WHAT IS WHAT IS WHAT IS WHAT IS WHAT IS WHAT IS WHAT IS WHAT IS WHAT IS WHAT IS OBSERVABILITY? OBSERVABILITY? OBSERVABILITY? OBSERVABILITY? OBSERVABILITY? OBSERVABILITY? OBSERVABILITY? OBSERVABILITY? OBSERVABILITY? OBSERVABILITY? OBSERVABILITY? OBSERVABILITY?
- 5. THE DEFINITION “In control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs” Wikipedia
- 6. LET'S ASK THE ORACLE
- 7. WHAT FOR? Way of determining state of the system Observe trends Spot anomalies Debug errors Gather data to support decision process Measure user experience
- 9. LOGGING PRIME DIRECTIVE Don't use System.out.println()
- 11. DEFINE LOGGER public class UserService { private static final org.slf4j.Logger LOGGER = org.slf4j.LoggerFactory.getLogger(UserService.class); (...) }
- 12. USE LOGGER private int myMethod(int x) { LOGGER.info("running my great code with x={}", x); int y = methodX(x); LOGGER.info("my great code finished with y={}", y); return y; } 2023-03-21 19:24:43,829 [main] INFO UserService - running my gre 2023-03-21 19:24:43,829 [main] INFO UserService - my great code
- 13. "TALKING TO A VOID" DEBUGGING private void thisCallsMethodX() { LOGGER.info("PLEASE WORK"); methodY(); LOGGER.info("SHOULD HAVE WORKED"); }
- 14. DOS Log meaningful checkpoints Use correct logging levels Add ids to logging context: e.g. correlation id, user id High-throughput = async appender
- 15. DO NOTS🍩 Logs can be lost Logs are not audit trail Logs are not metrics Logs are not data warehouse/lake Alerts based on logs are flimsy - metrics-based alerts are better
- 16. WHAT NOT TO LOG personal information, secrets, session tokens etc. (thanks ) OWASP Logging Cheat sheet @piotrprz
- 17. CENTRALIZED LOGS CENTRALIZED LOGS CENTRALIZED LOGS CENTRALIZED LOGS CENTRALIZED LOGS CENTRALIZED LOGS CENTRALIZED LOGS CENTRALIZED LOGS CENTRALIZED LOGS CENTRALIZED LOGS CENTRALIZED LOGS CENTRALIZED LOGS
- 18. CENTRAL STORAGE Software that can ingest, process and search the logs
- 19. PUBLISHING LOGS
- 20. LOGS FORMAT Unified format (e.g. JSON, logfmt, OpenTelemetry Protocol OTPL, vendor specific) regexes = two problems
- 21. GROWTH RULE OF THUMB
- 22. TOOLING Cloud providers: AWS: Cloudwatch Logs Azure: Monitor Logs GCP: Cloud Logging Elastic Stack = ElasticSearch + Logstash/Filebeat + Kibana Graylog Grafana Loki
- 24. METRICS numbers indexed over time and multiple additional dimensions === timeseries
- 27. FORMAT Storage specific Vendor neutral: OpenTelemetry
- 29. COUNTER Used for: requests count, jobs count, errors
- 30. GAUGE Used for memory usage, thread/connection pools count, etc. Tip: Store actual current and maximum value, not the percentages
- 31. HISTOGRAM used for request latencies and sizes can calculate any percentiles via query tip: preconfigured bucket sizes = performance better
- 32. SUMMARY Histogram - but with percentiles (e.g. p50, p75, p95) Calculated client side - lighter then histograms Caveat: calculated per scrape target
- 34. METRIC RESOLUTION Main dimension How often metrics are probed Long-term storage = Down sampling/roll up 1min up to 7 days -> 10 min upto 30 days, etc. YMMV - think about business cycles (e.g. Black Friday)
- 35. METRIC TAGS/LABELS Additional dimensions Metadata: HTTP attributes, service name, hosts, cloud regions, etc. Metrics labels don't like high cardinality route="/users/:id" ✅ route="/users/2137" ❌
- 36. WHAT TO MEASURE? Kirk Pepperdine's The Box model Your framework/libraries cover some basics already
- 37. KIRK PEPPERDINE'S THE BOX "People": incoming requests, messages, jobs... Application (Netty, Tomcat, Spring, etc.): thread pools, queue sizes, requests... Runtime metrics (JVM): GC, memory usage, threads... Hardware: CPU, memory, I/O, disk usage
- 38. HOW TO MEASURE?
- 39. HOW TO MEASURE? Latency Traffic Errors Saturation The Four Golden Signals:
- 40. MEASUREMENT METHODS RED vs USE
- 43. ONE METRICS TO RULE THEM ALL APDEX https://www.apdex.org/ More user experience focused - mesures satisfaction Single number
- 44. APDEX - SPLITTING THE POPULUS Minimal sample size 100 - adjust time window Measure as close to the user as possible
- 45. APDEX - CALCULATIONS Result is a decimal between 0 and 1
- 46. APDEX - INTERPRETING THE RESULT 0.94 ≤ X ≤ 1 : excellent 0.85 ≤ X ≤ 0.93 : good 0.70 ≤ X ≤ 0.84 : fair 0.50 ≤ X ≤ 0.69 : poor 0.00 ≤ X ≤ 0.49 : unacceptable
- 47. APDEX - CAVEATS Very generic metric - hides details Should be monitored closely after deployments Should measure one functionality Shouldn't be only metric of application success
- 48. PERCENTILES Latency distribution is not normal/Gaussian Use high percentiles (p99+) - not averages or means (p50)
- 50. COORDINATED OMISSION 1. Server mesures in wrong place 2. Requests can be stuck waiting for processing Measure both clients and servers - if possible "How not to measure latency" by Gil Tene
- 51. DEATH BY METRICS Storing unnecessary amount of metrics Made possible by automations Bad for infrastructure and cloud bills Bad for your mental health
- 52. ASSUMPTIONS ABOUT METRICS WRONG 1. Metrics can't be lost 2. Metrics are precise
- 53. GROWTH RULE OF THUMB
- 55. EXAMPLE - "MISSING" INDEX
- 56. TOOLS OSS: + Libraries: Cloud providers: AWS: Cloudwatch Metrics Azure: Monitor Metrics GCP: Cloud Metrics Prometheus Grafana Thanos Victoria Metrics Elastic Stack Micrometer - API abstraction OpenTelemetry
- 57. DISTRIBUTED TRACING DISTRIBUTED TRACING DISTRIBUTED TRACING DISTRIBUTED TRACING DISTRIBUTED TRACING DISTRIBUTED TRACING DISTRIBUTED TRACING DISTRIBUTED TRACING DISTRIBUTED TRACING DISTRIBUTED TRACING DISTRIBUTED TRACING DISTRIBUTED TRACING
- 58. TRACING A B C traceID = X span ID = A HTTP Request processed traceID = X span ID = B parent span ID =A HTTP Request processed traceID = X span ID = C parent span ID =B HTTP Request processed traceID = X span ID = D parent span ID =C db query executed traceID = X span ID = E parent span ID =A HTTP Request processed traceID = X span ID = F parent span ID =E HTTP Request processed traceID = X span ID = G parent span ID =F redis query executed Internet
- 59. ANATOMY OF A TRACE
- 60. Single trace = multiple spans Each span contains Trace ID Span ID timestamp and duration Parent span ID, if applicable All the metadata with any cardinality
- 61. STORAGE
- 63. TRACING IDS FORMAT Propagated in headers (e.g.HTTP) or in metadata/attributes (e.g. Kafka) Newer: Older: W3C Trace Context Zipkin's B3
- 64. TRACE PROPAGATION Trace ID/Span ID propagation exporting spans to storage
- 65. TRACE PROPAGATION IN SERVICES your stack might be already doing that for you
- 66. TRACE PROPAGATION IN INFRASTRUCTURE
- 67. STORING TRACES
- 68. HEAD SAMPLING VS TAIL SAMPLING
- 69. HEAD SAMPLING Decision made without looking at whole trace Random sample of traffic - law of large numbers makes its viable
- 70. TAIL SAMPLING Decision made after considering all or most spans from a trace Allows for custom behaviour
- 71. TAIL SAMPLING
- 72. THROUGHPUT-BASED TAIL SAMPLING samples up to defined number of spans per second
- 73. RULES-BASED TAIL SAMPLING Storing traces matching custom policies custom policies: spans with errors, slow traces, ignore healthchecks/metric endpoints/websockets, etc.
- 74. DYNAMIC TAIL SAMPLING Aims to have a representation of span attribute values among collected traces in a timeframe example: store 1 in 100 traces, but with representation of values in attributes http.status_code, http.method, http.route and service.name
- 75. MIXING SAMPLING METHODS Sampling methods can and should be mixed
- 77. TYPICAL PROBLEMS IN TRACING
- 78. MISSING INSTRUMENTATIONS investigation needed add missing instrumentation wrap with custom span
- 79. MISSING SPANS tools are smart enough to detect that buffers for spans are too small ...or you hit ratelimiter somewhere
- 80. GROWTH RULE OF THUMB
- 82. EXAMPLE
- 83. LAST TIP Put trace and span ID's in logs
- 84. TOOLS Storage and visualization: Abstraction API: Cloud providers: AWS: X-Ray Azure: Azure Monitor Logs GCP: Cloud Trace Zipkin Jaeger OpenTelemetry Micrometer Tracing
- 88. WHAT IS OPENTELEMETRY vendor neutral observability framework
- 89. EVENTS aka "true" observability aka Observability 2.0
- 90. WHAT ARE EVENTS? Events == spans
- 91. WHAT'S DIFFERENT? Events are the fuel Columnar storage and query engine are the superpower They give ability to slice and dice data and discover unknown unknowns
- 92. UNKNOWN UNKNOWNS Things we don't know we don't know Example: Migrating to OpenTelemetry and from logs to traces
- 93. INSPIRATION Scuba: Diving into Data at Facebook
- 94. THE O.G
- 95. How to Avoid Paying for Honeycomb
- 96. HONEYCOMB UI
- 97. THE NEW KID Very comprehensive observability package Distributed tracing module needs ClickHouse
- 98. COROOT DISTRIBUTED TRACING UI
- 99. DEBUGGING WITH HONEYCOMB Based on Sandbox on their website
- 100. HEATMAP
- 101. SELECTION
- 102. BUBBLEUP
- 103. BUBBLEUP - BASELINE VS SELECTION
- 104. BUBBLEUP - ENHANCE!
- 106. BUBBLEUP - CHOSEN ONE
- 107. BUBBLEUP - PARAMETER VALUE
- 108. BUBBLEUP - FILTERED HEATMAP
- 109. BUBBLEUP - SELECTING TRACE
- 110. CULPRIT TRACE
- 111. CULPRIT TRACE - ZOOM
- 112. CULPRIT TRACE - ZOOM
- 113. CULPRIT TRACE - SIMILAR SPANS
- 114. CULPRIT TRACE - SIMILAR SPANS
- 115. CULPRIT TRACE - DEPLOYMENT
- 116. CULPRIT TRACE - DEPLOYMENT
- 117. THINGS TO REMEMBER Observability = iterative & continuous process
- 118. WHAT AFTER? SLI and SLO Alerting and on-call
- 119. THANK YOU! THANK YOU! THANK YOU! THANK YOU! THANK YOU! THANK YOU! THANK YOU! THANK YOU! THANK YOU! THANK YOU! THANK YOU! THANK YOU! QUESTIONS?