A Practical Deep Dive into Observability of Streaming Applications with Kosta Chuturkov & Tim van Baarsen
1 like614 views
This document provides an overview of observability of streaming applications using Kafka. It discusses the three pillars of observability - logging, metrics, and tracing. It describes how to expose Kafka client-side metrics using interceptors, metric reporters, and the Spring Boot framework. It demonstrates calculating consumer lag from broker and client-side metrics. It introduces OpenTelemetry for collecting telemetry data across applications and exporting to various backends. Finally, it wraps up with lessons on monitoring consumer lag trends and selecting the right metrics to ship.
![Logs
2023-04-06T09:11:47.341Z INFO [spring-kafka-
producer,210d9dc16597a0a8f1a9746c4bbd8277,6b350c56f69a6ee6] 7 --- [nio-8080-exec-1]
c.e.rest.StockQuoteRestController : Produce stock quote via Rest API
{
"scope": {
"name": ”com.example.rest.StockQuoteRestController"
},
"logRecords": [
{
"timeUnixNano": "1680772307341000000",
"severityNumber": 9,
"severityText": "INFO",
"body": {
"stringValue": "Produce stock quote via Rest API"
},
"flags": 1,
"traceId": "210d9dc16597a0a8f1a9746c4bbd8277",
"spanId": "6b350c56f69a6ee6"
}
]
}](https://image.slidesharecdn.com/bs26-20230517-ksl23-ing-vanbaarsentimandchuturkovkosta-230602120532-58083d52/85/A-Practical-Deep-Dive-into-Observability-of-Streaming-Applications-with-Kosta-Chuturkov-Tim-van-Baarsen-11-320.jpg)
A Practical Deep Dive into Observability of Streaming Applications with Kosta Chuturkov & Tim van Baarsen
- 1. Hope is not a course of action! - A practical deep dive into Observability of Streaming Applications ING Tim van Baarsen & Kosta Chuturkov
- 2. About the Speakers The Netherlands - Amsterdam Team Dora Romania - Bucharest
- 3. ING https://www.ing.jobs/ • 60,000+ employees • Serve 37+ million customers • Corporate clients and financial institutions in over 40 countries
- 4. Kafka @ ING Frontrunners in Kafka since 2014 Running in production: • 8 years • 6000+ topics • Serving 1000+ Development teams • Self service topic management
- 5. Kafka @ ING Traffic is growing with +10% monthly 0 200.000 400.000 600.000 800.000 1.000.000 1.200.000 2015 2016 2017 2018 2019 2020 2021 2022 2023 Messages produced per second (average) Messages produced per second (average)
- 6. What are we going to cover today ? • What is Observability and why do we need this? • Three Pillars of observability • Exposing Kafka Client Side Metrics • How to interpret them? Consumer lag • Kafka client-side metrics demo • OpenTelemetry • OpenTelemetry Collector • Demo (Spring + Kafka + distributed tracing/logging/metrics) • Wrap Up • Questions
- 7. Introduction to application Observability Observability is the ability to measure the internal state of a system only by its external outputs. (logs, metrics, and traces)
- 8. Why do we need this?
- 9. Why do we need this? • Helps to investigate root causes of incidents • Improve our software • Prevent outages • Better user experience for our customers
- 10. Three Pillars of Observability Logging Tracing Metrics (Aggregatable) (Events) Tracing: Recording flow through the application(s) and the interactions between services. Helps: • Understand why something happened. Logging: Recording individual events and data. Helps: to understand what happened. Hard to tell: context Metrics: Recording time series data. Aggregate Helps: • understand the context • identify trends • alert Hard to tell: why something is not working as expected?
- 11. Logs 2023-04-06T09:11:47.341Z INFO [spring-kafka- producer,210d9dc16597a0a8f1a9746c4bbd8277,6b350c56f69a6ee6] 7 --- [nio-8080-exec-1] c.e.rest.StockQuoteRestController : Produce stock quote via Rest API { "scope": { "name": ”com.example.rest.StockQuoteRestController" }, "logRecords": [ { "timeUnixNano": "1680772307341000000", "severityNumber": 9, "severityText": "INFO", "body": { "stringValue": "Produce stock quote via Rest API" }, "flags": 1, "traceId": "210d9dc16597a0a8f1a9746c4bbd8277", "spanId": "6b350c56f69a6ee6" } ] }
- 12. Tracing Trace Id dfb987a57cbb9fb6e13a00d68413efa4 (Root) Span Id 06eeb7d5c3e8327e Span Id 0912845005f1a035 Span Id 808740d73506eee2 Time Sequence of operations Duration of single span Duration of total trace Service A Service B Service C Context is propagated over the network Kafka header
- 13. Metrics: Interceptors • Kafka-clients API is pluggable • Confluent Monitoring Interceptor • Interceptors (interceptor.classes) • Consumer • Producer • Send metrics to Kafka topic (_confluent-monitoring) • Confluent Control Center consumerProperties.put( ProducerConfig.INTERCEPTOR_CLASSES_CONFIG, " io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"); producerProperties.put( ConsumerConfig.INTERCEPTOR_CLASSES_CONFIG, " io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor");
- 14. Metrics: Interceptors Consumer lag metrics in Confluent Control Center
- 15. Metrics: Metric Reporters • Metric Reporters (metric.reporters) • Default JMX • View metrics in JConsole • Prefer Metrics in time series database!
- 16. Metrics: Java Agent Your Kafka Application HTTP JMX Exporter Prometheus Agent JVM process java -javaagent:./jmx_prometheus_javaagent-0.18.0.jar =1234:kafka_clients.yml -jar your-kafka-application.jar Expose a HTTP endpoint serving metrics of the local JVM
- 17. Metrics: Spring Boot & Micrometer What is Spring Boot? • Defacto standard building Java applications • Very opinionated • ‘Fat’ JAR (executable + all dependencies) • Embedded webserver • Production ready features • Micrometer default instrumentation / observability library
- 18. Metrics: Spring Boot & Micrometer What is Micrometer? • Vendor-neutral application observability façade • Think SLF4J, but for observability • Metrics & Traces • API to instrument your application • Used by Spring projects • Support for 19 popular monitoring systems • Out of the box metrics, traces for Kafka (using Spring Kafka) Micrometer.io
- 19. Metrics: Spring Boot & Micrometer Prometheus Grafana /actuator/prometheus Scrape Metrics Kafka Consumer Application Kafka client Spring Kafka Spring Boot Application <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> Helps to manage & monitor your application • Metrics • Health check • Application info (version, git commit, etc) /actuator/metrics Query
- 20. Metrics: Spring Boot & Micrometer Prometheus Grafana Elasticsearch Kibana Kafka Consumer Application Kafka client Spring Kafka Spring Boot Application <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-elastic</artifactId> </dependency> Push metrics
- 21. Metrics: Spring Boot & Micrometer Prometheus Grafana Elasticsearch Kibana OpenTelemetry Metrics backend Kafka Consumer Application Kafka client Spring Kafka Spring Boot Application <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-otlp</artifactId> </dependency> Push metrics (OTLP protocol)
- 22. Metrics: Client side metrics Demo 0 1 2 3 4 5 6 7 8 old new Kafka topic: ‘stock-quotes’ 3 partitions Producer Spring Kafka Kafka broker Consumer Spring Kafka 0100101001101 0100101001101 poll send Rest API Slow Service Http API Rest Call Prometheus Grafana HTTP Call: Scrape Metrics Kafka Lag exporter HTTP Call: Scrape Metrics Calculate consumer lag Based on broker data
- 23. Demo
- 24. Consumer lag metrics difference, why? Problem: as a developer I don’t expect a big difference. The consumer • Reports lag only for the partition(s) it is actively consuming • High lag? Doesn't switch partitions that often • Only aware of the progress of the last offset as far as it's most recent metadata pull • consumes to that offset and thinks the lag is gone as it read up to that message. The producer • Still producing. • Actual offset grew in the meantime • Consumer is not aware of that yet More instances you start the more partition metrics will get reported
- 25. Lessons Learned Take aways • Metric precision should not be something critical • Monitoring its trend • Important: know there is lag • Lag keeps on increasing? • The consumer has a problem! • Alert on increasing trend of consumer lag
- 26. What is OpenTelemetry? + = Open Census Open Tracing OpenTelemetry What is Software Telemetry ? • Collection of data on the use, performance and behaviour of applications and their components
- 27. OpenTelemetry Components API SDK`s Collector Protocol (OTLP) Per Programming Language Functionality to collect, process and export telemetry data Encoding Transport Delivery Prometheus Kafka Jaeger Application Telemetry Data Using Exporter Library Telemetry Data Using OTLP (gRPC or http/protobuf) Telemetry Data Backend specific format Observability and Storage Library Auto(Agent) Defines the interface for instrumenting code with traces, metrics and logs
- 28. OpenTelemetry Collector Components • Optional component • No change in Application needed when switching Logging/Metrics/Tracing Backends Collector Receivers Processors Exporters
- 30. Processors (How to handle received Data) Collector Receivers Exporters Memory Limiter Batch Processor Processors Filter Processor
- 32. OpenTelemetry Demo: Components ‘stock-quotes-exchange- nyse’ ‘stock-quotes-exchange- nasdaq’ ‘stock-quotes-exchange- ams’ Producer Spring Kafka Consumer Spring Kafka poll send Shaky Downstream Service Http Rest Call Client REST Call Kafka broker Kafka Streams App poll Kafka Plain Consumer 0 1 2 3 4 5 6 7 8 0 1 2 0 1 0 1 2 3 send ‘stock-quotes’
- 33. OpenTelemetry Demo Collector Plain Kafka Consumer Application Kafka Producer Application Kafka Kafka Streams Application Logs, Metrics, Traces Telemetry data (logs, metrics & traces) Kafka Consumer Application Shaky Downstream Service
- 34. OpenTelemetry Demo Collector Plain Kafka Consumer Application Kafka Producer Application Prometheus Scrape Metrics Kafka Streams Application Time series database for Metrics Telemetry data (logs, metrics & traces) Shaky Downstream Service Kafka Consumer Application
- 35. OpenTelemetry Demo Collector Plain Kafka Consumer Application Kafka Producer Application Grafana Loki Kafka Streams Application Logs Centralized logging Telemetry data (logs, metrics & traces) Grafana Kafka Consumer Application Shaky Downstream Service
- 36. OpenTelemetry Demo Collector Plain Kafka Consumer Application Kafka Producer Application Jaeger Grafana Tempo Kafka Streams Application Traces Traces Telemetry data (logs, metrics & traces) Kafka Consumer Application Shaky Downstream Service
- 37. Demo
- 38. Wrap up • Many different ways to observe your applications • Aim for vendor neutral solutions • Helps you migrate to different observability backend • Minimal changes to your applications • Micrometer • Overlap with OpenTelemetry • JVM only • No instrumentation for Kafka Streams (Traces) yet: • https://github.com/micrometer-metrics/micrometer/issues/3713 • Can send telemetry data using OpenTelemetry • ✅ Metrics • ✅ Traces (Spring Boot 3) • ❌ Logs
- 39. Wrap up • Consumer (lag) metrics • Client side vs broker side metrics • Monitor the trend! • Select and ship only the metrics you need
- 40. Wrap up • OpenTelemetry • Language agnostic • Specification for Logs not stable • Start small • Java Agent will give you a kickstart • Minimum dev effort using auto instrumentation • Sample traces. You most likely don’t want to store 100% of all traces • Limitations • Stateful stream processing: KAFKA-7718