Declarative observability management for Microservice architectures
0 likes76 views
This document discusses challenges with monitoring microservice architectures and proposes a solution strategy using a service mesh and API gateway. It outlines how these tools can be used to collect observability data across application and infrastructure layers in a consistent way. Two use cases are presented: monitoring a data platform and an integration platform. The conclusion emphasizes that applications and systems must be designed to be monitorable and that the amount of monitoring data collected should be limited and focused on key metrics like latency, traffic, errors and saturation.
Declarative observability management for Microservice architectures
- 1. © OPITZ CONSULTING 2023 / Interner Gebrauch Declarative o11y management for your µArchitectures 1 Gummersbach, June 3, 2023 Sven Bernhardt DECLARATIVE O11Y MANAGEMENT FOR MICROSERVICE ARCHITECTURES
- 2. © OPITZ CONSULTING 2023 / Interner Gebrauch THAT‘S ME Declarative o11y management for your µArchitectures 2 Sven Bernhardt Cloud-Native enthusiast, API & integration geek. Always curious how new technologies and concepts can help to make things more valuable and efficient. @sbernhardt https://svenbernhardt.wordpress.com/ Chief Architect / Integration Evangelist OPITZ CONSULTING Deutschland GmbH
- 3. © OPITZ CONSULTING 2023 / Interner Gebrauch Declarative o11y management for your µArchitectures 3 CHALLENGES 01 SOLUTION STRATEGY 02 CONCLUSION 04 USE CASES 03
- 4. © OPITZ CONSULTING 2023 / Interner Gebrauch Declarative o11y management for your µArchitectures 4 CHALLENGES 01
- 5. © OPITZ CONSULTING 2023 / Interner Gebrauch PREPARE FOR THE UNKNOWN Declarative o11y management for your µArchitectures 5 Things we are aware of and understand Things we understand, but are not aware of Things we are aware of but we don’t understand Things we are neither aware of nor understand
- 6. © OPITZ CONSULTING 2023 / Interner Gebrauch MONITORING Declarative o11y management for your µArchitectures 7 ¢ Historically: Up/Down-Status of a system ¢ Now: Depict the overall health status of a system ¢ The monitoring system should answer ¢ What is broken? ¢ Why is it broken? ¢ Usually includes levels ¢ Network ¢ Machine ¢ Application Quote from the SRE Book by Google “Your monitoring system should address two questions: what’s broken, and why? The “what’s broken” indicates the symptom; the “why” indicates a (possibly intermediate) cause. “What” versus “why” is one of the most important distinctions in writing good monitoring with maximum signal and minimum noise.”
- 7. © OPITZ CONSULTING 2023 / Interner Gebrauch EXAMPLE: SYMPTOMS & CAUSE Declarative o11y management for your µArchitectures 8 Google SRE Book about the correlation between symptoms and causes: “Note that in a multi-layered system, one person’s symptom is another person’s cause!” Symptom (“What?”) Cause (“Why?”) Webserver responds with 500 or 404 Database doesn’t accept new connections, as it is in maintenance mode Connection reset while initializing database connection pool Wrongly configured firewall rules (IP ranges) Slow responses for web frontend Backend system is busy with wrongly configured batch job
- 8. © OPITZ CONSULTING 2023 / Interner Gebrauch OBSERVABILITY – JUST A FANCY NEW TERM? Declarative o11y management for your µArchitectures 9 ¢ Oftentimes defined as a superset of monitoring ¢ Monitoring ¢ Alerting/visualization ¢ Distributed systems tracing infrastructure ¢ Log aggregation/analytics ¢ Goals: ¢ Enhance debuggability of a system ¢ Learn as much as you can about your system environment SRE book what isn’t the goal of monitoring: ”It can be tempting to combine monitoring with other aspects of inspecting complex systems, such as detailed system profiling, single-process debugging, tracking details about exceptions or crashes, load testing, log collection and analysis, or traffic inspection. While most of these subjects share commonalities with basic monitoring, blending together too many results in overly complex and fragile systems.”
- 9. © OPITZ CONSULTING 2023 / Interner Gebrauch CHARACTERISTICS OF MODERN MONITORING SOLUTIONS Declarative o11y management for your µArchitectures 10 ¢ Must be ¢ Understood as platform ¢ Simple to use ¢ Simple to onboard new applications to ¢ Should be built upon open standards (e.g. Open Telemetry) ¢ Should only collect data from which one can make sense of This is where many traditional monitoring solutions are strong
- 10. © OPITZ CONSULTING 2023 / Interner Gebrauch APPS / SYSTEMS MUST BE MONITORABLE! Declarative o11y management for your µArchitectures 11 ¢ Modern software today should be “able” ¢ “Able”-Criteria: Verifi Test Oper Read Evolv Maintain Observ Scal Recover Prov Afford Deploy Audit Impenetr Debugg Credits to Lucas Jellema: https://www.slideshare.net/lucasjellema/code-with-honor-on- the-importance-and-beauty-of-software-programming-rmoug-2021-training-days Able
- 11. © OPITZ CONSULTING 2023 / Interner Gebrauch FROM MONOLITHIC TO CLOUD-NATIVE ARCHITECTURES Declarative o11y management for your µArchitectures 13 Traditional approach Monolithic architecture Cloud-native approach Microservice architecture
- 12. © OPITZ CONSULTING 2023 / Interner Gebrauch FROM CENTRALIZED TO DECENTRALIZED ARCHITECTURES Declarative o11y management for your µArchitectures 14 Centralized STATIC ON-PREM MONOLITH VIRTUAL MACHINES MANUAL CHANGE PROCESS Decentralized DYNAMIC CLOUD / MULTI-CLOUD MICROSERVICES / SERVERLESS CONTAINERS, KUBERNETES AUTOMATED CI/CD TOOL CHAIN # Services & APIs CONTROL AND VISIBILITY
- 13. © OPITZ CONSULTING 2023 / Interner Gebrauch CONSISTENCE AND EFFICIENCY CHALLENGE Declarative o11y management for your µArchitectures ¢ Multi-layered Observability challenge ¢ Edge ¢ App ¢ In-App 15
- 14. © OPITZ CONSULTING 2023 / Interner Gebrauch Declarative o11y management for your µArchitectures 16 SOLUTION STRATEGY 02
- 15. © OPITZ CONSULTING 2023 / Interner Gebrauch CONCEPT: API GATEWAY Declarative o11y management for your µArchitectures 17 ¢ Single entry point for clients to access Services ¢ No matter the implementation technology ¢ No matter the deployment model (Monolithic or µService) ¢ Provides a consistent governance model ¢ Decouples Client and Service implementation ¢ Is deployed separately in its own instance ¢ Deployment models: ¢ Bundled data and control plane ¢ Independent data and control plane Source: https://tinyurl.com/yxbds3cd
- 16. © OPITZ CONSULTING 2023 / Interner Gebrauch CONCEPT: SERVICE MESH Declarative o11y management for your µArchitectures 18 ¢ Efficient implementation of cross-cutting concerns with respect to service integration challenges ¢ Everything is a service! ¢ Cloud-native apps deployed to Kubernetes ¢ Non Cloud-native workloads ¢ Should be independent of ¢ Architecture (e.g. Monolithic or µService) ¢ Platform (e.g. VMs, Containers, Kubernetes) Dedicated infrastructure layer that makes service-to-service communication more reliable, secure and observable
- 17. © OPITZ CONSULTING 2023 / Interner Gebrauch API GATEWAY & SERVICE MESH TO EMPOWER E2E CONNECTIVITY Declarative o11y management for your µArchitectures 19 ¢ API Gateway is integrated in the Mesh ¢ Is just another service ¢ Gateway proxy just handles outbound traffic ¢ Benefits: ¢ Increased Developer experience ¢ Seamless observability ¢ Consistent security ¢ Resilience & Reliable connectivity ¢ Flexibility GW DP CLIENT PUBLIC TRAFFIC GW DP MESH CP MESH 1 MESH 2
- 18. © OPITZ CONSULTING 2023 / Interner Gebrauch Business Logic (B) Policy (P) SHARED POLICY LAYER Declarative o11y management for your µArchitectures 20 OTHER MONOLITH MICROSERVICES 3RD-PARTY Routing Load Balancing Authentication Authorization Logs & Metrics Rate Limiting Caching Validation Transformation MULTI-CLOUD & KUBERNETES ON-PREMISE HOSTING B Shared Policy Layer (Kong Gateway) B B B B B B B B B Serverless (e.g. AWS Lambda) More applications and services More languages and protocols REST} { More deployment types
- 19. © OPITZ CONSULTING 2023 / Interner Gebrauch COLLECTING DATA AT THE EDGE LEVEL Declarative o11y management for your µArchitectures 21 ¢ Kong Plugins to emit respective data ¢ HTTP / TCP Log ¢ Prometheus ¢ Zipkin ¢ Open Telemetry ¢ Kong EE provides more information OOTB (Vitals) ¢ # API calls (per API resource) ¢ # Errors / Successful requests ¢ Gateway might be deployed as ¢ Kubernetes Ingress Controller ¢ Standalone Gateway (on VM or Bare Metal)
- 20. © OPITZ CONSULTING 2023 / Interner Gebrauch COLLECTING DATA AT THE APP-LEVEL Declarative o11y management for your µArchitectures 22 ¢ Kuma Observability policies are used to emit needed data ¢ TrafficLog ¢ TrafficMetrics ¢ TrafficTrace ¢ Metrics data can be collected for Data and Control plane ¢ Insights into Mesh Topology with Service Map ¢ Options for Mesh Gateway ¢ Kong ¢ Kubernetes Gateway API (if operated on K8s)
- 21. © OPITZ CONSULTING 2023 / Interner Gebrauch ANALYZING AND MONITORING THE DATA Declarative o11y management for your µArchitectures ¢ Using Grafana Stack to create a 360-degree view ¢ Component usage: ¢ Visualization: Grafana ¢ Logging: Loki (Log Shipping: FluentD / FluentBit / Promtail) ¢ Metrics: Prometheus (for long-term storage Cortex / Thanos) ¢ Tracing: Tempo ¢ Alerting: Prometheus Alert Manager ¢ Operating models ¢ Self-managed on-prem ¢ Grafana SaaS offering 23
- 22. © OPITZ CONSULTING 2023 / Interner Gebrauch CONCEPTUAL O11Y ARCHITECTURE Declarative o11y management for your µArchitectures 24 ¢ Flexible, cloud-agnostic approach ¢ Independent of architecture and platform ¢ VM / Bare Metal ¢ Containers / K8s ¢ Cloud / On-prem ¢ Easily extensible ¢ Completely based on Open Source ¢ Declarative approach (no code changes)
- 23. © OPITZ CONSULTING 2023 / Interner Gebrauch Declarative o11y management for your µArchitectures 25 USE CASES 03
- 24. © OPITZ CONSULTING 2023 / Interner Gebrauch #1 DATA PLATFORM – CHALLENGE / GOALS Declarative o11y management for your µArchitectures 26 ¢ Distributed on-prem architecture based on VMs and Containers ¢ Ensure SLAs and operability of the data platform ¢ Insights to data access and processing in a Data Lake scenario ¢ Using o11y data to being able to analyze and optimize data access and processing ¢ Ingestion ¢ Processing ¢ Analysis
- 25. © OPITZ CONSULTING 2023 / Interner Gebrauch #1 DATA PLATFORM – SOLUTION ARCHITECTURE Declarative o11y management for your µArchitectures 27 ¢ On-prem only scenario ¢ Challenge for platform team ¢ Patching / Upgrading ¢ Backup / Recovery ¢ Extending platform ¢ Monitoring platform ¢ Host-level: Checkmk ¢ App-level: Grafana Stack (Self-hosted)
- 26. © OPITZ CONSULTING 2023 / Interner Gebrauch #1 DATA PLATFORM – SOLUTION ARCHITECTURE Declarative o11y management for your µArchitectures 28 ¢ O11y data is delivered by agents: ¢ Host Metrics: Checkmk Agents ¢ Mesh / App Metrics: Envoy, Prometheus Exporters, Spring Boot Actuator ¢ Logs: Envoy, Promtail, FluentD ¢ Traces: Envoy, Spring Boot / Micrometer ¢ Agent distribution: ¢ Automation of agent distribution (Ansible) ¢ Envoy proxies for apps are automatically initiated (if applicable)
- 27. © OPITZ CONSULTING 2023 / Interner Gebrauch #2: INTEGRATION PLATFORM – CHALLENGES / GOALS Declarative o11y management for your µArchitectures 29 ¢ Distributed hybrid architecture based on VMs and Containers ¢ Ensure SLAs and operability of the integration platform (critical system) ¢ Insights to cloud-native integration flows ¢ Support API lifecycle management ¢ Support platform team to proatively operate the platform ¢ Increase debugability of the distributed platform in case of application errors ¢ Transparency about data usage ¢ Monitor overall platform state (not only infra) ¢ Insight into data flows with respect to state & performance
- 28. © OPITZ CONSULTING 2023 / Interner Gebrauch #2: INTEGRATION PLATFORM – SOLUTION ARCHITECTURE Declarative o11y management for your µArchitectures 30 ¢ Distributed, hybrid architecture ¢ Monitoring Platform: ¢ Grafana Stack (Grafana Cloud) ¢ O11y data is delivered by agents: ¢ Mesh / App Metrics: Envoy, Prometheus Exporters, Spring Boot Actuator ¢ Logs: Envoy ¢ Traces: Envoy, Java agent
- 29. © OPITZ CONSULTING 2023 / Interner Gebrauch #2: INTEGRATION PLATFROM – TRACING EXAMPLE VIEW Declarative o11y management for your µArchitectures 31
- 30. © OPITZ CONSULTING 2023 / Interner Gebrauch #2: INTEGRATION PLATFROM – EXAMPLE ERROR DASHBOARD Declarative o11y management for your µArchitectures 32
- 31. © OPITZ CONSULTING 2023 / Interner Gebrauch Declarative o11y management for your µArchitectures 33 CONCLUSION 04
- 32. © OPITZ CONSULTING 2023 / Interner Gebrauch HOPE IS NOT A STRATEGY! Declarative o11y management for your µArchitectures 34 ¢ We need to build monitorable apps / systems ¢ Having a consistent Monitoring strategy is key to build sustainable solutions ¢ NOTE: Having good monitoring in place doesn’t mean that we compeltly get rid of failures ¢ Just collect important information from which you make sense, e.g. in dashboards or alerts Source: Unsplash
- 33. © OPITZ CONSULTING 2023 / Interner Gebrauch THINK IN PLATFORMS! Declarative o11y management for your µArchitectures 35 ¢ Declarative o11y with API Gateway & Service Mesh ¢ Provide a shared policy layer to configure respective aspects ¢ Think in platforms! ¢ Make it easy to use, operate and extend ¢ For product teams (Developer experience) ¢ For platform teams
- 34. © OPITZ CONSULTING 2023 / Interner Gebrauch WHAT SHOULD BE MONITORED? Declarative o11y management for your µArchitectures 36 ¢ The amount of monitoring data is almost infinite ¢ Limit the amount of data collected to relevant ones ¢ Focus on the four golden signals mentioned in Google SRE Book: ¢ Latency ¢ Traffic ¢ Errors ¢ Saturation
- 35. © OPITZ CONSULTING 2023 / Interner Gebrauch Declarative o11y management for your µArchitectures 38 Q & A
- 36. © OPITZ CONSULTING 2023 / Interner Gebrauch IN CASE OF ANY QUESTIONS… Declarative o11y management for your µArchitectures 39 Sven Bernhardt Chief Architect / Integration Evangelist | Oracle ACE Director OPITZ CONSULTING Deutschland GmbH Kirchstrasse 6, 51647 Gummersbach, Germany Phone: +49 172 2193529 Mail: sven.bernhardt@opitz-consulting.com Twitter: @sbernhardt Blog: https://svenbernhardt.wordpress.com