[WSO2Con EU 2018] Tooling for Observability
0 likes455 views
The document discusses the concept of observability in systems, distinguishing it from monitoring and emphasizing its importance for understanding system performance and troubleshooting. It outlines various instrumentation techniques, including white box and black box methods, and introduces tools for logging, metrics monitoring, and distributed tracing, particularly in the context of WSO2 products. The conclusion highlights the ability of observability to provide in-depth insights into system functionality through integration with multiple analytical tools.
[WSO2Con EU 2018] Tooling for Observability
- 1. Director, WSO2 Tooling for Observability Anjana Fernando
- 2. What is Observability? ● A term dragged from Control Theory ● Formal definition: ○ In control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. The observability and controllability of a system are mathematical duals. (Wikipedia). ● It is a measure of how well we can understand a system from the work it does.
- 3. ● Monitoring only lets you know whether a system is functioning as expected ● Observability is an attribute of a system similar to functionality, performance, testability, maintainability, usability, etc. ● Observability is a noun, whereas monitoring is a verb Monitoring != Observability
- 4. ● Observability is achieved via instrumentation ● More instrumentation to explain what is happening on the runtime => A system is more “observable” ● WSO2 products are “observable” with various tools ● Instrumentation can be in two kinds: ○ White box instrumentation (e.g. logs, APM monitoring, distributed tracing) ○ Black box instrumentation (e.g. operating system, network communications) Observability and Instrumentation
- 5. ● The health of each end-to-end request is more important than the health of the system ● 99.99% availability doesn’t really matter if requests are not processed within the expected time (users will not be happy) Observability and Availability
- 6. ● When troubleshooting performance issues, we need to figure out the root causes ● When the system is “observable”, it’s more easier to troubleshoot Observability and Performance Engineering
- 7. The Three Pillars Of Observability Logging Metrics Monitoring Distributed Tracing Observability
- 8. Tools for Observability ● Logging ○ Elastic Stack ● Metrics ○ Application metrics ■ WSO2 Analytics Server ■ App Dynamics ○ System Metrics ■ Icinga ■ Cloud Watch ● Distributed Tracing ○ App Dynamics ○ WSO2 Analytics Server ○ OpenTracing vendors
- 9. Logging Instrumentation, Log Monitoring, and Analysis
- 10. ● White box instrumentation ● More in detail information of the application/request status/health ● Different log levels to understand the severity and details: ○ TRACE ○ DEBUG ○ INFO ○ WARN ○ ERROR ● WSO2 too uses logging standards (Log4j, etc.) which can log to a file Instrumentation
- 11. ● Elastic Stack: there are mainly four components: ○ Beats - Agents that ship data to Logstash or Elasticsearch. In our context, Filebeat will ship the WSO2 logs to Logstash. ○ Logstash - Used to process and structure the log files received from Filebeat and send to Elasticsearch. ○ Elasticsearch - Storage and indexing of the logs received by Logstash. ○ Kibana - Visualizes the data stored in Elasticsearch. Tools for Log Monitoring and Analysis
- 12. ● Elastic Stack Tools for Log Monitoring and Analysis
- 13. Tools for Log Monitoring and Analysis ● Elastic Stack - Kibana Dashboard
- 14. Metrics Monitoring Application and System Metrics Monitoring
- 15. 1) WSO2 Analytics Server Application Metrics WSO2 Servers (APIM, EI) WSO2 Analytics Server (DAS, SP)
- 16. ● Metrics monitored by WSO2 Analytics Server ○ WSO2 API Manager ■ Request/Response/Fault statistics based on APIs, resources and time range ■ API latency based on APIs, resources and time range ■ Availability of APIs ■ Many more ○ WSO2 Enterprise Integrator ■ Request/Response/Fault statistics based on proxy services/ APIs / endpoints ■ Message Latency analysis ■ Message flow with statistics and latencies Application Metrics
- 17. ● WSO2 Analytics Server - APIM Application Metrics
- 18. ● WSO2 Analytics Server - EI Application Metrics
- 19. 2) AppDynamics Application and Systems Metrics Controller Platform: SaaS or On-premise AppDynamics App Server Agent / Machine Agent Controller Event Service
- 20. ● Metrics monitored by AppDynamics ○ Request/Response/Error count statistics per services and nodes ○ Request latency statistics ○ Service and node status ○ End to end request tracing, and service dependency view ○ Systems metrics such as JVM Heap size, JVM Threads, etc. Application and Systems Metrics
- 21. AppDynamics Dashboard Application and Systems Metrics
- 22. 3) AWS Cloud Watch ● Well suited if the deployment is based on AWS instances ● Measured metrics include ○ Availability of a port ○ Availability of a service ○ Disk usage ○ CPU utilization ○ Memory utilization ○ No of processes ○ DB connectivity ○ JVM stat monitoring (heap usage, thread count) System Metrics
- 23. AWS Cloud Watch Dashboard System Metrics
- 24. 4) Icinga ● Monitors system metrics such as disk usage, load average, memory usage, JVM heap usage, availability, etc. System Metrics
- 26. 1) WSO2 Analytics Server ● A unique ID is attached to the header of the request and carried throughout the request path ● WSO2 EI and WSO2 APIM has default support for integration with WSO2 Analytics Server ● Provides ○ Message content in each entry and exit point of the service/API ○ Message before/after mediation flow ○ The time taken at each level Distributed Tracing
- 27. 1) WSO2 Analytics Server Distributed Tracing
- 29. 3) Jaeger, Zipkin, WSO2 Stream Processor ● WSO2 APIM 2.5.0 Microgateway supports OpenTracing supported tracers such as Jaeger, Zipkin Distributed Tracing
- 30. ● Future ○ OpenTracing standard (vendor neutral) instrumentation to WSO2 future releases. ○ No custom development effort is required to perform distributed tracing with any vendor that supports the OpenTracing standard ○ Can be integrated with ■ Zipkin ■ Jaeger ■ WSO2 Stream Processor 4.2.0 Distributed Tracing
- 31. ● Observability of systems lets you ○ Always be ready with a quantifiably answer to questions that developers/DevOps/users have about the systems with minimal effort ○ Have rich and in-depth mental models of how their systems function ● Observability is provided through logging, metrics monitoring and distributed tracing ● WSO2 products can be observed by integrating them to various tools, ranging from OSS (WSO2 Analytics solutions, Jaeger, Zipkin) to third party proprietary vendors (AppDynamics, AWS Cloud Watch) Conclusion