A practical introduction to observability
0 likes187 views
The document provides an introduction to observability, covering monitoring, logging, and distributed tracing in application systems. It emphasizes the importance of using appropriate metrics, structured logging, and symptom-based alerts to effectively manage application performance and diagnose issues. Key recommendations include avoiding high cardinality tags, leveraging percentiles for analysis, and ensuring alerts are actionable and meaningful.
A practical introduction to observability
- 1. A practical introduction to observability Nikolay Stoitsev Engineering Manager @ Halo DX
- 4. Monitoring
- 7. Monitoring
- 8. Monitoring system components Application Application Application Monitoring System Time Series Database Dashboard
- 9. Monitoring system components Application Application Application Monitoring System Time Series Database Dashboard Prometheus, Graphite, m3db
- 10. Monitoring system components Application Application Application Monitoring System Time Series Database Dashboard Prometheus UI, Grafana
- 11. Counter
- 12. Counter increase
- 13. Timer
- 14. Labels
- 15. What to watch out for?
- 16. Cardinality
- 17. Cardinality ● search.success, app_version=1, type=Patient ● search.success, app_version=1, type=Exam ● search.success, app_version=2, type=Patient ● search.success, app_version=2, type=Exam
- 18. #1. Don’t add high cardinality tags
- 19. Metrics are not accurate ● DB engine optimizes for faster operations ● When performing some operations for a different time resolution ● When archiving metrics for long term storage
- 20. #2. Don’t rely on metrics infrastructure for BI
- 21. Don’t use average values ● Averages hide the outliers ● Doesn’t represent typical behavior
- 22. Use percentiles ● Represents the worst experience in 90% of the time ● Can measure p90, p95, p99 p90
- 23. Histograms ● Shows the whole distribution ● Configurable buckets
- 24. #3. Use percentiles or histograms
- 26. Example alert
- 27. Alert Levels Send Slack/Teams Message
- 28. Alert Levels Send alert to oncall
- 29. Alerting tool is usually built into the metrics system
- 30. Alerts should be ● urgent ● important ● actionable ● real
- 31. Should represent either ongoing or imminent problems
- 32. What to watch out for?
- 33. 1. Better to remove an alert when it’s noisy
- 35. #2. Use success rate
- 36. Symptom-based monitoring ● Number of 5xx HTTP response codes ● Response time ● Email sending is not working ● Users can’t log in
- 37. Cause-based monitoring ● Free disk space on database server ● Memory utilisation ● Free file descriptors
- 38. Many causes may trigger a symptom
- 39. User impact is most important
- 40. #3. Focus on symptom-based alerts
- 41. Cause-based alerts are also necessary
- 42. Picking alerts to start with Front-end Load Balancer Back-end DB Count rate of successful log-in Count request success rate
- 44. Logging
- 48. Log messages
- 49. Finding logs Can search by: ● content of log message message : *notification* ● all logs from a service kubernetes.labels.app/name.keyword : "api-gateway" ● many more thanks to flexible query schema
- 50. What to watch out for?
- 51. #1. Use appropriate log level - info, warn, error
- 52. Structured logging ● Append useful key=value pairs ● Can group (aggregate) by the keys ● Can sort by aggregations
- 53. #2. Use structured logging
- 54. Too many logs Application Application Application Log Aggregation Real Time Search Engine Log Scraper Log Scraper Log Scraper Dashboard
- 55. Too many logs Application Application Application Log Aggregation Real Time Search Engine Log Scraper Log Scraper Log Scraper Dashboard Reduce log retention period
- 56. Too many logs Application Application Application Log Aggregation Real Time Search Engine Log Scraper Log Scraper Log Scraper Dashboard Cold Storage Query UI
- 57. #3. Use proper retention period or cold storage
- 60. End-to-end summary 1. Configure automated alerts
- 61. End-to-end summary 1. Configure automated alerts 2. Use metrics and tracing to pinpoint the problem
- 62. End-to-end summary 1. Configure automated alerts 2. Use metrics and tracing to pinpoint the problem 3. Use structured logging to find the root cause of the problem easily
- 63. End-to-end summary 1. Configure automated alerts 2. Use metrics and tracing to pinpoint the problem 3. Use structured logging to find the root cause of the problem easily 4. Fix problems and make sure all metrics are always back to normal
- 64. Thank you! Q&A Nikolay Stoitsev Engineering Manager at Halo DX Photo by Pixabay, Şahin Sezer Dinçer, Andrea Piacquadio, Ian Beckley from Pexels