Metrics, Logs, Transaction Traces, Anomaly Detection at Scale
3 likes4,864 views
This document discusses Sematext's monitoring and logging products and services. It introduces Sematext, which is headquartered in Brooklyn and has employees globally. It then discusses why performance monitoring, log searching, and anomaly alerting are needed capabilities (Why). The document proceeds to describe Sematext's SPM and Logsene products, which provide these capabilities using open source technologies like OpenTSDB, Elasticsearch, and Kafka. It covers how the SPM agent collects metrics and traces and how Logsene ingests and analyzes logs at scale.
Metrics, Logs, Transaction Traces, Anomaly Detection at Scale
- 1. Processing Metrics, Logs & Traces … at Scale Otis Gospodnetić
- 4. WHO Otis Gospodnetić Sematext founder Apache member Book author ex-Lucene/Solr dev
- 5. WHO Services Solr Elasticsearch* Kafka Spark HBase Cassandra... * We’ve got serious Solr & Elasticsearch ninjas on the team!
- 6. WHY
- 8. WHO Before you can fix things need to know what to fix WHY
- 9. WHY We need….INSIGHT Performance Metrics! Anomalies! Logs!
- 10. WHY i.e. Need Tools! Metrics monitoring Log searching Anomaly alerting
- 11. OSS Use the (Open) Source, Luke
- 14. OSS “I have an ELK stack that has been suffering as of late. The logstash service will continually crash, the elasticsearch cluster is hardly in the green, and it is taking a constant amount of maintenance.”
- 15. WHAT
- 16. WHAT SPM → monitoring Logsene → logging On PremisesCloud http://sematext.com/spm http://sematext.com/logsene
- 18. WHAT SPM Logsene
- 19. HOW
- 21. WHAT Agent Java Node.js Want Traces? Embed it! Collectd ⇒ SIGAR for OS Flume SpilloverChannel ES API
- 22. WHAT Interesting finds Variable Collectd support Collectd ⇒ SIGAR Apache Flume Elasticsearch Stats API Metrics 2nd class citizen
- 23. WHAT Transaction Tracing Java Bytecode Instrumentation Bottleneck finder AppMap maker
- 26. WHAT Custom Pointcuts <method signature="java.lang.String com.company. example.Service#getUserName(com.company.model. Company company)"/>
- 31. WHAT Anomaly Detection ExponentialSTDFromMA KNN ... boolean result = anomalyCount / (notAnomalyCount + anomalyCount) >= 3d / 4d;
- 32. WHAT Anomaly issues Warn early / create noise Normal abnormalities Slow change
- 35. Logging
- 36. Hot vs. Cold HOT COLD
- 37. Drop, don’t Delete HOT COLD drop
- 38. Pull, don’t Push GET QUEUE pull ES
- 39. Beware of Aggregations Circuit Breakers