Infrastructure monitoring made easy, from ingest to insight
0 likes559 views
The document discusses the advantages of using Elastic for infrastructure metrics, highlighting the increasing complexity of monitoring in evolving architectures like cloud-native environments. It outlines key features such as centralized ingest, automatic container monitoring, and flexible alerting, emphasizing the importance of observability. The document also provides instructions for setting up and utilizing Elastic for time series data storage and visualization.
Infrastructure monitoring made easy, from ingest to insight
- 1. 1 Why you should use Elastic for Infrastructure Metrics Karl Degenhardt Senior Solutions Architect
- 2. 2 Evolving Architectures ~↑ Monitoring Complexity Hardware & software trends are evolving in tandem Higher resource utilization increases monitoring complexity • Orchestration/Hypervisor • Dynamic/ephemeral jobs • You can no longer "point" to where that job lives Shift to cloud-native yields maintainable code, with costs • Traditional licensing models don't scale as well as your applications • Hurdles with autoscaling Monitoring Complexity
- 3. 3 Applications VMs/Containers Other DBs, Services & Middleware Orchestration Infrastructure APM Metrics Logs Uptime Uptime APM Metrics APM Logs APM APM Metrics Logs Uptime Metrics Logs Uptime APM
- 4. 4 • Support the full stack • Easily ingest from new sources • Monitor dynamic ecosystems • Ability to interact with your data – Aggregations and visualizations – Different views based on who is looking • Rich and flexible alerting • Long term, reliable storage • Bonus points for full Observability Needs from a monitoring solution Core features and functionally
- 6. 6
- 7. 7 Instructions right in Kibana Growing list of integrations ● Download and install Metricbeat ● Edit the configuration for destination ● Enable and configure the module ● Start the beats ● Explore!
- 8. 8 ● Deploy Elastic Agent ● Choose the integration type ● Register and configure the data source ● Specify the data you want to collect ● Explore! Elastic Fleet Centralized ingest and configuration
- 9. 9 Use your existing shippers Core features and functionality Your App Prometheus Exporter Your App Prometheus Exporter Metricbeat + Elasticsearch Prometheus Server Metricbeat + Elasticsearch Azure Monitor
- 10. 10 Autodiscover Automatically monitor new containers ● Perfect for dynamic ecosystems ● Automatically picks up new instances ● Works with K8s, Docker, AWS, etc. ● Hints based auto-discovery for K8s ● Full context backed by Elastic Common Schema
- 11. 11 Elastic for time series
- 12. Storing Metrics in Elasticsearch ● Metrics stored as numeric fields ○ Depending on expected values: float, double, integer... ● Dimensions/labels normally stored as keyword ● Several metrics per document ○ more efficient ○ one doc per combination of dimensions (time series) { "@timestamp": "2018-09-27T10:08:38", "system": { "cpu": { "nice": 8, "user": 2, }, “load”: 1.2, }, "host": "frontend01.bigorg.dev", "zone": “europe-west”, ... } Data model
- 13. Storing Metrics in Elasticsearch { "@timestamp": "2018-09-27T10:08:38", "system": { "cpu": { "nice": 8, "user": 2, }, "load": 1.2, }, "host": "frontend01.bigorg.dev", "zone": "europe-west", ... } Correlation
- 14. 14 Elastic Common Schema Established, predictable fields
- 15. ● Several types for numbers double, integer, float depending on size needs… ● Distributed Histograms (7.6 ● IPs query by IP/subnet ● Geo Map your metrics ● Dates Rich typing and filtering Much more than single type numbers and string labels
- 16. Powerful aggregations • Common metric aggs (sum, avg, count, min, max…) • With more choices on top! – Mutate data / calculate metrics at query time with scripting – Grouping is not limited to labels: Geo proximity, filters, ranges
- 17. Index lifecycle management Reduce storage costs as data ages 1 2 3 1 2 3 Hot Nodes Cold Nodes Warm Nodes 1
- 18. Rollups Reduce storage costs as data ages
- 19. Distributed by design • Horizontally scalable • Cross cluster search • Cross cluster replication Easy to scale
- 20. 20 Powerful data store Beyond Time Series ● Inverted index + columnar store ● Optimized numeric field types (BKD ● Powerful aggregations framework ● Fast response even for high-cardinality queries ● ILM & Data Rollups ● With all of the benefits of the Elastic Stack
- 21. 21 Making metrics actionable with Elastic
- 22. 22 Dashboards & Visualizations Out-of-the-box visibility ● Ship with most integrations ● Mix and match for your needs ● Leverage Kibana drilldowns for custom navigation paths ● Of course, dedicated Metrics and Logs apps
- 23. 23 Metrics App Birds-eye view or drill down
- 24. 24 Integrated Alerting Automatically detect and alert ● Many types of alerts ● Prefiltering based on context ● Multiple facets per alert ○ CPU and Memory ○ Network TX and RX ● Automatically split alerts on chosen field (per container/pod/host) ● Deviations in logging rates
- 25. 25 Machine Learning Automatically detect and alert ● Automate anomaly detection at scale and across disparate data sources ● Find patterns in your logs ● Automatically call out anomalies and outliers
- 26. 26 Full Observability Unified data, UI and alerting
- 27. 27 Thank You!