Testing and Monitoring and Broken Things | Nikki Attea | Sensu
0 likes176 views
Nikki Attea is a software engineer at Sensu Inc. who gave a presentation on testing, monitoring, and the Sensu monitoring event pipeline. The presentation covered how Sensu works to collect events from clients and filters, transforms, and routes them to handlers. It also discussed using Sensu to collect metrics and send them to time-series databases like InfluxDB for storage and visualization in Grafana.
Testing and Monitoring and Broken Things | Nikki Attea | Sensu
- 1. InfluxDays SF 2019 Nikki Attea Software Engineer Sensu Inc. Testing and monitoring and broken things
- 2. Hi, I’m Nikki. - 🏐🏖 - 🍷🍺 - 🐶🐶 nikki.dev @nikkixdev Hi, I’m Nikki. - 🏐🏖 - 🍷🍺 - 🐶🐶 nikki.dev @nikkixdev Hi, I’m Nikki. - 🏐🏖 - 🍷🍺 - 🐶🐶 nikki.dev @nikkixdev
- 3. How does work? Sensu is the original monitoring event pipeline. Events Sensu Event Pipeline Filter Transform Handler nikki.dev @nikkixdev How does work? Sensu is the original monitoring event pipeline. Events Sensu Event Pipeline Filter Transform Handler nikki.dev @nikkixdev
- 5. Building software is easy. nikki.dev @nikkixdev
- 8. Building software is easy. nikki.dev @nikkixdev
- 9. Building software is easy. Shipping software is hard. nikki.dev @nikkixdev
- 10. Releases should be simple, reliable, and reproducible. nikki.dev @nikkixdev
- 12. Releases were complex, chaotic, and inconsistent. nikki.dev @nikkixdev
- 13. #1: Eliminate bias by adding instruction. #2: Define acceptance criteria. #3: Automate, but don’t over-automate. #4: Implement test case management. #5: Hire employees with diverse experience. nikki.dev @nikkixdev #1: Eliminate bias by adding instruction. #2: Define acceptance criteria. #3: Automate, but don’t over-automate. #4: Implement test case management. #5: Hire employees with diverse experience. nikki.dev @nikkixdev
- 17. nikki.dev @nikkixdev Tests fail when an assertion in our code breaks.
- 18. Tests fail when an assertion in our code breaks. Monitoring detects and alerts us when our system breaks. nikki.dev @nikkixdev
- 19. nikki.dev @nikkixdev monitoring testing breakage nikki.dev @nikkixdev monitoring testing breakage
- 20. nikki.dev @nikkixdev monitoring testing breakage systems/infrastructure code/product nikki.dev @nikkixdev monitoring testing breakage systems/infrastructure code/product
- 21. nikki.dev @nikkixdev monitoring testing breakage systems/infrastructure code/product nikki.dev @nikkixdev monitoring testing breakage systems/infrastructure code/product
- 22. nikki.dev @nikkixdev monitoring testing breakage systems/infrastructure code/product state nikki.dev @nikkixdev monitoring testing breakage systems/infrastructure code/product state
- 25. nikki.dev @nikkixdev Stop! Demo time! nikki.dev @nikkixdev Stop! Demo time!
- 30. nikki.dev @nikkixdev What’s next? nikki.dev @nikkixdev What’s next?
- 31. nikki.dev @nikkixdev What does this mean for your org? nikki.dev @nikkixdev What does this mean for your org?
- 32. nikki.dev @nikkixdev What does this mean for your workflow?
- 33. nikki.dev @nikkixdev What does this mean for my org? nikki.dev @nikkixdev What does this mean for my org?
- 34. nikki.dev @nikkixdev What does this mean for CD?
- 37. datageddon
- 38. nikki.dev @nikkixdevnikki.dev @nikkixdevnikki.dev @nikkixdev
- 42. Time-series database InfluxDB Record Data visualization Grafana Visualize Monitoring event pipeline Sensu Collect nikki.dev @nikkixdev Time-series database InfluxDB Record Data visualization Grafana Visualize Monitoring event pipeline Sensu Collect nikki.dev @nikkixdev
- 43. Application Performance Monitoring Metric Aggregation Service Checks Network & Infrastructure Monitoring nikki.dev @nikkixdev Application Performance Monitoring Metric Aggregation Service Checks Network & Infrastructure Monitoring nikki.dev @nikkixdev
- 44. kube-apiserver-node-01 cpu=16.4 1550625843 InfluxDB influxdb_line CPU ok - CPU = 16.4 | kube-apiserver-node-01.cpu=16.4 Nagios nagios_perfdata kube-apiserver-node-01.cpu 16.4 1550625843 Graphite graphite_plaintext kube-apiserver-node-01.cpu 1550625843 16.4 OpenTSDB opentsdb_line nikki.dev @nikkixdev
- 45. --- type: CheckConfig api_version: core/v2 metadata: name: check-cpu-influxdb namespace: default spec: command: check-cpu.sh interval: 10 subscriptions: - metrics output_metric_format: influxdb_line output_metric_handlers: - influxdb Sensu Check Configuration and Event Data --- type: Event api_version: core/v2 spec: check: metadata: name: check-cpu-influxdb output: kube-apiserver-node-01 cpu=16.4 1550625843 metrics: handlers: - influxdb points: - name: kube-apiserver-node-01.cpu value: 16.4 timestamp: 1550625843 docs.sensu.ioecho $(hostname) cpu=$(ps -A -o %cpu | awk '{s+=$1} END {print s}') $(date +%s) check-cpu.sh
- 46. --- type: Handler api_version: core/v2 metadata: namespace: default name: influxdb spec: type: pipe command: sensu-influxdb-handler -d sensu env_vars: - INFLUXDB_ADDR=http://influxdb.default.local:8086 - INFLUXDB_USER=sensu - INFLUXDB_PASS=password filters: - has_metrics Insert time series-database here, seriously github.com/sensu/sensu-influxdb-handler --- type: Event api_version: core/v2 spec: check: metadata: name: check-cpu-influxdb output: kube-apiserver-node-01 cpu=16.4 1550625843 metrics: handlers: - influxdb points: - name: kube-apiserver-node-01.cpu value: 16.4 timestamp: 1550625843
- 47. The monitoring event pipeline The Sensu Go agent is a powerful cross-platform event producer. Sensu Event Pipeline Filter HandlerTransform Sensu Client Events Agent (event producer) Nagios Plugin (service check) StatsD Metrics (instrumentation library) Prometheus (/metrics endpoint) The monitoring event pipeline The Sensu Go agent is a powerful cross-platform event producer. Sensu Event Pipeline Filter HandlerTransform Sensu Client Events Agent (event producer) Nagios Plugin (service check) StatsD Metrics (instrumentation library) Prometheus (/metrics endpoint)