Testing and monitoring and broken things
- 1. Time Series Los Angeles Meetup 10/29/19 Nikki Attea Software Engineer Sensu Inc. Testing and monitoring and broken things
- 2. Hi, I’m Nikki. - 🏐🏖 - 🍷🍺 - 🐶🐶 nikki.dev @nikkixdev
- 3. How does work? Sensu is the original monitoring event pipeline. Events Sensu Event Pipeline Filter Transform Handler nikki.dev @nikkixdev
- 5. Building software is easy. nikki.dev @nikkixdev
- 8. Building software is easy. nikki.dev @nikkixdev
- 9. Building software is easy. Shipping software is hard. nikki.dev @nikkixdev
- 10. Releases should be simple, reliable, and reproducible. nikki.dev @nikkixdev
- 12. Releases were complex, chaotic, and inconsistent. nikki.dev @nikkixdev
- 13. #1: Eliminate bias by adding instruction. #2: Define acceptance criteria. #3: Automate, but don’t over-automate. #4: Implement test case management. #5: Hire employees with diverse experience. nikki.dev @nikkixdev
- 17. nikki.dev @nikkixdev Tests fail when an assertion in our code breaks.
- 18. Tests fail when an assertion in our code breaks. Monitoring detects and alerts us when our system breaks. nikki.dev @nikkixdev
- 25. nikki.dev @nikkixdev Stop! Demo time!
- 31. nikki.dev @nikkixdev What does this mean for your org?
- 32. nikki.dev @nikkixdev What does this mean for your workflow?
- 33. nikki.dev @nikkixdev What does this mean for my org?
- 34. nikki.dev @nikkixdev What does this mean for CD?
- 37. datageddon
- 42. Time-series database InfluxDB Record Data visualization Grafana Visualize Monitoring event pipeline Sensu Collect nikki.dev @nikkixdev
- 43. Application Performance Monitoring Metric Aggregation Service Checks Network & Infrastructure Monitoring nikki.dev @nikkixdev
- 44. kube-apiserver-node-01 cpu=16.4 1550625843 InfluxDB influxdb_line CPU ok - CPU = 16.4 | kube-apiserver-node-01.cpu=16.4 Nagios nagios_perfdata kube-apiserver-node-01.cpu 16.4 1550625843 Graphite graphite_plaintext kube-apiserver-node-01.cpu 1550625843 16.4 OpenTSDB opentsdb_line nikki.dev @nikkixdev
- 45. --- type: CheckConfig api_version: core/v2 metadata: name: check-cpu-influxdb namespace: default spec: command: check-cpu.sh interval: 10 subscriptions: - metrics output_metric_format: influxdb_line output_metric_handlers: - influxdb Sensu Check Configuration and Event Data --- type: Event api_version: core/v2 spec: check: metadata: name: check-cpu-influxdb output: kube-apiserver-node-01 cpu=16.4 1550625843 metrics: handlers: - influxdb points: - name: kube-apiserver-node-01.cpu value: 16.4 timestamp: 1550625843 docs.sensu.ioecho $(hostname) cpu=$(ps -A -o %cpu | awk '{s+=$1} END {print s}') $(date +%s) check-cpu.sh
- 46. --- type: Handler api_version: core/v2 metadata: namespace: default name: influxdb spec: type: pipe command: sensu-influxdb-handler -d sensu env_vars: - INFLUXDB_ADDR=http://influxdb.default.local:8086 - INFLUXDB_USER=sensu - INFLUXDB_PASS=password filters: - has_metrics Insert time series-database here, seriously github.com/sensu/sensu-influxdb-handler --- type: Event api_version: core/v2 spec: check: metadata: name: check-cpu-influxdb output: kube-apiserver-node-01 cpu=16.4 1550625843 metrics: handlers: - influxdb points: - name: kube-apiserver-node-01.cpu value: 16.4 timestamp: 1550625843
- 47. The monitoring event pipeline The Sensu Go agent is a powerful cross-platform event producer. Sensu Event Pipeline Filter HandlerTransform Sensu Client Events Agent (event producer) Nagios Plugin (service check) StatsD Metrics (instrumentation library) Prometheus (/metrics endpoint)