Signalilo: Visualizing Prometheus alerts in Icinga2 - Icinga Camp Zurich 2019
2 likes2,257 views
VSHN is a DevOps company based in Zurich, Switzerland with 42 employees specializing in DevOps, Docker, Kubernetes, and cloud operations. It has been in business since 2014 and has been cash flow positive since 2015. VSHN values taking responsibility, expert knowledge, cost savings through automation, equal opportunity employment, and collaboration with software developers. It is the leading Swiss partner for DevOps services and the first Kubernetes Certified Service Provider in Switzerland.
![VSHN - The DevOps Company 99
Icinga ApiUser
object ApiUser "signalilo_cluster.appuio.ch" {
password = "<redacted>"
permissions = [
{ permission = "objects/query/*"
filter = {{ host.name == "signalilo_cluster.appuio.ch" }}
},
{ permission = "objects/create/service"
filter = {{ host.name == "signalilo_cluster.appuio.ch" }}
},
{ permission = "objects/modify/service"
filter = {{ host.name == "signalilo_cluster.appuio.ch" }}
},
{ permission = "objects/delete/service"
filter = {{ host.name == "signalilo_cluster.appuio.ch" }}
},
{ permission = "actions/process-check-result"
filter = {{ host.name == "signalilo_cluster.appuio.ch" }}
}, ]
}](https://image.slidesharecdn.com/signalilo-visualizingprometheusalertsinicinga-191118083143/85/Signalilo-Visualizing-Prometheus-alerts-in-Icinga2-Icinga-Camp-Zurich-2019-9-320.jpg)
Signalilo: Visualizing Prometheus alerts in Icinga2 - Icinga Camp Zurich 2019
- 1. VSHN - The DevOps Company © 2018 VSHN AG VSHN AG The DevOps Company ICINGA Camp - GOLD SPONSOR
- 2. VSHN - The DevOps Company ● 42 employees, 25+ MSc. / BSc. in Computer Science ● Owner-managed, no external investors ● SCRUM, KanBan, Agile ● Offices location directly at Zurich HB ● Founded in 2014, cash flow positive since 2015 22 Our Company Our Values ● Take responsibility and guarantee 100% reliability ● Expert knowledge and Swiss quality in all work steps ● Cost savings through automation and standardization ● Equal Employment Opportunity ● Operation of software on any infrastructure - whether cloud or in-house ● Collaboration with software developers - stability despite agility ● Automation of DevOps processes ● First Kubernetes Certified Service Provider in Switzerland ● ISO27001 certified and FINMA audited Business & Operations processes Our Business
- 3. Learn more at: https://vshn.ch/ 3 VSHN (pronounced ˈvɪʒn like "vision") is the leading Swiss partner for DevOps, Docker, Kubernetes, OpenShift and 24/7 Cloud Operations. We look forward to hearing from you. Follow us on Twitter! @vshn_ch VSHN AG - Neugasse 10 - CH-8005 Zurich - Switzerland - +41 44 545 53 00 - https://vshn.ch/ - info@vshn.ch
- 4. VSHN - The DevOps Company Signalilo Visualizing Prometheus Alerts in Icinga Simon Gerber, VSHN AG
- 5. VSHN - The DevOps Company 55 Monitoring at VSHN
- 6. VSHN - The DevOps Company 66 Monitoring OpenShift ● 25 OpenShift Clusters ○ Custom Icinga checks ● Since OpenShift 3.11 ○ Prometheus ○ Alertmanager
- 7. VSHN - The DevOps Company 77 Signalilo Alertmanager OpenShift cluster Signalilogenerated alerts webhook VSHN API calls
- 8. VSHN - The DevOps Company 88 Prometheus alerts in Icinga
- 9. VSHN - The DevOps Company 99 Icinga ApiUser object ApiUser "signalilo_cluster.appuio.ch" { password = "<redacted>" permissions = [ { permission = "objects/query/*" filter = {{ host.name == "signalilo_cluster.appuio.ch" }} }, { permission = "objects/create/service" filter = {{ host.name == "signalilo_cluster.appuio.ch" }} }, { permission = "objects/modify/service" filter = {{ host.name == "signalilo_cluster.appuio.ch" }} }, { permission = "objects/delete/service" filter = {{ host.name == "signalilo_cluster.appuio.ch" }} }, { permission = "actions/process-check-result" filter = {{ host.name == "signalilo_cluster.appuio.ch" }} }, ] }
- 10. VSHN - The DevOps Company 1010 Icinga Host object Host "signalilo_cluster.appuio.ch" { display_name = "Signalilo signalilo_cluster.appuio.ch" check_command = "dummy" enable_passive_checks = false enable_perfdata = false vars.customer = "appuio" vars.ignore_service_icinga = true }
- 11. VSHN - The DevOps Company 1111 Icinga Heartbeat Service template Service "signalilo-heartbeat-template" { vars.dummy_text = {{ var service = get_service(macro("$host.name$"), macro("$service.name$")) var lastCheck = DateTime(service.last_check).to_string() return "Missing heartbeat pings from Signalilo. Last ping received at: " + lastCheck }} } object Service "heartbeat" { import "signalilo-heartbeat-template" host_name = "signalilo_cluster.appuio.ch" check_command = "dummy" check_interval = 300s max_check_attempts = 1 enable_perfdata = false vars.dummy_state = 1 }
- 12. VSHN - The DevOps Company Signalilo and Alertmanager Config 1212 containers: - env: - name: SIGNALILO_UUID value: <redacted> - name: SIGNALILO_ICINGA_HOSTNAME value: signalilo_cluster.appuio.ch - name: SIGNALILO_ICINGA_URL value: https://<vshn icinga>:5665 - name: SIGNALILO_ICINGA_CA value: |- -----BEGIN CERTIFICATE----- <snip> -----END CERTIFICATE----- - name: SIGNALILO_ICINGA_USERNAME value: signalilo_cluster.appuio.ch - name: SIGNALILO_ICINGA_PASSWORD value: <redacted> # or from Kubernetes secret - name: SIGNALILO_ALERTMANAGER_BEARER_TOKEN value: <redacted> # or from Kubernetes secret receivers: - name: default webhook_configs: - send_resolved: true http_config: bearer_token: <secret> url: http://signalilo.appuio-monitoring/webhook Alermanager Signalilo
- 13. VSHN - The DevOps Company 1313 Alerts as Services
- 14. VSHN - The DevOps Company 1414 Alerts as Services ● Passive service ● Service state updated by Signalilo ● alertname annotation as service display name ● message annotation as check output ● severity label translated to service state
- 15. VSHN - The DevOps Company 1515 Alerts as Services ● All alert annotations and labels visible as custom variables ● Services in OK state are GCd by Signalilo ○ state for GC process is stored as custom variables on alert
- 16. VSHN - The DevOps Company 1616 Alerts as Services ● The same alert results in update to same Icinga service ● Alert identity defined as set of labels and annotations ○ except for severity label
- 17. Signalilo is Open Source Coming Soon! https://github.com/vshn/signalilo 17
- 18. Come visit us for a coffee! VSHN AG - Neugasse 10 - CH-8005 Zürich - +41 44 545 53 00 - https://vshn.ch/ - info@vshn.ch https://vshn.ch/kontakt/ Follow us on Twitter! @vshn_ch 18