Dev.bg DevOps March 2024 Monitoring & Logging
0 likes196 views
The document discusses Marian Marinov's experience monitoring various systems and infrastructure. He aims to have a single solution for log and metrics collection but ends up with multiple Grafana dashboards and different log collectors. Collectd was found to be the easiest to set up and provides the most out-of-the-box metrics, while solutions like Elasticsearch and Kibana require too many resources for smaller setups. There is no single solution that can monitor everything.
Dev.bg DevOps March 2024 Monitoring & Logging
- 1. Следете актуалните обяви за DevOps Партньори: Monitoring & Logging Marian Marinov mm@yuhu.biz
- 2. Следете актуалните обяви за DevOps Партньори: Who am I? ● Director of Engineering at Web Hosting Canada ● Former partner and Head of DevOps at SiteGround ● A SysAdmin and System Architect
- 3. Следете актуалните обяви за DevOps Партньори: What I have to monitor? ● 13 physical linux machines ○ Storage capacity (df/df -i) ○ S.M.A.R.T. of the drives ○ RAID (HW or Soft) ○ Network (routes, traffic and usage) ○ Performance (CPU, Mem, I/O, Processes) ○ Kernel logs ○ Service logs
- 4. Следете актуалните обяви за DevOps Партньори: What I have to monitor? ● 1 UPS ● 2 APC PUDs ● 2 Switches (SNMP statistics) ● 2 Thermostat (traffic, temp, humidity) ● 40+ LXC containers ○ Performance (CPU, Mem, I/O, Processes) ○ Storage capacity (df/df -i) ○ Service logs ● 2-3 Wifi access points ○ number of attached devices ○ traffic per-device
- 5. Следете актуалните обяви за DevOps Партньори: What I have to monitor? ● A few things for which I want traffic and power on time ○ 3 TVs ○ 3 Amplifiers ○ 4 Cameras ○ 1 Washing machine ○ 1 Dryer
- 6. Следете актуалните обяви за DevOps Партньори: What I wanted ● Single solution for log and metrics collection ● Single central interface
- 7. Следете актуалните обяви за DevOps Партньори: What I ended up having ● multiple grafana dashboards ● monitor events, instead of reading logs ● a bunch of different log collectors
- 8. Следете актуалните обяви за DevOps Партньори: What tested ● syslog-ng ● rsyslog ● Filebeat ● Prometheus node_exporter ● Loki ● Fluentd ● Clolectd ● StatsD ● Graylog ● PostgreSQL+timescale ● Grafana
- 9. Следете актуалните обяви за DevOps Партньори: Conclusions ● there is no one solution to rule them all ● SNMP is still the king for networking ● too many logging formats and DSLs
- 10. Следете актуалните обяви за DevOps Партньори: Conclusions ● there is no one solution to rule them all ● SNMP is still the king for networking ● too many logging formats and DSLs ● collectd was the easiest ○ with the most metrics out-of-the-box
- 11. Следете актуалните обяви за DevOps Партньори: Conclusions ● there is no one solution to rule them all ● SNMP is still the king for networking ● too many logging formats and DSLs ● collectd was the easiest ○ with the most metrics out-of-the-box ● ElasticSearch + Kibana require too much resources ○ Not usable for smaller setups ● Graylog uses a lot of CPU for the work it does ○ alerts can be based on number of events instead of parsing logs
- 12. Следете актуалните обяви за DevOps Партньори: Installation / Setup ● basic apt-get: ○ rsyslogd, syslog-ng, fluentd, collectd, filebeat, loki, node_exporter ○ statsd wanted full npm
- 13. Следете актуалните обяви за DevOps Партньори: Pros and Cons ● Syslog pros ○ can easily ingest netconsole kernel logging ○ very good performance ○ well documented and standardized interface ● Syslog cons ○ fire and forget ○ the syslog protocol ○ not enough parsing flexibility ○ syslog-ng was heavier then rsyslogd
- 14. Следете актуалните обяви за DevOps Партньори: Pros and Cons ● Loki/Node_exporter/filebeat/fluentd ○ very good parsing capabilities ○ filebeat was the easiest for me ○ reliable log delivery ○ different integrations ○ ready made grafana dashboards ● Loki/Node_exporter/filebeat/fluentd ○ very heavy on CPU ○ Loki did not have sysv init script :)
- 15. Следете актуалните обяви за DevOps Партньори: Interesting ● OAIEvals Collector - by Nikolay Stankov
- 16. Следете актуалните обяви за DevOps Партньори: DB integrations 1. Prometheus node-exporter 2. Fluentd 3. filebeat 4. syslog
- 17. Следете актуалните обяви за DevOps Партньори: Not out of the box ● Custom local collectors still have to go directly to your metrics DB ● Having a producer/subscriber greatly reduces the performance hit ● Fluent and fliebeat were the only one supporting kafka out of the box ○ https://github.com/hikhvar/mqtt2prometheus ○ https://github.com/toyokazu/fluent-plugin-mqtt-io
- 18. Thank you! СЛЕДВАЩО СЪБИТИЕ Лектор Дата Език Следете актуалните обяви за DevOps Партньори: Monitoring & Logging Marian Marinov 19.Mar.2024 Български Contacts: Marian Marinov Github profile Facebook profile
- 19. Следете актуалните обяви за DevOps Партньори: What do I have on the containers? ● NextCloud ● Home Assistant ● Mirrors ● VPNs ● NetBox ● Monitoring (Grafana, StatPing) ● Games (Minecraft, CS, PVPGN) ● IRC (server, bouncers, bots) ● Matrix, Mattermost ● Backups ● Streaming (FOSDEM streamer setup) ● DBs (PostgreSQL, MySQL, Redis, DragonFly, Timescale, InfluxDB, Mongo) ● Vitess, ProxySQL ● MPI (Gearman, MQTT, Kafka, RabbitMQ) ● Web stuff - Wiki, HAproxy, Nginx, Varnish ● OpenShift, OpenStack, K8s on VMs and physical ● A lot of other experiments
- 20. Следете актуалните обяви за DevOps Партньори: What storage do I use? ● Local + LVM ● DRBD+OCFS2 ● iSCSI ● cLVM + iSCSI ● GlusterFS ● OrangeFS ● I had in the past: ○ Ceph ○ NFS ○ cLVM + ATAoE ○ cLVM + NBD