Kubernetes Logging
- 1. Kubernetes Logging Implement and test self-managed logs collection and visualization solution for your k8s cluster
- 2. Real world requirements ● Quick and easy deployment and testing (do it like a DevOps) ● Reliable enough ● Maximum performance by default ● Simple to understand and debug (KISS) ● Reuse existing solutions as much as possible
- 3. Stack decisions and tools ● Playground: Minikube/Microk8s/Kubeadm/Kops ● FluentD/Fluent Bit/Logstash/Telegraf ● EL(F)K/TICK (InfluxDB) stacks, Graphana? ● Using k8s operator, reliability concerns ● Helm/GitKube/Kapitan ● Ingress controller (Nginx/ALB)
- 4. Kubernetes on localhost Minikube ● Needs a VM ● Needs special treatment ● Good for developing services, not operations testing Microk8s ● Super simple ● Real cluster ● Docker (in Docker) ● Too much magic ● We need Cloud for UI Load balancing and ingress
- 5. Kubernetes in the Cloud Kubeadm ● Not really for cloud ● Needs manual job ● No mature enough automations available Kops ● Easy, scalable ● AWS native ● Cluster-as-a-Code ● Addons ● Generates Terraform! ● Actively developed
- 6. Logs collector and processor ● Logstash: heavy, do we need all its features? ● Telegraf (InfluxDB): looks cool, where is k8s support? ● FluentD: good stuff, recommended upstream, Kops plugin ● Fluent Bit: lightweight and fast, native support for Kubernetes, minimal configuration
- 7. EFK stack for logs visualisation ● ElasticSearch: deploy real cluster, rotate indexes, resize, backup, kubernetes operator is the only option ● Fluent Bit: read all container logs, enrich with kubernetes metadata and system metrics, does buffering and retries, deployed via DaemonSet ● Kibana: discover the index in ES, configure timestamp field, watch and query. TODO: create some pretty dashboards
- 8. Turn-key deployment ● GitKube looks nice… ● Kapitan is more Pythonic ● Helm appears to be widely adopted, many examples available ● Install all stack components from single chart ● Helm is great about vendorizing dependencies ● GitHub repo could become Helm repo! ● TODO: enable ingress rule
- 10. Concerns and future improvements ● Authentication, sensitive data, ES hardening, Cerebro, capasity, proper TLS setup ● “IFG” stack: InfluxDB, FluentD, Grafana