How to collect and utilize logs at Kubernetes with Elastic Stack
- Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. It groups containers that share resources into pods and allows containers in a pod to find each other and communicate using localhost. Pods run on nodes which are physical or virtual machines. - There are different approaches to logging in Kubernetes including sending logs from pods to a log backend directly or indirectly via nodes. Common backends include Elasticsearch, Splunk, and Kibana. Logs can be searched and alerts generated based on their contents. - Application performance monitoring (APM) tools integrate with applications like Rails to capture metrics on CPU, memory, transactions and send structured log data to backends for creating graphs and dashboards without
How to collect and utilize logs at Kubernetes with Elastic Stack
- 1. Nov 9, 2019 Keiichiro Seto EC Incubation Department Dept. Rakuten, Inc.
- 3. 3
- 5. 5
- 6. 6 pod • Same Volume • Same IP Address • Some containers Node • Real/Virtual Machine • Pods run on Node Cluster • Composed of Nodes manifest • File which configure kubernetes resource like Pod.
- 8. 8
- 9. 9 You
- 11. 11 Pod → Node → Backend Pod → Backend Container → Backend
- 13. 13 But…
- 14. 14 • • https://kubernetes.io/docs/concepts/cluster-administration/logging/ • • • • •
- 15. 15 Log Backend
- 16. 16 Pod → Node → Backend Pod → Backend Container → Backend
- 17. 17 • • •
- 18. 18 • • • • • •
- 19. 19 Pod → Node → Backend Pod → Backend Container → Backend
- 20. 20 • •
- 21. 21 Pod → Node → Backend Pod → Backend Container → Backend
- 22. 22 Each Node Sidecar Application Pros kubectl logs Less resources Flexibility Redundant Flexibility Redundant Cons Dependency More resources Enlargement of Image
- 23. 23 elastic.co
- 24. 24 ※
- 25. 25
- 26. 26
- 27. 27
- 28. 28 https://github.com/nginxinc/docker-nginx/blob/master/stable/alpine/Dockerfile
- 29. 29
- 31. 31
- 32. 32
- 33. 33 • • Searching Alerting Monitoring CPU Memory Transaction Exception
- 34. 34 APM works on Rails by gem. It sends structured data. It creates graphs like these automatically.
- 35. 35
- 36. 36 You can find logs in the way which goes well with kubernetes specific Node. some application. some version managed by label etc.
- 37. 37 kubernetes.pod.name: myapp-* AND nginx.status: >= 500
- 39. 39 Success Rate of each API
- 40. 40 Duration
- 41. 41 • • • •
- 42. 42
- 43. 43