Elk for applications on k8s
3 likes1,118 views
The document discusses the ELK stack (Elasticsearch, Logstash, and Kibana) used for monitoring Kubernetes applications, highlighting its components and use cases such as log collection, system metric analysis, and visualization. It outlines the setup process including configurations and examples for collecting logs from various sources like nginx. Additionally, the document compares ELK with other monitoring solutions and lists its advantages and disadvantages.
![35.185.145.221 - [35.185.145.221] - - [26/Dec/2018:10:44:22
+0000] "GET
/modules/actions/api?timestamp=1545821062&hash=e63ffc17319baa
5524a7ad10dbb3c4be822c4e519d88efdae511116498d6c9e2 HTTP/1.1"
200 3 "-" "python-requests/2.18.4" 416 0.013
[default-app-server-80] 10.24.10.14:8080 3 0.014 200
67ce5290d19e39b2a6b614a617a1cd98](https://image.slidesharecdn.com/elkforapplicationsonk8s-190122151655/85/Elk-for-applications-on-k8s-27-320.jpg)
![https://raw.githubusercontent.com/chechiachang/elk-kubernetes/master/logstash/
pipelines-configmap.yaml
input {
beats {
# The lisening port of logstash
port => 5044
host => "0.0.0.0"
}
}
filter {
# Nginx-ingress controller example
if [kubernetes][container][name] == "nginx-ingress-controller" {
……
}
} # filter
output {
elasticsearch {
hosts => ["https://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}"]
user => "${ELASTICSEARCH_USERNAME}"
password => "${ELASTICSEARCH_PASSWORD}"
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
manage_template => false
}
}](https://image.slidesharecdn.com/elkforapplicationsonk8s-190122151655/85/Elk-for-applications-on-k8s-29-320.jpg)
![filter { # Nginx-ingress controller example
if [kubernetes][container][name] == "nginx-ingress-controller" {
# Parse message with grok
grok {
match => { "message" => “...” }
#remove_field => "message"}
mutate {
add_field => { "read_timestamp" => "%{@timestamp}" }}
date {
match => [ "[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
remove_field => "[nginx][access][time]"
}
kv {
source => "[nginx][access][url_params]"
field_split => "&"
}
useragent {
source => "[nginx][access][agent]"
target => "[nginx][access][user_agent]"
remove_field => "[nginx][access][agent]"
}
geoip {
source => "[nginx][access][remote_ip]"
target => "[nginx][access][geoip]"
#fields => ["country_name","city_name","real_region_name","latitude","longitude","ip","location"]
}](https://image.slidesharecdn.com/elkforapplicationsonk8s-190122151655/85/Elk-for-applications-on-k8s-30-320.jpg)
![35.185.145.221 - [35.185.145.221] - - [26/Dec/2018:10:44:22 +0000] "GET
/modules/actions/api?timestamp=1545821062&hash=e63ffc17319baa5524a7ad10dbb3c4b
e822c4e519d88efdae511116498d6c9e2 HTTP/1.1" 200 3 "-" "python-requests/2.18.4"
416 0.013 [default-app-server-80] 10.24.10.14:8080 3 0.014 200
67ce5290d19e39b2a6b614a617a1cd98
%{IPORHOST:[ nginx][access][remote_ip]} - [%{IPORHOST:[nginx ][access][remote_ip_list ]}] -
%{DATA:[nginx][access][user_name]} [%{HTTPDATE:[nginx ][access][time_local]}]
"%{WORD:[nginx][access][method]} %{DATA:[ nginx][access][url]}
HTTP/%{NUMBER:[ nginx][access][http_version ]}" %{NUMBER:[ nginx][access][response_code ]}
%{NUMBER:[nginx][access][body_sent][bytes]} "%{DATA:[ nginx][access][referrer]}"
"%{DATA:[nginx][access][user_agent_original ]}" %{NUMBER:[ nginx][access][request_length ]}
%{NUMBER:[nginx][access][request_time ]} [%{DATA:[nginx ][access][proxy_upstream_name ]}]
%{DATA:[nginx][access][upstream_addr ]} %{NUMBER:[ nginx][access][upstream_response_length ]}
%{NUMBER:[nginx][access][upstream_response_time ]} %{NUMBER:[ nginx][access][upstream_status ]}
%{DATA:[nginx][access][req_id]}](https://image.slidesharecdn.com/elkforapplicationsonk8s-190122151655/85/Elk-for-applications-on-k8s-31-320.jpg)
Elk for applications on k8s
- 1. David Chang DevOps@mithril ELK 監測 Kubernetes 應用
- 3. What is ELK Stack?
- 4. 3 open source projects ● Kibana 分析和可視化平台 ● Elasticsearch 分布式全文搜索引擎 ● Logstash 數據收集和日誌解析引擎 ● Beats lightweight data shippers
- 6. Why ELK 為何要收 Log / Metrics？放DB 不好嗎？
- 7. Why Collect log ● Why log exists? ● Debug with your log ● Debug ahead of your error ML?
- 8. System logs ● System logs SSH access, events, stress… ● Access log (Nginx)
- 9. Logs Diary of your application
- 10. Metrics ● System Metrics CPU / Memory / storage / IO ● App Metrics resource usage / performance
- 11. Metrics How well your apps are
- 12. Kubernetes Things always get a little complicated on Kubernetes ;)
- 13. Kubernetes logging ● https://kubernetes.io/docs/co ncepts/cluster-administration /logging ● Log of container runtime ● Cluster-level logging ● Make sure your k8s store log
- 14. One Pod, kubectl logs :) ● App generates log ● Container runtime strout ● Log file visible on Node ● Apply a collector ● Mount log
- 15. 1000 pods, have fun :P ● Hard to locate error pod / log ● Terminated pod don’t log ● microservices, components ● Third party services, nginx, kafka, redis… ● Multi-clusters
- 16. Alternatives(?) to ELK ● Datadog ● Stackdriver for GCP ● Prometheus / Grafana ● Fluentd ● 弄清楚你的需求
- 18. ● Have a Elasticsearch + Kibana ● Config / Deploy beats to your apps / nodes ● Create index pattern on kibana ● Add visualization Quick Start ELK
- 19. Have a Elasticsearch + Kibana ● Elastic Cloud (Saas) No deployment, configuration, or cluster maitainance ● Self-hosted open source ELK no authentication
- 21. Elastic Cloud Elasticsearch cluster + kibana node
- 22. Get id and auth
- 23. Deploy beats
- 25. Beats config ● Elasticsearch authentication ● Where’s log ● Use module: system, nginx, docker ● beat-side data digestion
- 26. Logstash pipeline (for nginx) ● How to parse nginx log ● pattern match ● Reduce raw data ● Normalize log to basic type ● Minimize index size ● Tidying up your log
- 27. 35.185.145.221 - [35.185.145.221] - - [26/Dec/2018:10:44:22 +0000] "GET /modules/actions/api?timestamp=1545821062&hash=e63ffc17319baa 5524a7ad10dbb3c4be822c4e519d88efdae511116498d6c9e2 HTTP/1.1" 200 3 "-" "python-requests/2.18.4" 416 0.013 [default-app-server-80] 10.24.10.14:8080 3 0.014 200 67ce5290d19e39b2a6b614a617a1cd98
- 28. beat logstash Elasticsearch Kinana Chart Data Source Info ELKB for complex data Logstash pipeline: inputs → filters → outputs
- 29. https://raw.githubusercontent.com/chechiachang/elk-kubernetes/master/logstash/ pipelines-configmap.yaml input { beats { # The lisening port of logstash port => 5044 host => "0.0.0.0" } } filter { # Nginx-ingress controller example if [kubernetes][container][name] == "nginx-ingress-controller" { …… } } # filter output { elasticsearch { hosts => ["https://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}"] user => "${ELASTICSEARCH_USERNAME}" password => "${ELASTICSEARCH_PASSWORD}" index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" manage_template => false } }
- 30. filter { # Nginx-ingress controller example if [kubernetes][container][name] == "nginx-ingress-controller" { # Parse message with grok grok { match => { "message" => “...” } #remove_field => "message"} mutate { add_field => { "read_timestamp" => "%{@timestamp}" }} date { match => [ "[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ] remove_field => "[nginx][access][time]" } kv { source => "[nginx][access][url_params]" field_split => "&" } useragent { source => "[nginx][access][agent]" target => "[nginx][access][user_agent]" remove_field => "[nginx][access][agent]" } geoip { source => "[nginx][access][remote_ip]" target => "[nginx][access][geoip]" #fields => ["country_name","city_name","real_region_name","latitude","longitude","ip","location"] }
- 31. 35.185.145.221 - [35.185.145.221] - - [26/Dec/2018:10:44:22 +0000] "GET /modules/actions/api?timestamp=1545821062&hash=e63ffc17319baa5524a7ad10dbb3c4b e822c4e519d88efdae511116498d6c9e2 HTTP/1.1" 200 3 "-" "python-requests/2.18.4" 416 0.013 [default-app-server-80] 10.24.10.14:8080 3 0.014 200 67ce5290d19e39b2a6b614a617a1cd98 %{IPORHOST:[ nginx][access][remote_ip]} - [%{IPORHOST:[nginx ][access][remote_ip_list ]}] - %{DATA:[nginx][access][user_name]} [%{HTTPDATE:[nginx ][access][time_local]}] "%{WORD:[nginx][access][method]} %{DATA:[ nginx][access][url]} HTTP/%{NUMBER:[ nginx][access][http_version ]}" %{NUMBER:[ nginx][access][response_code ]} %{NUMBER:[nginx][access][body_sent][bytes]} "%{DATA:[ nginx][access][referrer]}" "%{DATA:[nginx][access][user_agent_original ]}" %{NUMBER:[ nginx][access][request_length ]} %{NUMBER:[nginx][access][request_time ]} [%{DATA:[nginx ][access][proxy_upstream_name ]}] %{DATA:[nginx][access][upstream_addr ]} %{NUMBER:[ nginx][access][upstream_response_length ]} %{NUMBER:[nginx][access][upstream_response_time ]} %{NUMBER:[ nginx][access][upstream_status ]} %{DATA:[nginx][access][req_id]}
- 33. Real Use Cases
- 34. System Metrics
- 37. Log 放 DB 不好嗎? ● 資料讀取的方便性 ● Data Define Language ● 分析工具 ● Elastic Data Storage ● full-text search ● visualization
- 38. 導入ELK心路歷程 ● Datadog, Stackdriver,... ● 訂立需求 ● 收集所有資料 ● 釐清需求 <-> 整理資料 ● 穩定需求與pipeline ● 產生視覺化報表 ● 提供資料 / 告警給團隊
- 39. ELK pros & cons ● lightweight collector ● powerful data pipeline ● elastic data write ● complex search and query ● Analysis tools & Visulization ● Storage consuming ● Not for permanant
- 40. Q&A