The Art of Container Monitoring
Download as PPTX, PDF0 likes156 views
The document discusses the importance of container monitoring in DevOps, highlighting the differences between blackbox and whitebox monitoring and advocating for the latter due to its detailed insights. It emphasizes the use of metrics, events, and logs to effectively monitor applications using tools like InfluxDB, Telegraf, and Icinga2, while explaining pull and push modes for data collection. Ultimately, it outlines the evolution of monitoring practices in the face of cloud and microservices architectures.
![Copyright 2016 Trend Micro Inc.6
Goal
Operating System
Application
Business Logic
Metrics
Events
Logs
Router
Destinations
Store
Visualize
Alert
Monitoring Framework
• Be metric, event, and log
• Allow us to easily visualize the
stat of our environment
• Provide contextual and useful
notification.
• Focus on whitebox monitoring
instead of blackbox monitoring
[image] https://www.artofmonitoring.com](https://image.slidesharecdn.com/theartofcontainermonitoring-170816220220/85/The-Art-of-Container-Monitoring-6-320.jpg)
![Copyright 2016 Trend Micro Inc.7
• The practice of system admin is changing quickly
– DevOps
– Infrastructure as Code
– Visualization
– Cloud Platform
• Commercial solution can’t keep up!
Why Open Source Software Monitoring?
[image] http://devops.com/2014/05/05/meet-infrastructure-code](https://image.slidesharecdn.com/theartofcontainermonitoring-170816220220/85/The-Art-of-Container-Monitoring-7-320.jpg)
![Copyright 2016 Trend Micro Inc.51
Last but Not least
• Data Flow
– Blackbox vs Whitebox
– Pull Mode vs Push Mode
• Data Content
– Opeating System, Application,
Business Logic
• Data Type
– Metric, Events, Logs
Operating System
Application
Business Logic
Metrics
Events
Logs
Router
Destinations
Store
Visualize
Alert
Monitoring Framwork
[image] https://www.artofmonitoring.com](https://image.slidesharecdn.com/theartofcontainermonitoring-170816220220/85/The-Art-of-Container-Monitoring-51-320.jpg)
The Art of Container Monitoring
- 1. The Art of Container Monitoring Derek Chen 2016.9.22
- 2. Copyright 2016 Trend Micro Inc.2 About me • DevOps Engineer at Trend Micro – Agile transformation – Micro service and cloud service – Docker integration – Monitoring system development • Automate all the things • Make everything smoother • Find me at derekhound@gmail.com
- 3. Copyright 2016 Trend Micro Inc.3 • We want to know when things go wrong • We want to know when things aren’t quite right • We want to know in advance of problems Why monitoring? Microscope? Magnifier? Telescope?
- 4. Copyright 2016 Trend Micro Inc.4 Blackbox vs Whitebox • Blackbox – Requires no participation of the monitored system – Observes external functionality, “what the user see” – End to end test • Whitebox – Collects data internally provided by the target system – Has more granular information about the system – Can provide warning of problems before they occur
- 5. Copyright 2016 Trend Micro Inc.5 Blackbox tests measure… • Can you ping the server • Can you fetch a page from the server • Does it have the correct contents Whitebox monitoring collects… • Network statistics (packets/bytes sent/received) • System load (cpu, memory, disk usage) • Application statistics (connection count, query per second)
- 6. Copyright 2016 Trend Micro Inc.6 Goal Operating System Application Business Logic Metrics Events Logs Router Destinations Store Visualize Alert Monitoring Framework • Be metric, event, and log • Allow us to easily visualize the stat of our environment • Provide contextual and useful notification. • Focus on whitebox monitoring instead of blackbox monitoring [image] https://www.artofmonitoring.com
- 7. Copyright 2016 Trend Micro Inc.7 • The practice of system admin is changing quickly – DevOps – Infrastructure as Code – Visualization – Cloud Platform • Commercial solution can’t keep up! Why Open Source Software Monitoring? [image] http://devops.com/2014/05/05/meet-infrastructure-code
- 8. Copyright 2016 Trend Micro Inc.8 Monolithic • If all-in-one gets the job done, then great • Good for smaller scale, non-tech-focused companies Modular • DevOps requires flexibility and innovation • Good for tech driven and ops-focused companies
- 9. Metric We’ll rely most heavily on metrics to help us understand what’s going on in our environment.
- 10. Copyright 2016 Trend Micro Inc.10 Metric • Provide a dynamic, real-time picture of the state of your infrastructure Collect Store Visualize Operating System Application Business Logic
- 11. Copyright 2016 Trend Micro Inc.11 Pull Mode A central collector periodically requests metrics from each monitored system Scrape Metrics Server #1 Service Server #2 Service Server #3 Service Monitoring Push Mode Metrics are periodically sent by each monitored system to a central collector Server #1 Agent Server #2 Agent Server #3 Agent Monitoring Push Metrics
- 12. Collectd The system statistics collection daemon
- 13. Copyright 2016 Trend Micro Inc.13 Database Host Machine OS Metric K8S Minion Collectd K8S Master Collectd Kubernetes Cluster Elasticsearch Monitor GrafanaInfluxDB Collectd Collectd K8S Minion Collectd
- 14. cAdvisor Collects, aggregates, processes, and exports information about running containers.
- 15. Copyright 2016 Trend Micro Inc.15 Database Container OS Metric K8S Minion cAdvisor Heapster K8S Master API Server cAdvisor Kubernetes Cluster Elasticsearch Monitor GrafanaInfluxDB K8S Minion cAdvisor
- 16. Copyright 2016 Trend Micro Inc.16 http://127.0.0.1:4194 http://127.0.0.1:4194/metrics
- 17. Telegraf Collects time series data from a variety of sources
- 18. Copyright 2016 Trend Micro Inc.18 Database Application Metric – Pull Mode K8S Minion K8S Master API Server Kubernetes Cluster Elasticsearch Monitor GrafanaInfluxDB NginxPostgres ES Telegraf K8S Minion Redis
- 19. Copyright 2016 Trend Micro Inc.19 Database Application Metric – Push Mode K8S Minion Telegraf K8S Master API Server Kubernetes Cluster Elasticsearch Monitor GrafanaInfluxDB Nginx Telegraf Postgres Telegraf ES K8S Minion Telegraf Redis Pod Pod
- 20. Copyright 2016 Trend Micro Inc.20 Telegraf • An agent written in Go • Easy to contribute or develop your plugins • Supported Input Plugins – System: cpu, mem, net, disk, processes and etc. – Application: docker, nginx, postgresql, redis, and etc. – Third party api: aws cloudwatch • Supported output plugins – influxdb, graphite, cloudwatch, datadog and etc.
- 21. Copyright 2016 Trend Micro Inc.21 Pull Mode • Collector discoveries services periodically • Collector needs to talk to target services – Overlay network. Ex: Flannel, Weave, etc. – Forward proxy Scrape Metrics Server #1 Service Server #2 Service Server #3 Service Monitoring
- 22. Copyright 2016 Trend Micro Inc.22 Push Mode • Agent sends metrics as soon as it starts up • Suitable for short-lived containers or dynamic environments Server #1 Agent Server #2 Agent Server #3 Agent Monitoring Push Metrics
- 23. InfluxDB Time series database stores all the metrics
- 24. Copyright 2016 Trend Micro Inc.24 • Similar scope to Graphite • Written in Go • No external dependency • Ready for billions row • Several client libraries • SQL style queries
- 25. Copyright 2016 Trend Micro Inc.25 InfluxDB Schema • Measurements (e.g. cpu, mem, disk, net, nginx) • Timestamp (nano second) • Tags (e.g. app=atom-apigw namespace=alpha) • Fields (e.g. accepts=30925, active=1, handled=30925)
- 26. Copyright 2016 Trend Micro Inc.26 InfluxDB Schema • Measurements (e.g. cpu, mem, disk, net, nginx) • Timestamp (nano second) • Tags (e.g. app=atom-apigw namespace=alpha) • Fields (e.g. accepts=30925, active=1, handled=30925)
- 27. Copyright 2016 Trend Micro Inc.27 InfluxDB Schema • Measurements (e.g. cpu, mem, disk, net, nginx) • Timestamp (nano second) • Tags (e.g. app=atom-apigw namespace=alpha) • Fields (e.g. accepts=30925, active=1, handled=30925)
- 28. Copyright 2016 Trend Micro Inc.28 InfluxDB Schema • Measurements (e.g. cpu, mem, disk, net, nginx) • Timestamp (nano second) • Tags (e.g. app=atom-apigw namespace=alpha) • Fields (e.g. accepts=30925, active=1, handled=30925)
- 29. Copyright 2016 Trend Micro Inc.29 InfluxDB Schema • Measurements (e.g. cpu, mem, disk, net, nginx) • Timestamp (nano second) • Tags (e.g. app=atom-apigw namespace=alpha) • Fields (e.g. accepts=30925, active=1, handled=30925)
- 30. Grafana Querying and visualizing time series and metrics
- 31. Finally! What we can to see! Less talk more demo…
- 32. Copyright 2016 Trend Micro Inc.32
- 35. Copyright 2016 Trend Micro Inc.35
- 37. Copyright 2016 Trend Micro Inc.37
- 38. Event We’ll generally use events to let us know about changes and occurrences in our environment.
- 39. Icinga2 A monitoring system which checks the availability of your resources, notifies users of outages
- 40. Copyright 2016 Trend Micro Inc.40 • Originally forked from Nagios in 2009 • Independent version Icinga2 since 2014 • Monitors everything • Gathering status • Collect performance data /use/lib/nagios/plugins/plugin_name Any program which returns • 0 – OK • 1 – WARNING • 2 – CRITICAL Message to STDOUT
- 41. Copyright 2016 Trend Micro Inc.41 Database Event Monitoring K8S Minion K8S Master API Server Kubernetes Cluster Elasticsearch Monitor GrafanaInfluxDBIcinga2 NginxPostgres ES K8S Minion Redis
- 43. Copyright 2016 Trend Micro Inc.43
- 44. Copyright 2016 Trend Micro Inc.44 Host Pod
- 45. Copyright 2016 Trend Micro Inc.45 Server Monitoring • About changes and occurrences in our environment – Is cpu load too high? – Is memory not enough? – Is docker engine still alive? External Monitoring • End to end test from user’s perspective • Can you connect ssh port 22? • Can you browse web page? • Can you request RESTful API successfully?
- 46. Dashboard • Provides a clear view for current environment Alerting • Notifies using email, slack, pager and etc. • Notification escalation Others… • Distributed Monitoring • Reloads config without interrupt checks
- 47. Log Logs are a subset of events. They’re often most useful for fault diagnosis and investigation
- 48. Copyright 2016 Trend Micro Inc.48 • EFK stack is a mature logging solution EFK Shipping all of your log to where it should go The main part to store your data with high availability Visualize will power your data. To know more about its value.
- 49. Summary
- 50. Copyright 2016 Trend Micro Inc.50 Collect Store Visualize Monitoring Service Stack Alert InfluxDB Elasticserach Grafana Kibana Icinga2Collectd Telegraf Fluentd cAdvisor
- 51. Copyright 2016 Trend Micro Inc.51 Last but Not least • Data Flow – Blackbox vs Whitebox – Pull Mode vs Push Mode • Data Content – Opeating System, Application, Business Logic • Data Type – Metric, Events, Logs Operating System Application Business Logic Metrics Events Logs Router Destinations Store Visualize Alert Monitoring Framwork [image] https://www.artofmonitoring.com
Editor's Notes
- #2: 大家好，很開心今天可以在這裡跟大家分享 the art of container monitoring。開始之前，大家想看看有沒有遇這樣的困擾。 因為 aws 帶來的便利性，我們很習慣把機器搬到 aws 上面，但是機器發生問題的時候，打開 cloud watch，發現這些 metric 都不是我想看的，你有這種困擾嗎？我有。 所以我就把 custom metric 放到 cloud watch 上，卻發現 cloud watch 萬萬稅，新增 metric 要錢，設定 alarm 要錢，五分鐘監控一次改成一分鐘監控一次也要錢，你有這種困擾嗎？我有。 把 container 放到 kubernetes 上面跑的時候，卻發現不知道怎麼計算 real time total current connection，你有這種困擾嗎？我有。 如果你跟我一樣有這些困擾的話，希望今天的演講結束以後，人人都可以量身訂造一個屬於你自己的 monitoring system。
- #3: 首先我要介紹一下我自己，我叫做 Derek ，目前任職於趨勢科技擔任 DevOps 的工作，主要將 CI/CD 導入生產團隊，對於 microservice 和 cloud service 有著濃厚的興趣。最近致力於將 kubernetes 整合進生產環境，並建構 container monitoring system。我喜歡將所有的事情盡可能的自動化，讓每件事情變得更簡單更順利。有問題想要找我的話，email在這裡。
- #4: 在容器的世界中，依據系統的規模大小，可能有數百個到數千個容器在運行。我們到底是應該用顯微鏡觀察每一個 container 的運行狀態，還是用放大鏡觀察 micro service 的整體狀態，還是拿個望遠鏡，遠遠的觀望整個系統的狀態，取決於觀察的精細度是否能幫助我們容易找到問題和解決問題。
- #8: 因為我們所處的環境不斷的在改變。近幾年提昌的 DevOps，需要團隊做敏捷的開發，需要整合 CI / CD 的 tool。最近開始要有一句響亮的口號 ‘Infrastructure as Code’，希望能夠自動化完成所有的部署工作。我們需要把數據視覺化。我們開始大量的採用 cloud platform, 例如 aws, google cloud, azure。 環境一直在改變，但是 commercial solution 在大部分的時候沒有這麼彈性，難以符合我們不斷變動的需求。
- #9: 右上方的圖是 google nexus 手機，是一個 all in one solution 的代表，右下方的圖是 google開發的模組化手機，可以根據你的需求替換不同的模組。當你在選擇 open source software 時，你會選擇那一種呢，all in one solution 或是 modular solution 呢？ 假如這個 all in one solution 剛好符合你的需求，那你真得非常幸運，Promethues 就是這類型的代表。 但是，大部份的時候我們沒有那麼幸運，我們必須將數個 software 放在一起去建構我們自己的 monitoring system。
- #41: http://www.slideshare.net/icinga/why-favour-icinga-over-nagios-rootconf-2015?qid=13b9dfe7-0eee-45cc-b16a-fcdaa2b54b22&v=&b=&from_search=23