Docker Monitoring Webinar
5 likes6,011 views
The document provides an overview of a Docker monitoring webinar, covering topics such as performance monitoring, anomaly detection, and centralized log management. It discusses deployment options for Docker agents and application monitoring within Docker containers, highlighting key metrics and monitoring strategies. Additionally, it includes technical examples and configurations for deploying monitoring agents and emphasizes the importance of monitoring for application performance and troubleshooting.
Docker Monitoring Webinar
- 1. Docker Monitoring Webinar 20% OFF 201509WNR20S 201509WNR20L sematext.com/spm sematext.com/logsene
- 3. Intro SPM: Performance monitoring, Anomaly Detection and Alerting Logsene: Centralized Log Management Search and Big Data Consulting Support for Solr and Elasticsearch
- 4. Agenda ● Docker Monitoring ○ Docker Monitoring Basics ○ Deployment Options for Docker Agents ● Application Monitoring on Docker ○ Monitoring Containerized Applications ○ Deployment Options for Application Monitoring Agents
- 5. Why we like Blue Whales ● Portable packaging ● Optimized resource usage ● Configurable resource limits ● Isolated environment for apps
- 6. What is Docker made of?
- 8. Docker Challenges ● New workflows for deployments ○ from developer laptop straight to the Cloud ● Orchestration of applications ● Logging ○ Webinars 29/30 September, @sematext ● Monitoring
- 9. Docker Monitoring Basics Let’s understand what we monitor!
- 10. Why Docker Monitoring You can’t fix problems you don’t see! ● Tuning ○ Resources / Container Limits ○ Application Performance ● Quality assurance ● Capacity planning ● Availability / SLA’s
- 11. What to monitor? Cluster Server 1 Container 1 Container 2 Container N Container 1 Elasticsearch Container 2 Node.js - web service Container N NGINX - reverse proxy Server N JVM Runtime Env JS Runtime Env
- 12. The Metrics Stack Changed! Server Metrics Runtime Metrics App Metrics Server Metrics Runtime Metrics App Metrics Container Metrics Runtime Metrics = JVM Metrics or Node.js Metrics, … Without Docker With Docker
- 13. Poll What is the result of ‘vmstat’ when started in a container? A) OS Metrics limited to the container B) OS Metrics from the docker host
- 14. Docker Stats Command > docker stats $(docker ps -q) CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O 19a9f4c9ecb7 0.00% 1.421 MB/3.944 GB 0.04% 0 B/0 B 7f043a32307c 0.00% 1.413 MB/3.944 GB 0.04% 0 B/0 B 889646bd003f 0.11% 4.235 MB/3.944 GB 0.11% 0 B/0 B 9a29f494bf7d 0.27% 156.8 MB/3.944 GB 3.98% 0 B/0 B Too simple!
- 15. Docker API for Metrics > echo -e "GET /containers/3236f3328bbd/stats HTTP/1.0rn" | nc -U /var/run/docker.sock HTTP/1.0 200 OK Date: Thu, 30 Jul 2015 08:37:31 GMT Content-Type: text/plain; charset=utf-8 {"read":"2015-07-30T04:37:31.13789208-04:00","network":{"rx_bytes":636450,"rx_packets":1944," rx_errors":0,"rx_dropped":0,"tx_bytes":495897,"tx_packets":2415,"tx_errors":0,"tx_dropped":0}," precpu_stats":{"cpu_usage":{"total_usage":0,"percpu_usage":null,"usage_in_kernelmode":0," usage_in_usermode":0},"system_cpu_usage":0,"throttling_data":{"periods":0,"throttled_periods": 0,"throttled_time":0}},"cpu_stats ... Too verbose! More than 70 Metrics are exposed
- 16. Watching a Metric > echo -e "GET /containers/3236f3328bbd/stats HTTP/1.0rn" | nc -U /var/run/docker.sock | logagent -y | grep rx_bytes rx_bytes: 667326 rx_bytes: 667354 rx_bytes: 667413 rx_bytes: 667538 github.com/sematext/logagent-js Which Metrics are most relevant?
- 18. Server Metrics ● CPU ● Memory Usage ● Disk Space ○ Docker Images consume a lot of disk space
- 19. Container Metrics ● Containers CPU + Throttled CPU time / CPU limits! ● Memory Limits + Memory Allocation Fail Counters ● Disk I/O + Wait Times ● Network I/O + Network Errors
- 20. Docker Key Metrics in SPM
- 21. Beyond Metrics: Events ● Docker Events ○ container auditing ○ create, destroy, die, export, kill, pause, restart, start, stop, unpause, oom
- 22. Beyond Metrics: Logs ● Logs ○ Docker collects console output of each container ○ API or Log Drivers for forwarding ● Correlation with Metrics ○ Faster troubleshooting blog.sematext.com/2015/08/11/docker-logging-webinar/ Docker Logging Webinar
- 24. Directly on each Docker Host ● The traditional way! ● Full access to the operating system ● Limitations in ’Container only’ systems - CoreOS - RancherOS - Many PaaS Solutions, e.g., DEIS
- 25. One Container on each Docker Host ● Yes. The Docker Way! ● Portable to any ‘Docker platform’ ● Requirements: - Access to the Host Metrics + Docker API - Extended privileges (e.g., for SELinux, GCE,...)
- 27. SPM for Docker Monitoring apps.sematext.com/spm-reports/registerApplication.do hub.docker.com/r/sematext/spm-agent-docker/ github.com/sematext/spm-agent-docker
- 28. Example 1: Run SPM for Docker docker run -d --name spm-agent -v /var/run/docker.sock:/var/run/docker.sock -e SPM_TOKEN=fe31fc3a-xxxx-47c6-b83c-be376bfxxx -e HOSTNAME=$HOSTNAME -e SPM_MONITORING_TAGS="project:dev-test,role:demo" -e LOGSENE_TOKEN=53a6c7e7-xxxx-4725-962e-ea47cebxxx --privileged sematext/spm-agent-docker
- 29. Example 2: CoreOS + DEIS ● CoreOS has no package manager ● systemd - init system ● fleet - as distributed init system ● etcd - for central configurations How to distribute a Monitoring Agent to all CoreOS servers in a cluster?
- 30. Example 2
- 31. Example 2: Distribute the Agent # Configure SPM app token etcdctl set /sematext.com/myapp/spm/token YOUR_SPM_TOKEN # Download the fleet unit file for SPM wget https://raw.githubusercontent.com/sematext/spm-agent-docker/master/coreos/spm-agent.service -O spm-agent.service # Start SPM Agent in the whole cluster fleetctl load spm-agent.service fleetctl start spm-agent.service
- 32. Example 3: RANCHER OS ● Minimalistic (~25 MB !) ● No package manager ● All system services are containers ● No cluster manager (by default) ● Use restart policy ‘always’ for services docker run --restart=always … sematext/spm-agent-docker
- 35. How are Applications monitored? ● Standalone Agents ○ Remote interface: JMX, MySQL, NGINX, Apache ○ No restart required, when agent gets updated ● In-Process Agents ○ Resource friendly ○ Node.js ○ JVM Apps (Elasticsearch, Solr, Kafka, ...) - Function Transaction/Call Tracing possible
- 36. Transaction Tracing + AppMap ● Only possible with in-process monitoring
- 37. Deployment Options for Application Monitoring … on Docker
- 38. Everything in Each Container
- 39. Standalone in Separate Container
- 40. In-Process in Separate Container
- 41. Setup for Elasticsearch Image spm-client: image: sematext/spm-client # Exports Volume /opt/spm container_name: spm-client environment: - SPM_CFG="${SPM_TOKEN} es javaagent jvmname:ES1" ELASTICSEARCH: image: elasticsearch ports: - "9200:9200" volumes_from: - spm-client environment: # inject in-process monitor from SPM client volume /opt/spm - ES_JAVA_OPTS="-Dcom.sun.management.jmxremote -javaagent:/opt/spm/spm-monitor/lib/spm-monitor-es.jar=${SPM_TOKEN}::ES1"
- 42. Setup for node.js Apps Dockerfile: FROM nodejs RUN git clone my/app RUN npm i RUN npm i spm-agent-nodejs CMD node -r spm-agent-nodejs app.js --- docker build -t myapp . docker run -e SPM_TOKEN=43A... myapp
- 43. Application + Docker Metrics
- 44. SPM Logsene METRICS + LOGS ⇒ BETTER TOGETHER
- 46. Docker Monitoring Webinar 20% OFF 201509WNR20S 201509WNR20L sematext.com/spm sematext.com/logsene
- 47. Thank you for your attention