Docker Usage Patterns - Meetup Docker Paris - November, 10th 2015
1 like1,512 views
The document presents an analysis of Docker usage patterns based on data from 7,000 organizations, revealing a fivefold increase in Docker adoption over the past year, with 6% of monitored hosts running Docker. It emphasizes the complexities of monitoring dynamic infrastructures and the need for service-level monitoring rather than a host-centric approach. Key recommendations include using tags and labels for better query formulation and understanding the underlying technology for effective metric collection.
Docker Usage Patterns - Meetup Docker Paris - November, 10th 2015
- 1. Docker Usage Patterns No Hype, Just Data Docker Meetup Paris Nov 10, 2015 Benjamin Fernandes Software Engineer — Datadog @LotharSee
- 2. About Me • MS, CS degree from Ecole Centrale Paris Former chairman at VIA Centrale Réseaux • Joined Datadog 3 years ago • Worked on Datadog’s integration with Docker and its ecosystem Benjamin Fernandes Software Engineer Datadog @LotharSee
- 3. Quick Overview of Datadog Datadog gathers performance data from all your application and infrastructure components. • Monitoring for modern applications 1. Dynamic Infrastructure 2. Containers (Docker, ECS, Mesos, k8s, and more…) 3. Microservices • Time series storage of metrics and events • Trending, alerting and anomaly detection ! • We’re hiring! (New York, Paris, Remote)
- 4. Monitor Everything Datadog gathers performance data from all your application components.
- 5. Why so much Docker?
- 7. Who is running Docker today? Docker Adoption
- 8. What about in production? Docker Adoption
- 9. Adopter: the average number of containers running during the month was at least 50% the number of distinct hosts run, or there were at least as many distinct containers as distinct hosts run during the month. Dabbler: used Docker during the month, but did not reach the “adopter” threshold. Abandoner: a currently active company that used Docker in the past, but hasn't used it at all in the last month. Study from 7000 organizations. Docker Adoption
- 10. Turns out you aren’t alone! Docker Adoption Source: http://dtdg.co/dckr-adopt
- 11. Fact 1: Docker Adoption Up 5x in 1 Year
- 12. Docker Adoption Growth We’ve see 5x increase of Docker adoption over the last year.
- 13. Fact 2: Docker now runs on 6% of hosts we monitor
- 15. Fact 3: Larger Companies Are the Early Adopters!
- 17. Fact 4: 2/3 of Companies That Try Docker Adopt It
- 19. Fact 5: Users 3x the Number of Containers They Use in 5 Months
- 21. Fact 6: Most Widely Used Images Are Registry, NGINX, and Redis
- 23. Fact 7: Docker Hosts Often Run Four Containers at a Time
- 24. Fact 8: VMs Live 4x Longer Than Containers
- 26. Bonus fact. Thanks The Onion.
- 28. Operational Complexity • Average containers per host: N (N=4, 10/2015)! • N-times as many “hosts” to manage! • Affects! • provisioning: prep’ing & building containers! • configuration: passing config to containers! • orchestration: deciding where/when containers run! • monitoring: making sure containers run properly
- 30. Operational Complexity: Scale 160 metrics per host 640 metrics per host Assuming 4 containers per host
- 31. Operational Complexity: Scale 100 instances 64,000 metrics Assuming 4 containers per host
- 33. So what does that mean for monitoring and management?
- 35. Monitoring Needs and Pains • Avoid Static config files tracking dynamic infrastructure.
- 36. Monitoring Needs and Pains • Avoid Static config files tracking dynamic infrastructure. • Avoid a host centric view. Focus on service level.
- 37. Monitoring Needs and Pains: Host Centric vs Service Centric
- 38. Monitoring Needs and Pains • Avoid Static config files tracking dynamic infrastructure. • Avoid a host centric view. Focus on service level. • Use tags, labels, etc on your hosts and metrics to form queries.
- 39. Monitoring Needs and Pains: Query Based Monitoring “Show me rate of HTTP 500 responses from nginx” “… in region us-east-1 across all availability zones” “… running my app version 2….” • Use tags, labels, etc on your hosts and metrics to form queries. • Pull in labels from your infrastructure whether EC2, Docker or your scheduler. • Ask questions that will ring true regardless of your scale that day.
- 40. Monitoring Needs and Pains • Avoid Static config files tracking dynamic infrastructure. • Avoid a host centric view. Focus on service level. • Use tags, labels, etc on your hosts and metrics to form queries. • Know your underlying tech. In this case Docker and how to pull metrics from it.
- 42. Collecting Docker Metrics: stats • Continuous live stream of basic CPU, memory, & network metrics.! • At least version 1.5.0 of Docker (released Feb 2015)! ! • docker stats --no-stream $(docker ps -q)
- 43. Collecting Docker Metrics: stats api • Stream of JSON, more detailed
- 44. Collecting Docker Metrics: Pseudo Files • If you want to do it manually! ! • CPU/Mem metrics! • Access via sysfs in /sys/fs/cgroup or /cgroup! • By default do not require root access! • Network metrics! • /proc/$PID/net/dev! • Disk IO metrics! • /proc/$PID/io
- 46. What to do with that?
- 47. Appendix A: Monitoring 101 Crash Course More at: http://goo.gl/t1Rgcg
- 48. Monitoring 101: Categorize Your Metrics More at: http://goo.gl/t1Rgcg
- 49. Monitoring 101: Focus on symptoms More at: http://goo.gl/t1Rgcg
- 50. Recurse until you find root cause. More at: http://goo.gl/t1Rgcg
- 51. Woof!