Prometheus and Docker (Docker Galway, November 2015)
8 likes10,678 views
Brian Brazil is an engineer passionate about reliable systems who has worked at Google SRE and Boxever. He discusses Prometheus, an open source monitoring system he helped create. Prometheus offers inclusive monitoring of services, is manageable and reliable, integrates easily with other tools, and provides powerful querying and dashboards. It is efficient, scalable, and helps provide visibility into systems through its data model and labeling.
![Prometheus configuration to use Consul
scrape_configs:
- job_name: 'consul'
consul_sd_configs:
- server: 127.0.0.1:8500
relabel_configs:
- source_labels: [__meta_consul_service]
regex: (.*) # Default next release
replacement: $1 # Default next release
target_label: job](https://image.slidesharecdn.com/prometheusanddocker-151125143942-lva1-app6891/85/Prometheus-and-Docker-Docker-Galway-November-2015-20-320.jpg)
![Example: Top 5 Docker images by CPU
topk(5,
sum by (image)(
rate(container_cpu_usage_seconds_total{
id=~"/system.slice/docker.*"}[5m]
)
)
)](https://image.slidesharecdn.com/prometheusanddocker-151125143942-lva1-app6891/85/Prometheus-and-Docker-Docker-Galway-November-2015-28-320.jpg)
Prometheus and Docker (Docker Galway, November 2015)
- 1. Prometheus and Docker Monitoring and Management Brian Brazil Founder
- 2. Who am I? Engineer passionate about running software reliably in production. β TCD CS Degree β Google SRE for 7 years, working on high-scale reliable systems such as Adwords, Adsense, Ad Exchange, Billing, Database β Boxever TL Systems&Infrastructure, applied processes and technology to let allow company to scale and reduce operational load β Contributor to many open source projects, including Prometheus, Ansible, Python, Aurora and Zookeeper. β Founder of Robust Perception, making scalability and efficiency available to everyone
- 3. Why monitor? β Know when things go wrong β To call in a human to prevent a business-level issue, or prevent an issue in advance β Be able to debug and gain insight β Trending to see changes over time, and drive technical/business decisions β To feed into other systems/processes (e.g. QA, security, automation)
- 4. Common Monitoring Challenges Themes common among companies Iβve talk to: β Monitoring tools are limited, both technically and conceptually β Tools donβt scale well and are unwieldy to manage β Operational practices donβt align with the business For example: Your customers care about increased latency and itβs in your SLAs. You can only alert on individual machine CPU usage. Result: Engineers continuously woken up for non-issues, get fatigued
- 5. Fundamental Challenge is Limited Visibility
- 6. Prometheus Inspired by Googleβs Borgmon monitoring system. Started in 2012 by ex-Googlers working in Soundcloud as an open source project, mainly written in Go. Publically launched in early 2015, and continues to be independent of any one company. Over 100 companies have started relying on it since then.
- 7. What does Prometheus offer? β Inclusive Monitoring β Manageable and Reliable β Easy to integrate with β Dashboards β Powerful data model β Powerful query language β Efficient β Scalable
- 10. Monitor as a Service, not as Machines
- 11. Inclusive Monitoring Donβt monitor just at the edges: β Instrument client libraries β Instrument server libraries (e.g. HTTP/RPC) β Instrument business logic Library authors get information about usage. Application developers get monitoring of common components for free. Dashboards and alerting can be provided out of the box, customised for your organisation!
- 12. Python Instrumentation Example pip install prometheus_client from prometheus_client import Summary, start_http_server REQUEST_DURATION = Summary('request_duration_seconds', 'Request duration in seconds') @REQUEST_DURATION.time() def my_handler(request): pass # Your code here start_http_server(8000)
- 13. Manageable and Reliable Core Prometheus server is a single binary. Doesnβt depend on Zookeeper, Consul, Cassandra, Hadoop or the Internet. Only requires local disk (SSD recommended). No potential for cascading failure. Pull based, so easy to on run a workstation for testing and rogue servers canβt push bad metrics. Advanced service discovery finds what to monitor.
- 14. Running Prometheus with Docker Docker images at https://hub.docker.com/r/prom/ To run Prometheus: docker run -p 9090:9090 prom/prometheus:latest This monitors itself. http://localhost:9090/consoles/prometheus.html has a console.
- 15. Easy to integrate with Many existing integrations: Java, JMX, Python, Go, Ruby, .Net, Machine, Cloudwatch, EC2, MySQL, PostgreSQL, Haskell, Bash, Node.js, SNMP, Consul, HAProxy, Mesos, Bind, CouchDB, Django, Mtail, Heka, Memcached, RabbitMQ, Redis, RethinkDB, Rsyslog, Meteor.js, Minecraft... Graphite, Statsd, Collectd, Scollector, Munin, Nagios integrations aid transition. Itβs so easy, most of the above were written without the core team even knowing about them!
- 16. Cadvisor natively supports Prometheus Run Cadvisor: docker run -v /var/run/:/var/run -v /sys:/sys -p 8080:8080 google/cadvisor http://localhost:8080/metrics will have metrics for all your containers
- 17. Service Discovery Itβs vital to know where each of your instances is meant to be running. Systems such as Docker Swarm spread applications across machines, so we need some form of service discovery to find them. Both Swarm and Prometheus support Consul, so letβs try that.
- 18. Setting up Swarm on localhost (1) MY_IP=1.2.3.4 # - Run consul docker run -d --name=consul --net=host gliderlabs/consul-server -bootstrap # - Pass the docker daemon -H tcp://0.0.0.0:2375 so itβll do TCP # - Register services in Consul docker run -d --net=host -v /var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator consul://localhost:8500 # - Join the swarm docker run -d swarm join --addr=$MY_IP:2375 consul://$MY_IP:8500/swarm
- 19. Setting up Swarm on localhost (2) # - Run some machine monitoring docker run -d -v /var/run/:/var/run -v /sys:/sys -p 8080:8080 google/cadvisor docker run -d -v /:/rootfs:ro -p 9100:9100 --net=host -e SERVICE_NAME=node prom/node-exporter # - Start the Swarm Manager docker run -d -p 2376:2375 swarm manage consul://$MY_IP:8500/swarm # - Check itβs working docker -H tcp://127.0.0.1:2376 info
- 20. Prometheus configuration to use Consul scrape_configs: - job_name: 'consul' consul_sd_configs: - server: 127.0.0.1:8500 relabel_configs: - source_labels: [__meta_consul_service] regex: (.*) # Default next release replacement: $1 # Default next release target_label: job
- 21. Run Prometheus docker run -d --net=host -p 9090:9090 -e SERVICE_NAME=prometheus robustperception/prometheus_local_consul Visit http://localhost:9090/status to see it running
- 23. Dashboards
- 24. Many Dashboarding Options β Grafana β Latest release has full Prometheus support β Promdash β Prometheus-specific Ruby-on-Rails dashboarding solution β Direction is moving towards Grafana instead β Console templates β Templating language inside Prometheus β Good for having your dashboards checked in, and for fully-custom dashboards β Expression Browser β Included in Prometheus, good for ad-hoc debugging β Roll your own β JSON API
- 25. Powerful Data Model All metrics have arbitrary multi-dimensional labels. No need to force your model into dotted strings. Can aggregate, cut, and slice along them. Supports any double value, labels support full unicode.
- 26. Example: Labels from Node Exporter
- 27. Powerful Query Language Can multiply, add, aggregate, join, predict, take quantiles across many metrics in the same query. Can evaluate right now, and graph back in time. Answer questions like: β Whatβs the 95th percentile latency in the European datacenter? β How full will the disks be in 4 hours? β Which services are the top 5 users of CPU? Can alert based on any query.
- 28. Example: Top 5 Docker images by CPU topk(5, sum by (image)( rate(container_cpu_usage_seconds_total{ id=~"/system.slice/docker.*"}[5m] ) ) )
- 29. Efficient Instrumenting everything means a lot of data. Prometheus is best in class for lossless storage efficiency, 3.5 bytes per datapoint. A single server can handle: β millions of metrics β hundreds of thousands of datapoints per second
- 30. Scalable Prometheus is easy to run, can give one to each team in each datacenter. Federation allows pulling key metrics from other Prometheus servers. When one job is too big for a single Prometheus server, can use sharding+federation to scale out. Needed with thousands of machines.
- 31. Final Thought: Instrument Once, Use Everywhere When an application is instrumented, tends to be limited in which monitoring systems it supports. Youβre left with a choice: Have N monitoring systems, or run extra services to act as shims to translate. Prometheus clients arenβt just limited to outputting metrics in the Prometheus format, can also output to Graphite and other monitoring systems. Prometheus clients can also take in metrics from other monitoring systems. Prometheus clients can be your instrumentation and metrics interchange, even when not using Prometheus itself!
- 32. What do we do? Robust Perception provides consulting and training to give you confidence in your production service's ability to run efficiently and scale with your business. We can help you: β Decide if Prometheus is for you β Manage your transition to Prometheus and resolve issues that arise β With capacity planning, debugging, infrastructure etc. We are proud to be among the core contributors to the Prometheus project.
- 33. Resources Official Project Website: prometheus.io Official Mailing List: prometheus-developers@googlegroups.com Demo: demo.robustperception.io Dockerfiles: https://github.com/RobustPerception/docker_examples Robust Perception Website: www.robustperception.io Queries: prometheus@robustperception.io