OSMC 2016 | ZMON: Zalando's OS approach to monitoring in the cloud and DCs by Jan Mussler

TITLE IN CAPITAL
LETTERS
SUBTITLE IN CAPITAL LETTERS
JAN MUSSLER
jan.mussler@zalando.de
Twitter: @JanMussler
zmon.io
#NETWAYS #OSMC 30-11-2016
ZMON
Open Source Monitoring in the Cloud

15 countries
19+ million active customers
160+ million visits per month
200k+ articles
3.0+ billion € revenue
~ 1.600 employees in tech
Europe's Leading Online Fashion Platform
Visit us: tech.zalando.com

Zalando’s Technology History

➊ One AWS account per Team
➋ Deployment with Docker
➌ Managed SSH Access
➍ REST/OAuth 2.0 mandatory
➎ Traceability of changes
IN A NUTSHELL
STUPS

Internet
*.abc.example.org *.xyz.example.org
Team ABC Team XYZ
ISOLATED AWS ACCOUNTS
EC2EC2
ELBELB
EC2

Host Host
Service 4 Service 4
Host
Service 3 Service 3
Service 1 Service 1Monitoring
Team?
Service 2 Service 2
Monitoring the old way?
Team
Team
Team
Team

Build with teams and services in mind ...
Host Host Host
Service 4 Service 4Service 4
Host
Team 3
Service 3 Service 3Service 3Team 2
Service 1 Service 1Service 1Team 1
Service 2 Service 2

Flexible and extendable: Checks & Alerts in Python
Integrate: REST APIs, OAUTH2, Auto Discovery
Configurable via UI / API: no restarts required!
Great for teams: autonomy and responsibility
Fast/Scaling metrics: Redis, KairosDB + Grafana 3
ZMON - Highlights ;-)

OSMC 2016 | ZMON: Zalando's OS approach to monitoring in the cloud and DCs by Jan Mussler

Full authentication for all endpoints
OAUTH2 login flow (e.g. Github login)
“TV Tokens” for “read-only” dashboard login
Grafana 3 bundled and API implemented
Proxy for KairosDB (timeseries db)
ZMON Controller - User Interface and REST API

Display historic data using Grafana 3

Various options for notifications ...
E-Mail
Twilio (phone call)

● hosts, databases, applications, instances ...
● generic key value object
● 20000+ entities in our deployment
Entities
{
"id": "node01:8080",
"type": "instance",
"host": "node01",
"ports": {"8080":8080,"8181":8181},
"application_id": "zmon",
"application_version": "0.1.0",
"dc":"dc1"
}
Entity "node01:8080"

Entity Service (part of controller)
id: localhost:5432
type: postgres
host: localhost
port: 5432
shards:
local_zmon_db: "localhost:5432/local_zmon_db"
local-postgres.yaml
Integrated easy-to-use entity store with REST API
Build your own discovery agent (K8S, …)
>zmon entities push local-postgres.yaml

● select subset of entities
● executes Python expression
○ powerful using eval with custom context
○ Builtins: HTTP, PostgreSQL, MySQL, CloudWatch,
Redis, SNMP/NRPE, tcp,Scalyr, ElasticSearch, …
○ Data filtering/formating/pivoting
● returns "value" object -> dicts everywhere
Checks

REST API to update or use web front end
zmon check-definitions update select-1-check.yaml
Managing checks
name: "Select 1"
owning_team: "Team ZMON"
command: |
sql().execute("select 1 as a").results()
entities:
- type: postgresql
interval: 15
description: "Test connection to PostgreSQL"
select-1-check.yaml

Trial Run - Quick feedback and easier development

● Attached to a single check, inspect check result
● Defines team and responsible team
● Allows inheritance from other alert
● Evaluates Python expression yielding True/False
● No "WARNING" state, no "UNKNOWN" state
● Priorities(color) and tags
Alerts

Downtimes
● Set or schedule downtimes using the UI
● Use API to automate downtimes, e.g. in deployment tool

Reuse existing checks for core infrastructure
Anyone can add alerts to checks
Monitor application boundaries/dependencies
Make use of inheritance to customize
Sharing and reuse of alerts and checks

Tokeninfo (GO)Tokeninfo (GO)
Provider (Java)
Provider (Java)
Tokeninfo (GO)Tokeninfo (GO)
C* Nodes
C* Nodes
C* Nodes
C* Nodes
Plan B Deployment - Multi Region Setup (JWT issue/verification)
C* NodesProvider (Java)ELB
Tokeninfo (Go)ELB
C* NodesProvider (Java)ELB
Tokeninfo (Go)ELB

Will create “entities” to describe deployment
ELBs, ASGs, Application, instances,...
Crawls AWS API every 60 sec to update
ZMON AWS Agent - Auto Discovery

➜ ~ zmon entities get "planb-tokeninfo-cd44-oFM6x[aws:999:eu-west-1]"
id: planb-tokeninfo-cd44-oFM6x[aws:999:eu-west-1]
type: instance
application_id: planb-tokeninfo
host: 172.31.169.6
infrastructure_account: aws:999
instance_type: c4.xlarge
ip: 172.31.169.6
ports: { '9020': 9020, '9021': 9021 }
region: eu-west-1
source: registry.opensource.zalan.do/stups/planb-tokeninfo:cd44
stack_name: planb-tokeninfo-eu-west-1
stack_version: cd44
Example Instance Entity

➜ ~ zmon entities get " elb-data-service-cd79c9[aws:...:eu-central-1] "
id: elb-data-service-cd79c9[aws:...:eu-central-1]
type: elb
name: data-service-cd79c9
active_members: 5
cloudwatch_name: app/data-service-cd79c9/18b164bfa427486d
dns_name: data-service-cd79c9-961635181.eu-central-1.elb.amazonaws.com
dns_traffic: 'true'
dns_weight: 200
elb_type: application
members: 5
region: eu-central-1
scheme: internet-facing
Example Instance Entity

Instance Metrics
● Memory usage
● Disk space usage
● CPU usage
● Application logs
● Application metrics
Monitoring Plan-B EC2 instances on AWS
Scalyr Agent
Log shipping
Prometheus
Node Agent
:9100/metrics
Taupage AMI (Ubuntu base)
Application Container
Go / Spring Boot / Cassandra
Docker run time
:8080 -> app
:7979 -> metrics

Check result - Grafana 3 link
AWS UI deep link

Monitor your deployments … data tagged with version

Annotated Metric Data in Grafana

HTTP requests reading JSON application metrics
Read JMX data via Jolokia/HTTP for Cassandra
Read Prometheus Node data for EC2 metrics
CloudWatch() queries for ELB metrics
Scalyr API queries for application logs
Check commands used so far

Workers
(Python)
Workers
(Python)
ZMON Core + UI + KairosDB
Scheduler
(jvm)
Redis
Worker
(Python)
KairosDB
(Java)
Controller
(Java)
PostgreSQL
Queue/State
CLI
(Python)
Check/Alert definition
Entity data
Cassandra
Frontend
(AngularJS)
Metric Cache

● Scheduler supports queue filters by entity
○ e.g. {"dc":"dc1"} vs {"dc":"dc2"} queue filters
● Scheduler can apply base filter
○ only handles entities with {"dc":"dc1"}
● Worker can report home using:
○ Redis (we use this across DCs)
○ HTTPS (AWS->DC)
Multi DC / Zone deployment possible

ZMON in AWS / Multi DC Setup
*.foo.example.org *.bar.example.org
Team "Foo" Team "Bar"
EC2
Instance
EC2
InstanceEC2
Instance
EC2
Instance
ZMON
Appliance
ZMON
ApplianceEC2
Instance
EC2
Instance
ZMON
Data Service
ELB ELB

Spring Boot (extending metrics)
https://github.com/zalando/zmon-actuator
Python (Swagger first on Flask)
https://github.com/zalando/connexion
Clojure (Swagger first)
https://github.com/zalando-stups/friboo/
Scala Play
https://github.com/zalando-incubator/markscheider
Example libraries and framework support ...

Demo:
https://demo.zmon.io
ZMON and Slack:
https://zmon.io && https://slack.zmon.io
Documentation:
https://docs.zmon.io
Zalando Tech:
https://tech.zalando.com

Expose your data / Convention on key names/structure
{
"zmon.response.200.GET.checks.all-active-check-definitions.count": 10,
"zmon.response.200.GET.checks.all-active-check-definitions.fifteenMinuteRate": 0.18071,
"zmon.response.200.GET.checks.all-active-check-definitions.fiveMinuteRate": 0.15181,
"zmon.response.200.GET.checks.all-active-check-definitions.oneMinuteRate": 0.10512,
"zmon.response.200.GET.checks.all-active-check-definitions.75thPercentile": 1173,
"zmon.response.200.GET.checks.all-active-check-definitions.max": 1282,
"zmon.response.200.GET.checks.all-active-check-definitions.median": 1161,
"zmon.response.200.GET.checks.all-active-check-definitions.min": 1114
}

OSMC 2016 | ZMON: Zalando's OS approach to monitoring in the cloud and DCs by Jan Mussler

More Related Content

What's hot (19)

Similar to OSMC 2016 | ZMON: Zalando's OS approach to monitoring in the cloud and DCs by Jan Mussler (20)

Recently uploaded (20)

OSMC 2016 | ZMON: Zalando's OS approach to monitoring in the cloud and DCs by Jan Mussler