OpenStack Day 2 Operations

confidential
Day 2 operations
OpenStack Meetup, Ottawa, Sep 26, 2017

confidential
Dirk Wallerstorfer
 Cloud Technology Strategist @ Dynatrace
 Tech enthusiast
 Husband
 Father
 Son
 Austrian (no kangaroos)
 Never seen “Sound of music”
 Yes, I own a lederhosn
 No, I don’t know how to yodel
@wall_dirk
dirk.wallerstorfer@dynatrace.com

confidential
Day 0 Day 1 Day 2 Day 3

confidential
NRPE (Nagios Remote Plugin Executor)
...
server_port=5666
nrpe_user=nrpe
allowed_hosts=10.0.0.10
...
/etc/nagios/nrpe.cfg
1
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2

confidential
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
1
2
controller
compute1
compute2
Configuration

confidential
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
1
2
controller
compute1
compute2
define host {
address 10.0.0.3
...
}
...
define service {
...
check_command nova-list
}
etc/servers/controller.cfg
...
define command{
command_name nova-list
command_line /usr/local/nagios/plugins/nova-list
}
etc/objects/commands.cfg
#!/bin/env bash
export OS_USERNAME=admin
export OS_PASSWORD=admin
export OS_AUTH_URL=http://10.0.0.3:5000/v2.0
export OS_TENANT_NAME=admin
data=$(openstack server list 2>&1)
rv=$?
if [ "$rv" != "0" ] ; then
echo $data
exit $rv
fi
echo "$data" | grep -v -e '--------' -e '| Status ' -e '^$' | wc -l
/usr/local/nagios/plugins/nova-list

confidentialhttps://github.com/rochaporto/dashing-openstack

confidential
Nagios
 Monitoring IT infrastructure – and more ...
 Monitoring static entities
 Possibility to monitor dynamically created VMs
 Modify code of nova-api to reconfigure nagios after creation of new VM
 Otherwise shutdown instance will be displayed as outage
 Image that contains Nagios plugins || CAPS scripts to install plugins
 Alternative: Nagios XI – Enterprise
 Nagios Log Server
 Great talk from Nagios world 2014: Monitoring OpenStack
 https://www.youtube.com/watch?v=1U5fo6aPS-k

confidential
Elastic
 Previously: ELK Stack
 Now: Elastic Stack

confidential
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2
1 Setup & Configure Elasticsearch
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html
2 Setup & Configure Kibana
https://www.elastic.co/guide/en/kibana/current/setup.html
server.port: 5601
server.host: "10.0.0.10”
elasticsearch.url: "http://10.0.0.10:9200"
...
/etc/kibana/kibana.yml
network.host: 10.0.0.10
http.port: 9200
...
/etc/elasticsearch/elasticsearch.yml

confidential
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2
3 Setup & Configure Filebeats
filebeat.prospectors:
- input_type: log
paths:
- /var/log/keystone/*.log*
- /var/log/httpd/keystone*.log*
tags:
- keystone
exclude_files:
- ".gz$”
...
output.elasticsearch:
hosts: ["10.0.0.10:9200"]
#username: "elastic"
#password: "changeme"
...
filebeat.modules:
- module: apache2
access:
var.paths: ["/var/log/httpd/*.log*"]
/etc/filebeat/filebeat.yml

confidential
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2
4 Setup & Configure Metricbeats
metricbeat.modules:
- module: system
metricsets:
- cpu
- load
- filesystem
- fsstat
- memory
- network
- process
enabled: true
period: 10s
processes: ['.*']
output.elasticsearch:
hosts: ["10.0.0.10:9200"]
#username: "elastic"
#password: "changeme"
/etc/metricbeat/metricbeat.yml

confidential
influxdbelasticsearch mysql

confidential
Monasca Agent
The Monasca Agent supports collecting metrics from a variety of sources as follows:
 System metrics
 Nagios plugins
 Statsd
 Host alive (icmp/ssh)
 Process checks (# instances, memory, io, threads)
 Http Endpoint checks
 Service checks (mysql, rabbitmq)
 OpenStack process metrics
The Agent is extensible through configuration of additional plugins, written in Python.

confidential
OpenStack Dynatrace Cloud
Dynatrace
Mission Control
OpenStack
Services
Nova,
Neutron,
Keystone,
...
Dynatrace cluster
Security Gateway
WebUI
Access to
OpenStack APIs
Server nodes
Tenant 1
9999
Tenant 2
9999
Tenant 3
9999
443 https
8443
443 https

confidential
Setup Dynatrace Security Gateway
# wget -O sgw.sh https://zzv94586.live.dynatrace.com/....
# /bin/sh swg.sh
1
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2
SGW
Setup Dynatrace OneAgents
# wget -O agent.sh https://zzv94586.live.dynatrace.com/....
# /bin/sh agent.sh
2
Configure Keystone endpoint3

confidential
Correlation
 Correlation is a statistical measure that indicates the extent to which two or more variables fluctuate
together.

confidential
Correlation doesn‘t imply causation
B Z
Host CPU > 90%Service response time
increases by 2 seconds
B X
Z
C
A W
D
Y

confidential
Resource capacity and utilization
OpenStack service availability/performance
Supporting services
Log analytics
Applications running on top
Dependencies
Correlation of metrics/events/data
Real user monitoring, UX affects $
PaaS

confidential
This is NOT a
REGULAR SIZE
application
environment!
B X
Z
C
A W
D
Y

Confidential, Dynatrace, LLC
Thanks for
having me!

OpenStack Day 2 Operations

More Related Content

What's hot (20)

Similar to OpenStack Day 2 Operations (20)

More from Dirk Wallerstorfer (6)

Recently uploaded (20)

OpenStack Day 2 Operations

Editor's Notes