Monitoring system for OpenStack,using a OSS products

Copyright© 2015,NTT Software Corporation. All rights reserved. 1
２０１5．10．27
NTT Software Corporation
Cloud and Security Business Department

Copyright© 2015,NTT Software Corporation. All rights reserved.
Agenda
• Introduction
• Server monitoring system
– Monitoring of the physical machine
– Monitoring of the virtual machine
– Integrate multiple of Zabbix screen
• Log monitoring system
2

Introduction
• NTT Software uses OpenStack from Essex version.
• Monitoring and Log analysis has been an
important issue during the development ,test
and operation
• I introduce our system that was resolved with a
combination of OSS products.
3

what do you need to monitoring
for the OpenStack.
Common
• Develop only a little.
• One screen.
• Automation
physical
• Resource Monitoring
• Middleware Monitoring
• Service Monitoring
Log Monitoring
• Collect
• Visualize
• Efficiency of error analysis.
• Automatic analysis and
notification
Virtual
• Automatic registration of
monitoring
• Automatic release of
monitoring
• Monitoring of virtual resources
4
1 2
4 3

for the OpenStack.
Common
• One screen.
• Automation
physical
Log Monitoring
• Collect
• Visualize
notification
Virtual
monitoring
monitoring
5
1 2
4 3

Keyword
I think the OpenStack as one application.
Don’t think as a same layer ,
physical and virtual machine monitoring method .
Failure of the physical side , detect by the physical side.
Failure of the virtual side , detect by the virtual side.
Log monitoring is EFK stack + Norikra + Zabbix
6

Separate the physical and virtual
7
Physical Server
Middle
ware
OpenStack
VM VM
Physical
side
Virtual
side

Server Monitoring System
8

for the OpenStack.
Common
• One screen.
• Automation
physical
Log Monitoring
• Collect
• Visualize
notification
Virtual
monitoring
monitoring
9
1 2
4 3

Server Monitoring(Physical)
• It need what you need in the general application monitoring.
• It is particularly necessary for the monitoring of OpenStack.
– Middleware Monitoring
– Service Monitoring
– Resource Monitoring
10

Physical Server Monitoring
(UserParameter)
11
How to Monitor Middleware and Service
• UserParameter
• You get a script execution results as a monitoring
items.
• You can use plugin for Sensu, Nagios...and so on

Physical Server Monitoring
(UserParameter)
• UserParameter for nova service
– It collects results of Nova hypervisor-show
•
(※)
引用：https://github.com/sensu-plugins/sensu-plugins-openstack
12
UserParameter=nova.hypervisor-state.running_vms,python /etc/zabbix/bin/nova-hypervisor-
metrics.py -u admin -p admin -t admin -a http://192.168.0.10:35357/v2.0 | awk -F '[. t]'
'$5=="running_vms" { x=x+$6 } END{print x}'
$ python /etc/zabbix/bin/nova-hypervisor-metrics.py -u admin -p
devstack -t admin -a http://192.168.0.5:35357/v2.0 | awk -F '[. t]'
'$5=="running_vms" { x=x+$6 } END{print x}'
1

for the OpenStack.
Common
• One screen.
• Automation
physical
Log Monitoring
• Collect
• Visualize
notification
Virtual
monitoring
monitoring
13
1 2
4 3

Where is Zabbix-Server
14
Project A
VM VM
Hypervisor Zabbix
Project A
Hypervisor
VM
Zabbi
x
Project B
VM
Zabbi
x
Pattern A Pattern B

Virtual Machine Monitoring
(auto-registration)
• You can automatically register the monitored host to
zabbix-server.
• If it is in set to zabbix-agent,zabbix-server's function ,the
monitoring settings automatically.
• ex)
– zabbix-agent.conf
metadata="<foo>"
– Zabbix-server ‘s setting
IF metadata=controller then template = controller
15

(auto-registration)
16
WEB-server
WEB-server
WEB-server
metadata=web
DB-server
DB-server
DB-server
DB-server
DB-server
metadata=db
APP
server
APP
server
metadata=app
Zabbix
Server
IF metadata=web then template = webTemplate
IF metadata=db then template = dbTemplate
IF metadata=app then template = appTemplate

(network discovery)
• The network discovery function that is used in automatic
monitoring setting added,
It use to release the monitoring settings.
• It doesn’t distinguish between
the power-off / delete / failure of VM.
• If you don't restore , when event been abnormality
notification.
→ No recovery plan
→ Safe to delete from the monitoring target
→ auto-registration after restoration
17

(network discovery)
18
Node1 Node2 Node3 Node4 Node5 zabbix
Monitoring
Network
192.168.100.0/24
Alert

Fault of virtual resources
19
Compute
Node
Storage
Node
Nova Cinder
VM
NW 192.168.0.0/24
VM VM
VM VM VM

What do you monitor ?
• PingAlive
• ALL ProcessProcess
• CPU
• Memory
• Disk
Resource
• HW
• NWSNMP
20

Fault of virtual resources
• PingAlive
• ALL ProcessProcess
• CPU
• Memory
• Disk
Resource
• HW
• NWSNMP
21
Detectable

Integrate multiple of Zabbix screen
22

Where is Zabbix-Server
23
Project A
VM VM
Hypervisor Zabbix
Project A
Hypervisor
VM
Zabbi
x
Project B
VM
Zabbi
x
Pattern A Pattern B

Too many tabs
24
Tenant A Tenant B
Tenant C Tenant D
Tenant Hatohol
Hatohol
L3

Hatohol
25
Zabbix
1
Zabbix
2

Summary of server monitoring
• Failure of the physical side , detect by the physical side.
• Failure of the virtual side , detect by the virtual side.
• You can use plugin for sensu, Nagios...etc.
for middleware and OpenStack service monitoring.
• Add to host「 auto-registration 」
• Del to host「 network discovery 」
• Integrate multiple of Zabbix screen that using hatohol.
26

Log Monitoring System
27

for the OpenStack.
Common
• One screen.
• Automation
physical
Log Monitoring
• Collect
• Visualize
notification
Virtual
monitoring
monitoring
28
1 2
4 3

EFK+NZ Log Monitoring
Use Name
Search engine Elasticsearch
Log collect Fluentd
Log visualize Kibana
Log analysis Norikra
Notification Zabbix
29

30

• What is Norikra
31
Schema-less Stream Processing with SQL
Norikra is a open source server software provides "Stream
Processing" with SQL, written in JRuby, runs on JVM, licensed under
GPLv2.
Quotation ： http://norikra.github.io/

• Streaming log
Cut sometime→Analysis→Loop
32
Quotation：
Esper: Event Processing for Java
http://www.espertech.com/products/esper

EFK+NZ Structure
33
OpenStack
Fluentd
Monitoring
Server
Fluentd Norikra
Elastic
search
File
Zabbix
Kibana

• Most important things,
how to write the rules of Norikra
• Our experience and know-how
will be rule of the Norikra.
– develop・test・trouble
– Log list
• Every version.
• It does not work if there is no rule after
construction.
34

Detect the variation of the Kibana graph
• ex1) Detect the variation of the Kibana graph.
– Suspicious activity
• ex2) Detect a failure as a pattern.
– Error analysis
35

• keystone don't have mechanism to detect the dos attack.
So, I want to detect it.
• When under attack, 401 error log increases.
• discover is possible if you look at the graph of Kibana.
However, I must always watch.
36

• keystone don't have mechanism to detect the dos attack.
So, I want to detect it.
• When under attack, 401 error log increases.
• discover is possible if you look at the graph of Kibana.
However, I must always watch.
37

38
To Norikra of rules it
writes with "If from
the same IP address
per unit time 401
error in ○ times
more access
occurs." ,

39
Keystone was
attacked !!

ERROR Analysis
• Detection of the patterned error
• Log AAA+ Log BBB = Error Type 002
• Manually when you analyze.
– Lock at log file
nova-api.log nova-conductor.log nova-compute.log…and so on
– Use grep command.
– He requires a log analysis skills.
40

ERROR Analysis
41
• Ex) Using a Flavor that
is too high memory.
Failure to start.

ERROR Analysis
42
what's happened.

ERROR Analysis
43
Error cause is not
writing.

ERROR Analysis
44
Zabbix was notified as
a error type 002

Future Action
• Our experience and know-how from the
Essex version will continue efforts to expand
the rule of Norikra .
45

Announcement
• I will demonstrate the log monitoring system.
– Place：NTT Group booth S14
– Date：29(Thu) 10:30～
46

ご清聴ありがとうございました。
THANK YOU FOR LISTENING.
47

References Quotation
• Monasca
https://wiki.openstack.org/wiki/Monasca
• Monasca/Monitoring Of Monasca
https://wiki.openstack.org/wiki/Monasca/Monitoring_Of_Monasca
• Monasca/Logging
https://wiki.openstack.org/wiki/Monasca/Logging
• Zabbix Documentation 2.2
https://www.zabbix.com/documentation/2.2/
• Elastic
https://www.elastic.co/jp/
• Norikra
http://norikra.github.io/
• EsperTech
http://www.espertech.com/products/esper.php
• Treasure Data Inc
http://www.treasuredata.com/
48

Trademark
OpenStackは、米国におけるOpenStack,LLCの登録商標です。
Zabbixはラトビア共和国にあるZabbix LLCの商標です。
Erasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.
logstash is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.
Kibana is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.
その他、文中に記載されている商品・サービス名、および会社名は、それぞれ各社の商標また
は登録商標です。
49

Monitoring system for OpenStack,using a OSS products

More Related Content

What's hot (20)

Similar to Monitoring system for OpenStack,using a OSS products (20)

Recently uploaded (20)

Monitoring system for OpenStack,using a OSS products

Editor's Notes