Monitoring using Sensu

R.I.Pienaar
Malta DevOps December 2016
Monitoring using Sensu

R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Who am I?
• Malta since December 2015
• Consultant for 20+ years
• Government, Finance, Health, Social Media,
Fortune 50, Startups
• DevOps, Automation, Architect,
Development
• Open Source @ github.com/ripienaar
• Linux since Kernel 99 alpha p11

Tool
vs
Framework

Checks
API
MetricsDynamic
External InputsSecure
Events
Plugins
JSON

Architecture - Overview

Architecture - Schedule

Architecture - Event

Architecture - Integrate

{
"checks": {
"procs_rhel7": {
"command": "/usr/lib64/nagios/plugins/check_procs -w 50 -c 70 -k",
"interval": 300,
"subscribers": [
"RedHat-7"
]
}
}
}
Conﬁg - Checks

{
"checks": {
"procs_rhel7": {
"interval": 300,
"subscribers": [
"RedHat-7"
]
}
}
}
Conﬁg - Checks
Name
Command to run
How often
Where

{
"client": {
"name": "dev3.devco.net",
"address": "139.162.163.118",
"subscriptions": [
"all",
"linode_fra",
"de",
"RedHat-7"
],
"redact": [
],
"socket": {
"bind": "127.0.0.1",
"port": 3030
},
"procs": {
"total": {
"warn": 80,
"crit": 90
}
}
}
}
Conﬁg - Clients
Local event sync
What am I?
Node Parameters

{
"client": {
…,
"procs": {
"total": {
"warn": 80,
"crit": 90
}
}
}
}
Conﬁg - Clients
Node Parameters
{
"checks": {
"procs_rhel7": {
"interval": 300,
"subscribers": [
"RedHat-7"
]
}
}
}

{
"client": {
…,
"procs": {
"total": {
"warn": 80,
"crit": 90
}
}
}
}
Conﬁg - Clients
Node Parameters
{
"checks": {
"procs_rhel7": {
"command": “…/check_procs -w :::procs.total.warn|50::: -c :::procs.total.crit|70::: -k",
"interval": 300,
"subscribers": [
"RedHat-7"
]
}
}
}
Check Tokens

require ‘sensu-plugin/check/cli'
class CheckFile < Sensu::Plugin::Check::CLI
option :file,
long: “—file FILE”,
description: “The file to check”,
default: nil
def run
unknown(“no file specified, please use —file”) unless config[:file]
if File.exist?(config[:file])
ok(“file %s exist” % [config[:file]])
else
critical(“file %s does not exist” % [config[:file]])
end
end
end
Checks - Writing in Ruby

require “sensu-plugin/check/cli”
class CheckFile < Sensu::Plugin::Check::CLI
option :file,
long: “—file FILE”,
description: “The file to check”,
default: nil
def run
unknown(“no file specified, please use —file”) unless config[:file]
if File.exist?(config[:file])
ok(“file %s exist” % [config[:file]])
else
critical(“file %s does not exist” % [config[:file]])
end
end
end
Checks - Writing in Ruby

# yum install nagios-plugins-all
# /usr/lib64/nagios/plugin/check_file_age —help
…
Usage:
check_file_age [-w <secs>] [-c <secs>] [-W <size>] [-C <size>] [-i] -f <file>
check_file_age [-h | --help]
check_file_age [-V | --version]
# /usr/lib64/nagios/plugins/check_file_age -f /keepalive; echo $?
FILE_AGE UNKNOWN: No file specified
3
FILE_AGE CRITICAL: File not found - /keepalive
2
FILE_AGE WARNING: /keepalive is 353 seconds old and 0 bytes
1
FILE_AGE OK: keepalive is 4 seconds old and 0 bytes
0
Checks - Reuse

Checks - Reuse
http://sensu-plugins.io/

$ cat <<EOF > /dev/tcp/localhost/3030
{
"name":"my_awesome_check",
"output":"everything is fine",
"ttl":600,
“status”:0
}
EOF
Checks -Your own App

{
"mailer": {
"mail_from": "sensu@example.com",
"mail_to": "monitor@example.com",
"smtp_address": "smtp.example.org",
"smtp_port": "25",
"smtp_domain": "example.org",
"template": “/etc/sensu/templates/mail.erb”,
"subscriptions": {
"subscription_name": {
"mail_to": "teamemail@example.com"
}
}
}
}
Handlers - Email
https://github.com/sensu-plugins/sensu-plugins-mailer
/etc/sensu/handlers/mailer.json
Custom mail body

<%= output %>
Admin GUI: <%= admin_gui %>
Host: <%= @event['client']['name'] %>
Timestamp: <%= Time.at(@event['check']['issued']) %>
Address: <%= @event['client']['address'] %>
Check Name: <%= @event['check']['name'] %>
Command: <%= command %>
Status: <%= status_to_string %>
Occurrences: <%= @event['occurrences'] %>
Handlers - Email
https://github.com/sensu-plugins/sensu-plugins-mailer
/etc/sensu/templates/mail.erb

Handlers - Thirdparties

events = JSON.parse(
RestClient.get(“http://example:4567/events").body
)
statusmap = {0 => "OK", 1 => "WARNING", 2 => "CRITICAL", 3 => "UNKNOWN"}
puts "Found %d events" % [events.size]
events.each do |event|
puts "%30s: %-10s %s" % [
event["client"]["name"],
statusmap[event["check"]["status"]],
event[“check"]["output"]
]
end
API - JSON
Found 2 events
puppet1.example.net: CRITICAL DISK CRITICAL - /run/docker/netns/default is not accessible:
Permission denied
dev1.example.net: WARNING PROCS WARNING: 183 processes

API - Uchiwa
https://uchiwa.io/

Links
https://sensuapp.org/
https://uchiwa.io/
https://graphiteapp.org/
http://grafana.org/
http://sensu-plugins.io/

Questions?
twitter: @ripienaar
email: rip@devco.net
blog: www.devco.net
github: ripienaar
freenode: Volcane
slack.puppet.com: ripienaar
https://www.devco.net/

Monitoring using Sensu

More Related Content

What's hot (6)

Similar to Monitoring using Sensu (20)

Recently uploaded (20)

Monitoring using Sensu