Deep dive into Nagios analytics

A Deep Dive into
Nagios Analytics
Alexis Lê-Quôc (@alq)
http://datadoghq.com

@alq
Dev & Ops
Nagios user since 2008
Datadog co-founder

That woke me up
That I responded to
last week
That I responded to
5 weeks ago
Top 3 failed checks
That most of my team
responded to at least once
That impacts our business
the most?

At best, finding local optimums
Using memory to
prioritize remediation...
At worst, brownian motion

Performance Metrics Nagios Traffic Other Sources
In the “Cloud”
Real-time graphs + analytics

Real-time Analytics
(Nagios et al.)

Nagios Traffic
In the “Cloud”
Real-time graphs + analytics

Nagios a “chatty” source
out of 40+ Datadog supports

Deep dive into Nagios analytics

Almost 13000 Nagios “events”
over past week

A diNaolto a sgci ewntifiict shtud ydata

6
4
2
0
0 250 500 750
Host count
Population
factor(quartile)
1
2
3
4
Nagios samples
Population
25%
50%
75%
100%
20
93
322
904

40
30
20
10
0
40
30
20
10
0
40
30
20
10
0
40
30
20
10
0
1 2 3 4
0 250 500 750 1000
Nagios alert per host
count per week
Weekly count per host split by quartile

40
30
20
10
0
40
30
20
10
0
40
30
20
10
0
40
30
20
10
0
1 2 3 4
0 250 500 750 1000
Nagios alert per host
count per week
Weekly count per host split by quartile
Outliers
Sick hosts,
silenced checks

Notifications
1-3% of alerts notify
Little difference per quartile

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
12
8
4
12
8
4
12
8
4
12
8
4
1 2 3 4
0 5 10 15 20
Hour of Day (UTC)
Alerts per hour

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Mean about the same
across quartiles
●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
12
8
4
12
8
4
12
8
4
12
8
4
1 2 3 4
0 5 10 15 20
Hour of Day (UTC)
Alerts per hour
Time-based deviation?

40
30
20
10
0
40
30
20
10
0
40
30
20
10
0
40
30
20
10
0
1 2 3 4
Sun Mon Tue Wed Thu Fri Sat
Day of week
Alerts per hour
Notifying Alerts per Day

40
30
20
10
0
40
30
20
10
0
40
30
20
10
0
40
30
20
10
0
1 2 3 4
Sun Mon Tue Wed Thu Fri Sat
Day of week
Alerts per hour
Notifying Alerts per Day
Not really

30
20
10
0
30
20
10
0
30
20
10
0
30
20
10
0
1 2 3 4
0 50 100 150 200 250
Checks ranked by noise
Alerts per hour
Noisiest checks (overall)
Outlier

●
●
●
● ●
● ●
● ● ● ● ● ● ● ● ● ●
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
● ● ●
30
20
10
0
0 20 40
Alerts per hour
Noisiest checks (outlier)
Outlier in more detail

8
6
4
2
0
8
6
4
2
0
8
6
4
2
0
8
6
4
2
0
1 2 3 4
0 50 100 150 200
Alerts per hour
Noisiest checks (without outlier)
Long Tail

30
20
10
0
30
20
10
0
30
20
10
0
30
20
10
0
1 2 3 4
0 50 100 150 200
Hosts ranked by noise
Alerts per hour
Noisiest hosts (overall)
Same outlier

●
●
● ●
●
●
●
●
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
● ● ● ● ● ● ● ● ●
●
30
20
10
0
3
0 20 40 60
Hosts ranked by noise
Alerts per hour
Noisiest hosts (outlier)
Similar pattern as checks

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
● ● ●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●●
●
●
●
● ●
●
●
●●
●
●
●
●
●
● ●
● ●
● ●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
● ●
●
●●
●●●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
● ●
●
●
●
●
●
●●
●
●
● ●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
● ●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●●
●●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
● ●
●
●
● ● ●
●
●
●
●
●●
●
●
●
●●
●
●
● ●
●
●●
●●
●
●
●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
● ●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ● ●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
● ●
●
●
● ●
●
●
●
●●
●
●●●
●●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●● ●
●●
●
●● Happens
150
Often
100
50
0
0 100 200 300
Age between earliest and latest occurrence
Number of days occurring
factor(quartile)
●
●
●
●
1
2
3
4
Alert age & frequency of occurrence
Young Old
Seldom
happens

Occur often, for a long time Tolerated
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
● ● ●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●●
●
●
●
● ●
●
●
●●
●
●
●
●
●
● ●
● ●
● ●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
● ●
●
●●
●●●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
● ●
●
●
●
●
●
●●
●
●
● ●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
● ●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●●
●●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
● ●
●
●
● ● ●
●
●
●
●
●●
●
●
●
●●
●
●
● ●
●
●●
●●
●
●
●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
● ●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ● ●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
● ●
●
●
● ●
●
●
●
●●
●
●●●
●●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●● ●
●●
●
●● 150
100
50
0
0 100 200 300
Age between earliest and latest occurrence
Number of days occurring
factor(quartile)
●
●
●
●
1
2
3
4
Alert age & frequency of occurrence
Happen once in a while

Awk
R
ggplot2
Find out tomorrow!
Postgres
d3

Take-aways
• Don’t rely on your memory to prioritize
• Your Nagios logs are a treasure trove
• Have a dialog with your data
• Presentation matters

Curious about Datadog?
Like cute logos?
http://dtdg.co/nagios2012

Deep dive into Nagios analytics

More Related Content

Similar to Deep dive into Nagios analytics (16)

More from Datadog (20)

Recently uploaded (20)

Deep dive into Nagios analytics