Building a Scalable Monitoring System

@molly_struve
Building A Scalable
Monitoring System
1

@molly_struve
Monitoring Mistakes
Overhauling the
System
The Payoff
2

@molly_struve
Monitoring Mistakes
3
Overhauling the
System
The Payoff

@molly_struve
Monitoring Mistakes
5
Overhauling the
System
The Payoff

@molly_struve
Monitoring Mistakes
Overhauling the
System
6
Overhauling the
System
The Payoff

@molly_struve
Monitoring Mistakes
Overhauling the
System
The Payoff
7

@molly_struve
Monitoring Mistakes
Overhauling the
System
The Payoff
8

@molly_struve
Monitoring Mistakes Overhauling the System The Payoff
9

@molly_struve
10

@molly_struve
11

@molly_struve
12

@molly_struve
13

@molly_struve
14

@molly_struve
15

@molly_struve
16

@molly_struve
17

@molly_struve
18

@molly_struve
19

@molly_struve
20

@molly_struve
21

@molly_struve
22

@molly_struve
23

@molly_struve
24

@molly_struve
25

@molly_struve
Incredibly Inconsistent
26

@molly_struve
Inconsistent Alerts
Required no actionReported data
27

@molly_struve
Inconsistent Alerts
Required no actionReported data
Immediate action required
28

@molly_struve
make on-call devs miserable
29

@molly_struve
30

@molly_struve
31

@molly_struve
32

@molly_struve
Monitoring Mistakes
Overhauling the
System
The Payoff
33

@molly_struve
Monitoring Mistakes
Overhauling the
System
The Payoff
34

@molly_struve
Monitoring Must Haves
1
2
3
4
Consolidate Monitoring To a Single Place
35

@molly_struve
!
" #
$
%
36

@molly_struve
37

@molly_struve
38

@molly_struve
39

@molly_struve
40

@molly_struve
41

@molly_struve
42

@molly_struve
43

@molly_struve
Make ALL Alerts Actionable
1
2
3
4
44

@molly_struve
45
Action

@molly_struve
46
Action

@molly_struve
Action Required No action Needed
47

@molly_struve
#ops_alerts
#dev_alerts
48

@molly_struve
#ops_alerts
#ops_reporting
#dev_alerts
#dev_reporting
49

@molly_struve
Make Alerts Mutable
1
2
3
4
50

@molly_struve
Make Alerts Mutable
1
2
3
4
51

@molly_struve
30 60 90
minutes
52

@molly_struve
53

@molly_struve
54
Miss new alerts

@molly_struve
55

@molly_struve
56

@molly_struve
57

@molly_struve
58

@molly_struve
59

@molly_struve
Track Alert History
Make Sure Alerts Are Mutable
1
2
3
4
60

@molly_struve
Tracking Alert History
61

@molly_struve
62

@molly_struve
63

@molly_struve
Track Alert History
1
2
3
4
64

@molly_struve
Monitoring Mistakes
Overhauling the
System
The Payoff
65

@molly_struve
Monitoring Mistakes
Overhauling the
System
The Payoff
66

@molly_struve
On boarding is a breeze
67

@molly_struve
3 On-boarding steps:
1
2
3
68

@molly_struve
Show them the monitoring setup1
2
3
69

@molly_struve
2
3
If an alert goes off you have to address it
70

@molly_struve
2
3
How to mute a triggered alert
71
If an alert goes off you have to address it

@molly_struve
On boarding is a breeze
72

@molly_struve
Happier on-call developers
73

@molly_struve
All alerts must be actionable
74

@molly_struve
75

@molly_struve
No more noise
76

@molly_struve
77

@molly_struve
Developers began helping to
improve our monitoring system
78

@molly_struve
30-40 Alerts
79

@molly_struve
>90 Alerts
80

@molly_struve
81

@molly_struve
If you want…
1
2
3
82

@molly_struve
On boarding to be a breeze.1
2
3
If you want…
83

@molly_struve
On-call developers to be a lot happier
1
2
3
If you want…
On boarding to be a breeze.
84

@molly_struve
Developers to help improve your monitoring
systems
1
2
3
If you want…
On boarding to be a breeze.
85
On-call developers to be a lot happier

@molly_struve
Track Alert History
1
2
3
4
86

Building a Scalable Monitoring System

More Related Content

More from Molly Struve (11)

Recently uploaded (20)

Building a Scalable Monitoring System