From Zero To Visibility
- 1. From Zero To Visibility Bridget Kromhout
- 2. 8thbridge.com small social commerce startup acquired in the last month by Fluid, Inc. small devteam I am the ops team http://www.thedirtbox.com/wp-content/uploads/2013/01/ping-pongart.jpg
- 3. twisty maze of little shell scripts http://www.pcgameshardware.de/screenshots/1280x1024/2007/07/CA01.jpg
- 4. time-consuming to understand difficult to modify doesn’t scale artisanal monitoring?! http://shop.bespokebacon.com/images/bespoke-logo.final(3).png
- 5. New Relic pros: nice graphs application-level view good error analysis cons: slow to update many false-positive alerts high prices (better now)
- 7. as hideous as you remember
- 8. “Horrendous interface” “Well, it’s more “old” than anything else. At least everything is in the same place as you left it because it’s been the same since 1912.” https://laur.ie/blog/2014/02/why-ill-be-letting-nagios-live-on-a-bit-longer-thank-you-very-much/ not alone!
- 9. “Sensu has so many moving parts that I wouldn’t be able to sleep at night unless I set up a Nagios instance to make sure they were all running.” who watches the RabbitMQ? -- @murphy_slaw (via @lozzd) http://images.sodahead.com/profiles/0/0/0/5/1/6/6/3/9/Watchmen-trademark-symbol-62141795529.jpeg http://portertech.ca/images/2011-11-01/sensu-diagram.png
- 10. hating on nagios: the middle years
- 11. “hadoop does not suffer from a paucity of configuration options” http://jaganesundar.wordpress.com/2011/12/05/installing-and-configuring-hadoop-0-20-205-using-it-rpm/ monitor all the ports?! best way to monitor HBase: hbck: the HBase consistency checker nagios -> bash script -> parsing output of hbck http://www.ymc.ch/en/how-to-monitor-hbase-health-by-nagios
- 14. “Cyber” monday: 1988 called; wants its word back.
- 15. wow. such nosql. very webscale. “a single write operation holds the lock exclusively, and no other read or write operations may share the lock.”
- 16. “If it moves, we track it. Sometimes we’ll draw a graph of something that isn’t moving yet, just in case it decides to make a run for it.” Ian Malpass, Etsy http://codeascraft.com/2011/02/15/measure-anything-measure-everything/
- 17. the (former) state of our graphite & statsd ● Graphite 0.9.9 ○ hand-rolled ○ over 2 years old ○ missing new features (Consolidate by!) ● StatsD was newish, but… ○ hand-rolled ○ running in a screen session ○ on a special snowflake box
- 18. http://media-cache-ec0.pinimg.com/736x/68/c2/9d/68c29deb72bad94cd4e3c1aa0f3cdcd8.jpg this is wrong tool. never use this.
- 19. Community cookbooks? ● StatsD ○ https://github.com/librato/statsd-cookbook ● Graphite ones good, but… ○ focus on Apache (we use nginx) ○ we haven’t moved to Chef 11 (gasp!)
- 20. when in doubt: tcpdump is your friend http://blog.johngoulah.com/2012/10/looking-under-the-covers-of-statsd/
- 21. carbon-aggravator (between 0.9.10 & 0.9.12) # If set true, metric received will be forwarded to # DESTINATIONS in addition to # the output of the aggregation rules. If set false # the carbon-aggregator will # only ever send the output of aggregation. FORWARD_ALL = True
- 22. carbonate: A+++ would clone again whisper-fill.py backfill datapoints between whisper files
- 23. life as a third wheel party thresholds: because not every outage is abrupt normal traffic decision to turn off decision to turn back on accidental removal
- 25. all the things StatsD Application-level error analysis Alarms for autoscaling Timers & counters Log & host-level Hadoop & HBase visualization MongoDB Graphs Time-series data graphing client-side plugins Threshold-based alarmsDashboard external checks
- 27. what even is ideal monitoring solution http://www.quickmeme.com/img/f5/f512ff9bee084263df5571d3c81388019dcb063173e1dbcfa2babac9274576b6.jpg ❏ finds real problems ❏ actionable alerting ❏ usable by all ❏ …?
- 28. questions; comments; whatnot Twitter: @bridgetkromhout Email: bridget@kromhout.org In person: DevOps Days Minneapolis (devopsdays.org)