Monitoring with Syslog and EventMachine
19 likes2,896 views
This document discusses monitoring systems using syslog and EventMachine. It proposes building a lightweight, polyglot system that aggregates syslog events and displays metrics and visualizations using various protocols like WebSockets, Server-Sent Events, and Graphite. Event sources would send syslog messages which an EventMachine server would parse and pass to an EM:Channel. A JavaScript client could subscribe to the channel for real-time updates.
![monitor |ˈmänəәtəәr|
verb [ with obj. ]
observe and check the progress or
quality of (something) over a period of
time; keep under systematic review:
equipment was installed to monitor air quality.
Oxford Dictionary](https://image.slidesharecdn.com/monitoringsyslogeventmachine-111109100835-phpapp02/85/Monitoring-with-Syslog-and-EventMachine-8-320.jpg)
![haproxy logs
haproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17.654] px-http
px-http/srv1 9/0/7/14/+30 200 +243 - - ---- 3/3/3/1/0 0/0
"GET /image.iso HTTP/1.0"](https://image.slidesharecdn.com/monitoringsyslogeventmachine-111109100835-phpapp02/85/Monitoring-with-Syslog-and-EventMachine-39-320.jpg)
![Syslog Format
Date Hostname Program : Message
Jan 1 12:12:12 10.245.3.99 foo[421] : this is a message](https://image.slidesharecdn.com/monitoringsyslogeventmachine-111109100835-phpapp02/85/Monitoring-with-Syslog-and-EventMachine-46-320.jpg)
![New Handler
class Handler < EM::Connection
def initialize(*args)
@channel = args[0]
@counter = 0
super *args
end
def receive_data(data)
if data && data.size > 0
@counter += 1
@channel.push(@counter)
end
end
end](https://image.slidesharecdn.com/monitoringsyslogeventmachine-111109100835-phpapp02/85/Monitoring-with-Syslog-and-EventMachine-62-320.jpg)
Monitoring with Syslog and EventMachine
- 1. Monitoring with Syslog and EventMachine
- 2. About me Patrick Huesler Twitter/Github: phuesler developer@wooga Berlin, Germany
- 6. Wooga Social Games 3# game developer on facebook ~ 6.5 Million users per day
- 7. Monitoring with Syslog and EventMachine
- 8. monitor |ˈmänəәtəәr| verb [ with obj. ] observe and check the progress or quality of (something) over a period of time; keep under systematic review: equipment was installed to monitor air quality. Oxford Dictionary
- 11. Visualization of Ticket Sale
- 13. Concurrent users of Magic Land
- 17. Motivation
- 18. DevOps
- 19. Moving Target
- 20. Debugging
- 21. “Since the last deploy, the number of signup errors has gone up by 300 %. We might have broken something.”
- 22. Money Hose
- 23. Counting Things
- 24. Concurrent Users
- 25. Logins
- 26. Signups
- 27. Errors
- 28. A rough sketch
- 29. Application Application Application Server Server Server Event Aggregation Dashboard
- 30. Criteria
- 31. Simple
- 32. Fire And Forget
- 33. Non blocking
- 34. Criteria Simple/Lightweight Polyglot Fire and Forget
- 35. A Little Story
- 37. Not Supported node.js NewRelic RPM erlang NewRelic RPM
- 38. Load Balancer haproxy node.js node.js node.js
- 39. haproxy logs haproxy[674]: 127.0.0.1:33320 [15/Oct/2003:08:32:17.654] px-http px-http/srv1 9/0/7/14/+30 200 +243 - - ---- 3/3/3/1/0 0/0 "GET /image.iso HTTP/1.0"
- 40. send logs custom syslog haproxy server NewRelic Ruby New Relic Agent
- 43. Syslog
- 44. Syslog • Standard logging solution for Unix/Linux • Facility (daemon, cron, user, local0, etc.) • Priority/Level (Alert, Critical, Error, Warning, etc.) • Client and server • Since 1980
- 45. ... It also provides devices which would otherwise be unable to communicate a means to notify administrators of problems or performance. http://en.wikipedia.org/wiki/Syslog
- 46. Syslog Format Date Hostname Program : Message Jan 1 12:12:12 10.245.3.99 foo[421] : this is a message
- 47. UDP
- 48. Fire And Forget
- 49. Checklist
- 50. Simple/Lightweight ✔ Polyglot ✔ Fire and Forget ✔
- 51. Build it
- 52. send message Event Source UDP : 514 Event Aggregator Dashboard
- 53. send message Event Source UDP : 514 Syslog forward Dashboard Event Aggregator
- 54. Event Source # man logger logger -p local0.notice -t HOSTIDM My Message
- 55. Event Source require 'syslog' def log(message, level = :warning) script_name = $0 syslog_option = Syslog::LOG_PID | Syslog::LOG_CONS Syslog.open($0, syslog_option) do |s| s.send(level, message) end end
- 56. Server • Listen on UDP • Or tail a log file • receive and parse syslog message • update a counter
- 57. Server require 'eventmachine' class Handler < EM::Connection def receive_data(data) log_line = SyslogParser.parse(data) puts log_line.message end end EM.run { host = "127.0.0.1" port = "3000" EM::open_datagram_socket(host, port, Handler) }
- 58. Syslog Parser https://github.com/jordansissel/experiments/blob/master/ ruby/eventmachine-speed/basic.rb
- 59. Websocket
- 60. require 'em-websocket' EM.run do channel = EM::Channel.new options = {:host => "0.0.0.0", :port => 8080} EM::WebSocket.start(options) do |ws| ws.onopen do sid = channel.subscribe{|msg| ws.send msg} end ws.onclose do channel.unsubscribe(sid) end end end
- 61. Pass on EM:Channel EM.run { channel = EM::Channel.new # define your websocket server here # start udp server host = "127.0.0.1" port = "3000" # pass in the channel to the data handler EM::open_datagram_socket(host, port, Handler, channel) }
- 62. New Handler class Handler < EM::Connection def initialize(*args) @channel = args[0] @counter = 0 super *args end def receive_data(data) if data && data.size > 0 @counter += 1 @channel.push(@counter) end end end
- 63. JavaScript var socket; var host = "ws://localhost:8080"; var socket = new WebSocket(host); socket.onopen = function(){ console.log('open'); } socket.onmessage = function(msg){ console.log(msg.data); } socket.onclose = function(){ console.log('closed'); }
- 64. Server-sent events • Push only • http://dev.w3.org/html5/eventsource/ • http://en.wikipedia.org/wiki/Server- sent_events • http://www.html5rocks.com/en/tutorials/ casestudies/sunlight_streamcongress.html
- 65. What else?
- 67. Steal from statsd https://github.com/etsy/statsd
- 68. Inspiration • http://codeascraft.etsy.com/2011/02/15/ measure-anything-measure-everything/ • http://code.flickr.com/blog/2008/10/27/ counting-timing/
- 69. Thank You
- 70. Special thanks to • @wooga for supporting me • @knutin for facebook faces dashboard • @hukl for hummingbird video (http:// vimeo.com/20840998)
- 72. wooga.com/jobs WANTED dead or alive BACKEND DEVELOPER REWARD www.wooga.com/ jobs/