Real time system_performance_mon
5 likes3,391 views
This document discusses using message queues and AMQP for real-time system performance monitoring and job queuing. It describes how message queues allow producers and consumers to communicate asynchronously. The author details their experience building a job server using RabbitMQ, Catalyst, and Web::Hippie to queue and monitor long-running jobs. While the system is still a work in progress, all the major components are there and it is in use in production.
Real time system_performance_mon
- 1. Real time system performance monitoring AMQP and CatalystX::JobServer Tomas (t0m) Doran São Paulo.pm perl workshop 2010
- 2. Aka the “my code doesn’t work yet” talk • I wrote half the slides for this at 6am Friday (São Paulo time) • The title is a bit misleading, as I wrote the talk abstract in March, then wrote entirely different software. • Sorry about that.
- 3. Message queueing • Lets not talk about the (not working) code just yet... • This talk is about Message Queueing. • What is Message Queueing? • Why would I want to use it?
- 4. What is message queueing. • In it’s simplest form, it’s just a list. • 1 (or more) ‘producers’ (writers) • 1 (or more) ‘consumers’ (readers) • Queue if rate of production > rate of consumption
- 5. Why do I want it? • Decouples producers and consumers (probably across the network). • Lets you manage load, and spikes in load (only n consumers). • In a web environment - lets you serve more pages, quicker.
- 6. Other cases • View as a list a little simplistic. • Depending on your application, you may want: • One to one • One to many • One to many (broadcast) • Many to all of the above • More complex topology?
- 7. AMQP • Queuing protocol • Fast • Interoperable (maybe?)
- 8. AMQP AMQP enables complete interoperability for messaging middleware, both the networking protocol and the semantics of broker services are defined in AMQP.
- 10. AMQP • Full abstraction of all the mechanics of messaging across the network. • Serialization and framing of messages. • Wiring of message routing is part of the protocol.
- 11. AMQP • So all your clients know (at least half) of the wiring. • Different topologies depending on routing configuration. • Can specify other options such as durability • Nice when your server dies - no ‘current config’
- 12. AMQP Concepts RabbitMQ vhost Publisher Exchange Queue Consumer
- 13. Concepts - Exchanges • Named • Messages are published (sent) to one exchange • Can be durable (lasts till deleted), or auto- deleted
- 14. Queues • Queues can be named or (or named by the server - private queues). • Queues are bound to one (or more) exchanges. • Queues can have 0 or more clients • Queues may persist with 0 clients or not. • FIFO (if you have 1 consumer) • Message never delivered to > 1 client
- 15. Bindings • Binding is what joins a queue and an exchange.
- 16. Traditional queue • Named exchange • Bound to 1 Named queue • 0 or more listeners get round-robin messages • Messages queue when nobody listens / if consumers are slow
- 17. Broadcast • Named exchange • Each client creates an anonymous ephermeral queue • Client binds the queue to that exchange • All clients get all messages • Messages go to /dev/null if no clients
- 18. Others • Different exchange types - direct, topic, fanout • You can do a lot using just these and a mix of named and anonymous queues • More complex topologies possible
- 19. I accidentally wrote software • I try to avoid doing this • I’m not very good at it ;)
- 20. I blame Net::RabbitFoot • It’s written using AnyEvent and Coro • Which I hadn’t used before, but looked good for this type of thing. • I felt my way around writing simple things standalone. • Got all the web servers logging into Rabbit
- 21. Logging into message queuing • Good example of broadcast • Want to aggregate logs to files • And be able to ‘tail’ them • Logging directly from the application • Also tailing (normal) log files to message queue
- 22. Getting the logs back
- 23. Ok, there is some setup
- 24. Ok, there is some setup
- 25. Dumping them to a file • That’s pretty simple after that.. • Except: • Log rotation • Not flushing to disk once per message • etc...
- 26. Viewing them live • Someone wrote an AMQP client in flash • AMQP security model not useful publicly • Cute prototype • (Sorry, no demo - refuses to work locally)
- 27. Queueing Jobs • Next application, simple job queue. • 1 worker, serialized.. (Crap, good enough) • Log JSON into RabbitMQ (again), inflate, call the ->run method. • Works!
- 28. Job Server • I now have several scripts, all doing bits with queuing. All duplicating code. • I want to aggregate messages • Send messages of aggregates at regular intervals • Need something more generic
- 29. Job server • Wrote a simple abstraction on getting channels, making exchanges and queues, etc • Was still going to end up with a load of scripts/jobs that needed running and managing.Yuk. • Inspecting the state of each job ‘service’ hard / boring..
- 30. But wait • Each ‘thing’ (timer, listener, logger, emitter, worker etc) can be an instance of a class with attributes for the state. • Construct a set from some config file. • Lets me aggregate jobs together / share code (Can use AnyEvent/Coro for threads). • Less processes to manage. Working out the state just got harder still.
- 31. But wait • Make all the classes use MooseX::Storage • Catalyst makes instances of things from config... • So, I could adapt all these instances into Catalyst models, write a controller, done! • Erm, but Catalyst, and message queueing, and what?!?!?
- 32. I blame Miyagawa • For Plack, and Catalyst::Engine::Plack, and Corona (the Coro PSGI server). • I had this INSANE idea one night. • I tried it. • IT WORKED. • WOAH.
- 33. I blame clkao • Next step - browser kicks off a job. • Status via long poll? • He did it already! • With long hair!
- 34. Hippie • Async pipe to the browser. • Abstracts all the nasty ajax details (also does long poll, or websockets) • Applications: • Automatically updated monitor of app components • Start a job and then get an AJAX update screen for the job progress.
- 35. Useable? • The basics work. • Running it production at work • (Since WedneThursday....) • If you need a high volume, simple, production ready job scheduler right now, use Gearman.
- 36. Useable? • Needs my github versions of Catalyst::Engine::PSGI and Web::Hippie • Running jobs from the web app doesn’t work/exist yet. • All the pieces are there - next week?
- 37. Demo?
- 38. Next steps? • Production battle scars ;) • Hippie for watching jobs run, not just component status updates. • Docs would be good... • Hippe::Pipe for bidirectional • More traits / plugins / components
- 39. What to go look at • http://www.rabbitmq.com • Docs, setup info, FAQ • Slideshare - lots of good and more detailed AMQP introductions.
- 40. What to go look at (CPAN) • Net::RabbitFoot • App::RabbitTail • Plack • Web::Hippie • CatalystX::JobServer (github)
- 41. Bonus slide! • I forgot to mention this originally... • DO NOT FORK AFTER USING Net::RabbitFoot (and then try to use it in the child process) • This will go horribly horribly wrong.
- 42. Other solutions • Gearman • STOMP • ActiveMQ (or Rabbit has a STOMP adaptor) • Catalyst::Engine::STOMP
- 43. Thanks! • Questions? • Anyone interested in playing, grab me on irc • Happy to hand-hold as I need more people using this. • I’ll love you forever if you write docs.