RabbitMQ + CouchDB = Awesome
24 likes8,262 views
This document proposes a system using RabbitMQ and CouchDB to provide a scalable and flexible backend that can handle various frontends. RabbitMQ is used for messaging between daemons, while CouchDB is used to define and manage workflows and store persistent messages to allow asynchronous callbacks to continue workflows. The system addresses challenges of scaling, different frontends, cloud hosting, maintenance, asynchronous tasks, and message correlation.
![{
"_id": "f7a4408898e5...a05b4181045",
"_rev": "3-633060011",
"name": "billing_info",
"type": "workflow",
"user": "iwmn",
"path": [
"logger",
"billing",
"billing_iwmn_cc"
]
}](https://image.slidesharecdn.com/one-system-to-rule-them-all-091014153738-phpapp02/85/RabbitMQ-CouchDB-Awesome-26-320.jpg)
![{
"_id": "22cd37afd06c82129adb665d3150f570",
"_rev": "3-2669449653",
"status": "done",
"last_update": 1252296819,
"name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789",
"message": {
"response": {
"expiry": 710,
"name": "Lenz Gschwendtner",
"last4": 0,
"cc_type": "master"
},
"data": {
"options": {
"billing_id": "E16003D4-F8DB-11DD-B312-BEA23D9AB789"
},
"command": "billing_info"
},
"meta": {
"lang": "de",
"name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789",
"orig_reply_to": "D234ADE8-9B64-11DE-A92C-E5EF11C6FBDB",
"reply_to": "workflow",
"user": "iwmn",
"log": [
"logger",
"billing",
"billing.iwmn.info"
],
"platform": "iwmn"
}
},
"created": 1252296815,
"type": "bill"
}](https://image.slidesharecdn.com/one-system-to-rule-them-all-091014153738-phpapp02/85/RabbitMQ-CouchDB-Awesome-37-320.jpg)
RabbitMQ + CouchDB = Awesome
- 1. one system to rule them all
- 4. • write a backend that scales well • handle various frontends (web, iPhone, ...) • runs “somewhere in the cloud” • easy to maintain and extend
- 5. the solution
- 7. Part I
- 8. RabbitMQ • AMQP based Message Queue • lots of additional transports (XMPP, HTTP, STOMP, ...) • erlang based
- 9. frontend frontend frontend daemon RabbitMQ daemon daemon daemon
- 10. • every functionality gets a backed daemon • every daemon has a queue associated • communication via a JSON message that gets passed around
- 11. the JSON message
- 13. { "response": { "expiry": 710, "ts": "1234425918", "name": "Lenz Gschwendtner", "last4": 0, "cc_type": "master" }, "data": { "options": { "billing_id": "E16003D4-B312-BEA23D9AB789" }, "command": "billing_info" }, "meta": { "lang": "en", "reply_to": "B312-E16003D4-BEA23D9AB789", "platform": "iwmn" } }
- 14. works great for ... • RPC style communication • if the flow through the system is static • fast transactions
- 15. gets fiddly when ... • there are asynchronous callbacks from outside • message flow can change based on various factors • tasks can take days to complete
- 16. Part II
- 17. workflows
- 19. ... again ...
- 20. what we have ... • various frontends • backend daemons • JSON messages
- 21. what we need • a way to describe message flow • a easy flexible way to write and update it
- 22. CouchDB
- 23. ... to the rescue
- 25. workflow • really only a path definition through backends • some sort of condition management • a backend definition
- 26. { "_id": "f7a4408898e5...a05b4181045", "_rev": "3-633060011", "name": "billing_info", "type": "workflow", "user": "iwmn", "path": [ "logger", "billing", "billing_iwmn_cc" ] }
- 27. ... and the backend definition
- 28. { "_id": "33adf2e3efc3...f423929f057ac", "_rev": "3-3507820969", "conn": "rpc", "type": "worker", "name": "billing_iwmn_cc", "location": "billing.cc", "description": "the CC interface", "command": "billing_verify_cc" }
- 29. now we need a workflow daemon
- 30. frontend frontend frontend daemon RabbitMQ workflow daemon daemon daemon
- 31. where are we now? • we can call backends as before • we can call workflows via the workflow queue • we can manage workflows in CouchDB
- 32. Part III
- 35. problem • we start a workflow • we call some external API • the external API somehow notifies us (Mail, Callback URL, ...) about progress • we need to continue in the workflow
- 36. we need a persistent message store
- 37. { "_id": "22cd37afd06c82129adb665d3150f570", "_rev": "3-2669449653", "status": "done", "last_update": 1252296819, "name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789", "message": { "response": { "expiry": 710, "name": "Lenz Gschwendtner", "last4": 0, "cc_type": "master" }, "data": { "options": { "billing_id": "E16003D4-F8DB-11DD-B312-BEA23D9AB789" }, "command": "billing_info" }, "meta": { "lang": "de", "name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789", "orig_reply_to": "D234ADE8-9B64-11DE-A92C-E5EF11C6FBDB", "reply_to": "workflow", "user": "iwmn", "log": [ "logger", "billing", "billing.iwmn.info" ], "platform": "iwmn" } }, "created": 1252296815, "type": "bill" }
- 38. one more problem how do we correlate messages
- 39. ... with some CouchDB awesome ...
- 40. • the backend daemon for the callback polls a view in CouchDB • we can search for any value in the request we sent • all we need is a view that searches for the very value this external API uses as correlation ID
- 41. external frontend frontend frontend callback daemon RabbitMQ workflow daemon daemon daemon
- 42. me Lenz Gschwendtner CTO iWantMyName @norbu09
- 43. credits • RabbitMQ • CouchDB • Erlang and Perl
- 44. more credits • http://www.flickr.com/photos/djpd • http://www.flickr.com/photos/lilyapp • http://www.flickr.com/photos/b-tal • http://www.flickr.com/photos/mikebaird • http://www.flickr.com/photos/eyetwist