Ironmq slides
0 likes472 views
Mikko Koivunalho presents on Iron.io services IronMQ, IronWorker, and IronCache. He discusses how they enable building loosely coupled, asynchronous cloud applications that can better handle uncertainty. IronMQ is a message queue that allows separation and asynchronous messaging. IronWorker is a serverless computing platform that allows running code in a scalable way. IronCache provides key-value storage. Together these services provide solutions for problems like processing GitHub notifications asynchronously and sending automated messages like reminders.
![IronMQ - #2
cUrl Example
# Send a message to queue :
c u r l −i −H ” Content−Type : a p p l i c a t i o n / j s o n ” −H ”
A u t h o r i z a t i o n : OAuth {TOKEN}” −X POST −d ’{ ” messages ” : [ {
”body” : ” h e l l o world ! ” }]} ’ ” h t t p s ://mq−aws−us−east −1. i r o n
. i o /1/ p r o j e c t s /{PROJECT ID}/ queues / t e s t q u e u e / messages ”
# Get a message from queue :
c u r l −i −H ” Content−Type : a p p l i c a t i o n / j s o n ” −H ”
A u t h o r i z a t i o n : OAuth {TOKEN}” ” h t t p s ://mq−aws−us−east −1.
i r o n . i o /1/ p r o j e c t s /{PROJECT ID}/ queues / t e s t q u e u e /
messages ”
# E x t r a c t MESSAGE ID from JSON .
# A f t e r p r o c e s s i n g , remove from queue :
c u r l −i −H ” Content−Type : a p p l i c a t i o n / j s o n ” −H ”
A u t h o r i z a t i o n : OAuth {TOKEN}” −X DELETE ” h t t p s ://mq−aws−
us−east −1. i r o n . i o /1/ p r o j e c t s /{PROJECT ID}/ queues /
t e s t q u e u e / messages /{MESSAGE ID}”](https://image.slidesharecdn.com/ironmq-slides-160120124759/85/Ironmq-slides-15-320.jpg)
![IronMQ - #3
Webhooks - the Flexible Way
Send a message to an IronMQ queue from an application that can
make a simple HTTP POST call.
# Send a message to a queue v i a the webhook i n t e r f a c e :
c u r l −i −H ” Content−Type : a p p l i c a t i o n / j s o n ” −X POST −d ’{ ”
messages ” : [ { ”body” : ” h e l l o world ! ” }]} ’ ” h t t p s ://mq−
rackspace −lon . i r o n . i o /1/ p r o j e c t s /56996827428876000
a000060/ queues /Webhooks−from−Github−All −Hands −2016/
messages /webhook? oauth=lrBW053LN7c2sMTBhjYkJkt 123”](https://image.slidesharecdn.com/ironmq-slides-160120124759/85/Ironmq-slides-16-320.jpg)
![IronMQ - #3
Webhooks - the Flexible Way
Send a message to an IronMQ queue from an application that can
make a simple HTTP POST call.
# Send a message to a queue v i a the webhook i n t e r f a c e :
c u r l −i −H ” Content−Type : a p p l i c a t i o n / j s o n ” −X POST −d ’{ ”
messages ” : [ { ”body” : ” h e l l o world ! ” }]} ’ ” h t t p s ://mq−
rackspace −lon . i r o n . i o /1/ p r o j e c t s /56996827428876000
a000060/ queues /Webhooks−from−Github−All −Hands −2016/
messages /webhook? oauth=lrBW053LN7c2sMTBhjYkJkt 123”
Or the opposite: IronMQ queue will send (push) the message to any
HTTP address](https://image.slidesharecdn.com/ironmq-slides-160120124759/85/Ironmq-slides-17-320.jpg)
Ironmq slides
- 1. IronMQ, IronWorker and IronCache Iron.io Services in Action Mikko Koivunalho mikko.koivunalho@cxense.com Cxense All Hands Developer Conference Jan 20th, 2016 Oslo (30 minutes)
- 2. Who Am I Mikko Koivunalho Senior Developer Cxense Stockholm office Maintaining Cxense Display Provide Technical Support to CxDisplay Developing new GpsLocator Devops: Git Experiences in Integration Customers with integration system Finnish Defence Forces UPM (Paper Industry Supply Chain logistics) Kelisec (cryptology private key exchange) My interests Git hooks IPC, message queues Security Adventure Games Cooking
- 3. Hobbyist’s viewpoint! But the same principles and architectures apply to enterprise needs. Feel free to ask questions any time!
- 4. The Nature of Cloud Applications Building for Uncertainty The Cloud is everywhere and all the time. . . . . . but not reliably! In a ”one server-room” like environments we can reasonably assume that our ”contact partners”, i.e. the services we use, are always accessible and service is instantenuous. But not in the cloud. There we have no control over the services we use. APIs for Google Maps, Facebook, Github, Wikidata, OpenExchangeRates, MapMyFitness, Bloomberg, Sales Force, Wordpress, . . . They all provide valuable services for us, but as we know, sometimes they are down! . . . or just slow . . .
- 5. Our Problem We must: 1. Assume that we cannot establish contact with the other party whenever we need to. 2. Assume that the execution time and network speed are not constants but depend on uncontrollable factors. We must not: 1. Build applications which rely on the accessability of services outside our control! 2. Build tightly connected applications!
- 6. Solution? If not tightly-coupled applications then loosely-coupled! The ”tenets” of loose coupling: separation Strict separation of different functionality. transaction One task: Complete or fail. No between. connection No straight socket connections. messaging Asynchronous messages between different parts of the application.
- 7. Solution? If not tightly-coupled applications then loosely-coupled! The ”tenets” of loose coupling: separation Strict separation of different functionality. transaction One task: Complete or fail. No between. connection No straight socket connections. messaging Asynchronous messages between different parts of the application. messaging but no socket connections, . . .
- 8. Solution? If not tightly-coupled applications then loosely-coupled! The ”tenets” of loose coupling: separation Strict separation of different functionality. transaction One task: Complete or fail. No between. connection No straight socket connections. messaging Asynchronous messages between different parts of the application. messaging but no socket connections, . . . But that’s a message queue!
- 9. Solution? If not tightly-coupled applications then loosely-coupled! The ”tenets” of loose coupling: separation Strict separation of different functionality. transaction One task: Complete or fail. No between. connection No straight socket connections. messaging Asynchronous messages between different parts of the application. messaging but no socket connections, . . . But that’s a message queue! separation, transaction . . .
- 10. Solution? If not tightly-coupled applications then loosely-coupled! The ”tenets” of loose coupling: separation Strict separation of different functionality. transaction One task: Complete or fail. No between. connection No straight socket connections. messaging Asynchronous messages between different parts of the application. messaging but no socket connections, . . . But that’s a message queue! separation, transaction . . . . . . and a worker platform . . .
- 11. Interlude What the Heck’s a Message Queue Anyway? Reliable transmission The queue handles itself; it is its own program, outside the sending and/or receiving end. . . . (Except in ZeroMQ, but ZeroMQ is always a special case!) The receiving program will signal the message queue when it has processed the message, otherwise the message will return to the queue; so every message will be processed.
- 12. Interlude What the Heck’s a Worker Platform Anyway? The ”worker” is a program which only does one thing. You upload the workers to the platform. The platform runs the worker when scheduled or triggered. The (possible) parameters are provided independently via scheduling or triggering mechanism. The (possible) result is delivered via platform defined methods or via external services.
- 13. Iron.io Cloud Services IronMQ Message Queue IronCache Key-value Storage IronWorker Worker Queue (uploaded workers) Iaas, PaaS Main competitors: Heroku, Gigaspaces (high-end enterprise level)
- 14. IronMQ Message Queue in the Cloud - on Could’s Terms A message queue in the cloud, not tied to one server, not brokering messages between nodes. High availability. Delay message delivery when required. Monitoring performance and status via Web interface. REST API. Client libraries for many languages, including Python, Java, Ruby, Go, Perl and .Net. Usable via cUrl or using IronMQ command line tool.
- 15. IronMQ - #2 cUrl Example # Send a message to queue : c u r l −i −H ” Content−Type : a p p l i c a t i o n / j s o n ” −H ” A u t h o r i z a t i o n : OAuth {TOKEN}” −X POST −d ’{ ” messages ” : [ { ”body” : ” h e l l o world ! ” }]} ’ ” h t t p s ://mq−aws−us−east −1. i r o n . i o /1/ p r o j e c t s /{PROJECT ID}/ queues / t e s t q u e u e / messages ” # Get a message from queue : c u r l −i −H ” Content−Type : a p p l i c a t i o n / j s o n ” −H ” A u t h o r i z a t i o n : OAuth {TOKEN}” ” h t t p s ://mq−aws−us−east −1. i r o n . i o /1/ p r o j e c t s /{PROJECT ID}/ queues / t e s t q u e u e / messages ” # E x t r a c t MESSAGE ID from JSON . # A f t e r p r o c e s s i n g , remove from queue : c u r l −i −H ” Content−Type : a p p l i c a t i o n / j s o n ” −H ” A u t h o r i z a t i o n : OAuth {TOKEN}” −X DELETE ” h t t p s ://mq−aws− us−east −1. i r o n . i o /1/ p r o j e c t s /{PROJECT ID}/ queues / t e s t q u e u e / messages /{MESSAGE ID}”
- 16. IronMQ - #3 Webhooks - the Flexible Way Send a message to an IronMQ queue from an application that can make a simple HTTP POST call. # Send a message to a queue v i a the webhook i n t e r f a c e : c u r l −i −H ” Content−Type : a p p l i c a t i o n / j s o n ” −X POST −d ’{ ” messages ” : [ { ”body” : ” h e l l o world ! ” }]} ’ ” h t t p s ://mq− rackspace −lon . i r o n . i o /1/ p r o j e c t s /56996827428876000 a000060/ queues /Webhooks−from−Github−All −Hands −2016/ messages /webhook? oauth=lrBW053LN7c2sMTBhjYkJkt 123”
- 17. IronMQ - #3 Webhooks - the Flexible Way Send a message to an IronMQ queue from an application that can make a simple HTTP POST call. # Send a message to a queue v i a the webhook i n t e r f a c e : c u r l −i −H ” Content−Type : a p p l i c a t i o n / j s o n ” −X POST −d ’{ ” messages ” : [ { ”body” : ” h e l l o world ! ” }]} ’ ” h t t p s ://mq− rackspace −lon . i r o n . i o /1/ p r o j e c t s /56996827428876000 a000060/ queues /Webhooks−from−Github−All −Hands −2016/ messages /webhook? oauth=lrBW053LN7c2sMTBhjYkJkt 123” Or the opposite: IronMQ queue will send (push) the message to any HTTP address
- 18. IronWorker Worker Platform in the Cloud - on Could’s Terms Run your programs in a containerized environment. High availability. Automatic scalability. Parallel execution when necessary / by request, automatic. Scheduled according to clock or triggered by direct message or IronMQ message. Monitoring performance and status via Web interface. REST API. Client libraries for many languages, including Python, Java, Ruby, Go, Perl and .Net. Usable via cUrl or using Ironworker command line tool.
- 19. IronWorker - #2 Worker Platform in the Cloud - on Could’s Terms Isolate the heavy duty functions out of your program: Image processing Web requests ”credit card payment”, ”report generation”, ”statistical analysis”, . . . Package the code together with its dependencies and upload into IronWorker.
- 20. IronCache Key/Value Storage in the Cloud - on Could’s Terms Just A Simple Database. Key/value map. Only indexed by key. Value can be any string, conventionally JSON, or integer. Atomized commits, increment/decrement integer values. Used to support IronWorkers (e.g. during parallel runs) High availability. REST API. Client libraries for many languages. Usable via cUrl or using command line tool (Perl package IO::Iron::Applications).
- 21. The Cost Free for hobbyists: 1M API calls / month 5 simultaneous workers, 10 hours / month
- 22. We Have a Problem Iron.io Has a Solution? Two examples. Two ways to use/trigger IronWorker. No code examples, just architecture.
- 23. We Have a Problem Iron.io Has a Solution? The Problem I have a Github repo: https://github.com/mikkoi/All-Hands-2016 I have three buddies who need to know when I update the repo Buddy Local Wants to receive an SMS whenever there is a commit which starts with the word ”Admin” Buddy Remote Wants to receive an email for all the changes but only during weekdays Buddy Longway (of family Derib) Wants to receive at most one email a day, preferably early morning
- 24. Architecture Github IronMQ error queue IronCache IronWorker Main Filter Papertrail Logging Platform IronWorker SMS Gateway IronWorker Email Gateway IronWorker Email Gateway Webhook event ”push” Queue Push Message Error Queue for Push Messages Query Schedule Immediately Schedule Immediately Schedule Later
- 25. We Have a Problem Iron.io Has a Solution? The Problem 2 Automatic Romantic SMS Alarms Time is 17:30 and still working. ”Honey, I’m gonna be late for dinner.” Check if still working. Don’t send at precisely 17:30. Delay sending 5–45 minutes. Construct message from template and alternative parts.
- 27. Some Best Practices Configure IronWorker project to log everything (all stdout/stderr/error queues) to Papertrail (or similar service) or syslog. Create a dedicated queue for all incoming messages (Webhooks-from-Github-All-Hands-2016). Make the queue push queue: it ”pushes” the message to IronWorker. Push queues always with error queues. Monitor error queues with alarms! Prefer push queues with IronWorkers instead of workers waking up to query the queue.
- 28. Questions Questions? And maybe answers Blog: Exercises in Restful Integration and Continuous Delivery http://www.koivunalho.org/blogs/exercises-in-integration-and-delivery