ITB2019 Multi-language / multi-OS communication using RabbitMQ - Wil de Bruin
1 like283 views
The document discusses the implementation of multi-language and multi-OS communication using RabbitMQ, led by Wil de Bruin, a software/system engineer. It covers various topics including messaging concepts, RabbitMQ setup, and use cases such as a web hosting panel with various integrations. Key challenges include limitations in API management, messaging configurations, and the complexity of setting up RabbitMQ consumers with ColdFusion.
![Commands and responses
COMMAND
Headers:
action = string (for example: archive of getmailboxsize)
Properties:
correlation_id: string
content_type: string (application/json of text/plain)
Body: [string,json]
{
"params": {
"domainname": "example.com",
"param2": "Second parameter"
}
}
Response
Properties:
correlation_id: string
content_type: string (application/json)
Body: json
{
data: [string, json]
status: [ok,warning,error]
messages: string
}](https://image.slidesharecdn.com/itb2019-multilanguageoscomm-wildebruin-190515160745/85/ITB2019-Multi-language-multi-OS-communication-using-RabbitMQ-Wil-de-Bruin-29-320.jpg)
![Flexible messaging daemon for hosting servers
• RabbitMQ Consumer written in Python
• Executes any command based on configuration, e.g:
[mysite4u.sharedmail.cmd:archive]
script: /usr/local/scripts/archiveMail/archiveMailDomain.php $domainname
run-as: vmail
reply-to: mysite4u.sharedmail.response
• Output of script is piped into RabbitMQ producer response.](https://image.slidesharecdn.com/itb2019-multilanguageoscomm-wildebruin-190515160745/85/ITB2019-Multi-language-multi-OS-communication-using-RabbitMQ-Wil-de-Bruin-34-320.jpg)
ITB2019 Multi-language / multi-OS communication using RabbitMQ - Wil de Bruin
- 1. Multi-language / multi-OS commmunication using RabbitMQ Wil de Bruin
- 2. Who am I • Wil de Bruin – Software / system engineer • Wageningen,The Netherlands • Graduated in Environmental Sciences • Reseach microbiologist (1987) • CTO – Founder Site4u (1994) • CFML since Allaire ColdFusion 1.5 • Coldfusion training, consultancy, software development and hosting • Coldbox user since 2008
- 3. Agenda • Use case:Web Hosting Panel • RPC vs messaging • RabbitMQ Concepts explained • Our RabbitMQ messaging setup • Demo / code • Conclusions
- 4. Use case: Web hosting panel
- 5. Web hosting panel • Dns, Email functionality: just like Cpanel, plesk etc • Customers demanding SLA reports: integration with Monitoring • Resource usage:VMWare integration • Management of servers and networks: complex networks require more automation
- 6. Server 700… Server 2 VueJS frontend Portal DB MS SQL CF Rest API backend Redirects mySQL Accounting MS SQL Domain registrar XML API Domain registrar (no CFML) API Antispam service ( API ) Nagios monitoring API ?? Mail Server Mail DB /var/vmail/domain DNS Server powerDNS Commandline Commandline Zabbix trend monitoring API ?? Server 1 Commandline
- 7. Some limitations: Mailserver • Mail Server is OpenSource (postfix / Dovecot (IMAP) / webmail (Roundcube) • User config is defined in a database • User mailboxes as files and directories.Will NOT be deleted when removed from database. • Roundcube in separate database • Report sizing with commandline tools, noAPI available. • Environment: Linux, bash and Python
- 8. More limitations: DNS and DNSsec • PowerDNS has mySQL backend, but no management of DNSsec • Workflow • Create DNSsec in powerDNS (commandline) • Send key to registrar • Registrar 1: XML API, several languages • Registrar 2: API for CFML is hard, several other language libs available (e.g PHP) • Change status in control panel • Environment: Linux, bash, commandline,
- 9. Still more limitations: SSL and HTTP configs • SSL installation is manual, customers need overview (commandline) • Mixture of • Custom certificates from several providers • Let’s encrypt: automated management, but how to notify our control panel? • Environment: • Linux, bash, commandline, python • Windows, powershell
- 10. RPC vs messaging
- 11. RPC and REST limitations • Only one receiver • Sender knows about receiver • Receiver knows about sender • Blocks request usage
- 12. RPC
- 13. RPC
- 15. Messaging advantages • Cross platform, multiple technologies: Flexibility • Decoupling Producers and Consumers • Event driven programming: scalability • Queuing • Load balancing
- 16. Rabbit MQ concepts explained
- 17. AMQP Advanced Message Queuing Protocol (www.amqp.org) • Binary protocol • Diverse programming languages can communicate • Queuing • Routing • Exchanges • Secure • Reliable
- 18. RabbitMQ • Multiple messaging protocols (AMQP, STOMP, MQTT) • Open source, well known • Lots of libraries in different languages (Java, .NET, Python,Ruby, Javascript) • Http management • Simple concept: just accepts and forwards messages • Developer friendly: deploy with Docker, Puppet and more
- 19. RabbitMQ messaging concepts (1/5)
- 20. RabbitMQ messaging concepts (2/5)
- 21. RabbitMQ messaging concepts (3/5)
- 22. RabbitMQ messaging concepts (4/5)
- 23. RabbitMQ messaging concepts (5/5)
- 24. Our RabbitMQ messaging setup
- 25. Sending commands in our hosting environment CF control panel Mail server Delete Domain Get Mailbox Size Return Result CF control panel Dns server Enable DnsSec Return Result CF control panel ANY server List SSL certificates Return Result
- 27. Messaging
- 28. Our messaging setup • Topic exchange • Routing keys: • mysite4u.servicename.cmd for commands to a service • mysite4u.servicename.response for responses of a service • Queues: servicename-commands and mysite4u-responses • Exchange: mySite4u-topic • Queues and messages are DURABLE i.e persistent.
- 29. Commands and responses COMMAND Headers: action = string (for example: archive of getmailboxsize) Properties: correlation_id: string content_type: string (application/json of text/plain) Body: [string,json] { "params": { "domainname": "example.com", "param2": "Second parameter" } } Response Properties: correlation_id: string content_type: string (application/json) Body: json { data: [string, json] status: [ok,warning,error] messages: string }
- 31. Setup • CF Rabbit Producer (hosting control panel) • RabbitMQ consumer on hosting servers • RabbitMQ producer on hosting servers • CF Rabbit Consumer • Polling, the easy way • Polling at higher frequency, still easy • Pushing messages (a.k.a the hard way)
- 32. CFML RabbitMQ producer • Let’s start with • https://github.com/lmajano/messaging-polyglot • https://www.rabbitmq.com/getstarted.html RabbitMQ tutorials • https://www.rabbitmq.com/api-guide.html Java API Guide • Create cfproject, download jar and dump in /lib folder • Create mappings in wirebox • Add rabbitsettings in coldbox.conf • Create onAppInit to establish RabbitMQ connection • Create a Service to send messages
- 34. Flexible messaging daemon for hosting servers • RabbitMQ Consumer written in Python • Executes any command based on configuration, e.g: [mysite4u.sharedmail.cmd:archive] script: /usr/local/scripts/archiveMail/archiveMailDomain.php $domainname run-as: vmail reply-to: mysite4u.sharedmail.response • Output of script is piped into RabbitMQ producer response.
- 36. CF RabbitMQ consumer • Pulling vs pushing • Pushing should be easy (just like other languages….) • Pushing more flexible • Missing correct examples! • Pulling is easy • 1 minute limitation • CFConcurrent to the rescue..
- 37. What’s missing • Security: • Use SSL in rabbit connections • Make sure your reply is a valid one: use correlationID for command and match this in handling the reply • Monitoring • Make sure your commands are processed • Monitor your queue lengths
- 38. Conclusions • Messaging is very flexible, new ways of communication • No more coding barriers, many languages can be used • CF rabbitMQ producers easy to achieve • CF rabbitMQ consumers are hard to configure, due to language limitations • Is CF the best tool for a rabbit Consumer in a micro services architecture? • https://github.com/wpdebruin/rabbitmq-demo
- 39. Thank you! Questions? More questions? E-mail: wil@site4u.nl Slack: @wil-site4u