Data integration with rabbit mq and celery
- 1. Data Integration with RabbitMQ and Celery Guaranteed Delivery @edytarcio
- 2. Message Styles - Recap Scp. RSync. Database replication. Rest/Soap. RPC. Messages patterns
- 3. What is RabbitMQ RabbitMQ Broker/ Message Server Erlang AMQP Language Agnostic Celery ● Dynamic high level lib ● Synchronous/Asynchronous task queue processing ● Scheduling tasks ● Monitoring based ● Python ● RabbitMQ client ● It’s also a server
- 4. Some RabbitMQ Features No message loss Persistent messages Publish confirms Message acknowledgment Mirrored queues Message priority Dead letter exchange Alternate exchange Messages and Queues TTLs Federation
- 6. Message flow
- 7. Messages Patterns - Publish/Subscribe
- 8. Messages Patterns - Request/Reply
- 9. Topologies - We started with Canonical Data Models Do not go for “fully connected”, Canonical approach is best!
- 10. What exactly are we trying to achieve? Low latency. The message shouldn’t take long to get to its destination. Leaky buffer control. Acknowledgement. Resilient system. Able to recover from failures. You don’t lose data. Connection failure. When the server is offline ● Fault tolerance. Celery Retry/ Dead Letter
- 11. Refining RabbitMq Model... High Availability Clustering. Load balancer. Which way to go?
- 12. Federation & Shovel for rescue … but how do they help us?
- 13. That’s how Federation/Shovel helps us
- 14. Distributed RabbitMq using Federation/Shovel
- 15. Anatomy of a message Numbers max msg publish/s 149910 max msg consume/s 64315 To conclude...
- 16. Thanks! https://www.rabbitmq.com/ http://zeromq.org/ http://www.celeryproject.org/ https://derickbailey.com/email-courses/rabbitmq-patterns-for-applications/ http://www.enterpriseintegrationpatterns.com/ https://www.google.com.br/ Pictures were taken from the WEB! edytarcio.silva@armazemparaiba.com.br
- 17. Wait there’s more... Brokerless ● Peer-to-peer ● Multi-connection