RabbitMQ Data Ingestion at Craft Conf
9 likes3,628 views
This document discusses building a distributed data ingestion system using RabbitMQ. It begins with an introduction to RabbitMQ and its key features like being multi-protocol, open source, polyglot and written in Erlang. It then discusses the problem of distributing data across multiple servers. RabbitMQ federation is proposed as a solution, allowing replication across servers. Federation uses queues and exchanges to replicate data and can be configured using parameters and policies. The document also discusses scaling the system using sharded queues and federated queues to load balance consumers.
RabbitMQ Data Ingestion at Craft Conf
- 1. Building a Distributed Data Ingestion System with RabbitMQ Alvaro Videla - RabbitMQ
- 2. Alvaro Videla • Developer Advocate at Pivotal / RabbitMQ! • Co-Author of RabbitMQ in Action! • Creator of the RabbitMQ Simulator! • Blogs about RabbitMQ Internals: http://videlalvaro.github.io/internals.html! • @old_sound — alvaro@rabbitmq.com — github.com/videlalvaro
- 3. About Me Co-authored! ! RabbitMQ in Action! http://bit.ly/rabbitmq
- 4. About this Talk • Exploratory Talk • A ‘what could be done’ talk instead of ‘this is how you do it’
- 5. Agenda • Intro to RabbitMQ • The Problem • Solution Proposal • Improvements
- 6. Intermission
- 9. The Hungarian Connection • I’m from Uruguay
- 10. The Hungarian Connection • I’m from Uruguay • I live in Switzerland
- 11. The Hungarian Connection • I’m from Uruguay • I live in Switzerland • I’m in Hungary right now
- 12. WHAT?
- 14. The Hungarian Connection Switzerland World Cup - 1954
- 15. The Hungarian Connection Switzerland World Cup - 1954
- 16. The Hungarian Connection Switzerland World Cup - 1954 Hungary 4 - Uruguay 2
- 17. The Hungarian Connection II Tivadar Puskás Telephone Switch Inventor
- 18. The Hungarian Connection III Old Hungarian Alphabet
- 19. The Hungarian Connection III Erlang
- 20. The Hungarian Connection III As Neumann János Lajos said
- 21. The Hungarian Connection III As Neumann János Lajos said Use Erlang
- 22. What is RabbitMQ
- 23. RabbitMQ
- 24. RabbitMQ
- 25. RabbitMQ • Multi Protocol Messaging Server
- 26. • Multi Protocol Messaging Server! • Open Source (MPL) RabbitMQ
- 27. • Multi Protocol Messaging Server! • Open Source (MPL)! • Polyglot RabbitMQ
- 28. • Multi Protocol Messaging Server! • Open Source (MPL)! • Polyglot! • Written in Erlang/OTP RabbitMQ
- 31. Polyglot
- 32. Polyglot
- 33. Polyglot • Java
- 35. Polyglot • Java! • node.js! • Erlang
- 36. Polyglot • Java! • node.js! • Erlang! • PHP
- 37. Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby
- 38. Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby! • .Net
- 39. Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby! • .Net! • Haskell
- 41. Some users of RabbitMQ
- 42. • Instagram Some users of RabbitMQ
- 43. • Instagram! • Indeed.com Some users of RabbitMQ
- 44. • Instagram! • Indeed.com! • Telefonica Some users of RabbitMQ
- 45. • Instagram! • Indeed.com! • Telefonica! • Mercado Libre Some users of RabbitMQ
- 46. • Instagram! • Indeed.com! • Telefonica! • Mercado Libre! • NHS Some users of RabbitMQ
- 47. • Instagram! • Indeed.com! • Telefonica! • Mercado Libre! • NHS! • Mozilla Some users of RabbitMQ
- 48. The NewYork Times on RabbitMQ This architecture - Fabrik - has dozens of RabbitMQ instances spread across 6 AWS zones in Oregon and Dublin. Upon launch today, the system autoscaled to ~500,000 users. Connection times remained flat at ~200ms. http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2014-January/032943.html
- 50. Messaging with RabbitMQ A demo with the RabbitMQ Simulator https://github.com/RabbitMQSimulator/RabbitMQSimulator
- 53. The Problem
- 56. ConnectionFactory factory = new ConnectionFactory(); factory.setHost("localhost"); ! Connection connection = factory.newConnection(); ! Channel channel = connection.createChannel(); Data Producer Obtain a Channel
- 57. channel.exchangeDeclare(EXCHANGE_NAME, "direct", true); Data Producer Declare an Exchange
- 58. String message = "Hello Federation!"; channel.basicPublish(EXCHANGE_NAME, "", null, message.getBytes()); Data Producer Publish a message
- 59. ConnectionFactory factory = new ConnectionFactory(); factory.setHost("localhost"); ! Connection connection = factory.newConnection(); ! Channel channel = connection.createChannel(); Data Consumer Obtain a Channel
- 60. channel.queueDeclare(QUEUE_NAME, true, false, false, null); channel.exchangeDeclare(EXCHANGE_NAME, "direct", true); channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, ""); Data Consumer Declare Queue and bind it
- 61. QueueingConsumer consumer = new QueueingConsumer(channel); channel.basicConsume(QUEUE_NAME, false, consumer); Data Consumer Start a consumer
- 62. while (true) { QueueingConsumer.Delivery delivery = consumer.nextDelivery(); String message = new String(delivery.getBody()); System.out.println("Received '" + message + "'"); ! channel.basicAck( delivery.getEnvelope(). getDeliveryTag(), false); } Data Consumer Process messages
- 63. Ad-hoc solution
- 64. A process that replicates data to the remote server
- 65. Possible issues • Remote server is offline • Prevent unbounded local buffers • Prevent message loss • Prevent unnecessary message replication • No need for those messages on remote server • Messages that became stale
- 66. Can we do better?
- 68. RabbitMQ Federation • Supports replication across different administrative domains • Supports mix of Erlang and RabbitMQ versions • Supports Network Partitions • Specificity - not everything has to be federated
- 72. RabbitMQ Federation • It’s a RabbitMQ Plugin
- 73. RabbitMQ Federation • It’s a RabbitMQ Plugin • Internally uses Queues and Exchanges Decorators
- 74. RabbitMQ Federation • It’s a RabbitMQ Plugin • Internally uses Queues and Exchanges Decorators • Managed using Parameters and Policies
- 75. Enabling the Plugin rabbitmq-plugins enable rabbitmq_federation
- 76. Enabling the Plugin rabbitmq-plugins enable rabbitmq_federation rabbitmq-plugins enable rabbitmq_federation_management
- 77. Federating an Exchange rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}'
- 78. Federating an Exchange rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}' ! rabbitmqctl set_policy --apply-to exchanges federate-me "^amq." '{"federation-upstream-set":"all"}'
- 81. Config Options rabbitmqctl set_parameter federation-upstream name ‘json-object’
- 82. Config Options rabbitmqctl set_parameter federation-upstream name ‘json-object’ ! json-object: { ‘uri’: ‘amqp://server-name/’, ‘prefetch-count’: 1000, ‘reconnect-delay’: 1, ‘ack-mode’: on-confirm } http://www.rabbitmq.com/federation-reference.html
- 83. Prevent unbound buffers expires: N // ms. message-ttl: N // ms.
- 85. Speed vs No Message Loss ack-mode: on-confirm ack-mode: on-publish ack-mode: no-ack
- 86. AMQP URI: amqp://user:pass@host:10000/vhost http://www.rabbitmq.com/uri-spec.html
- 87. Config can be applied via • CLI using rabbitmqctl • HTTP API • RabbitMQ Management Interface
- 90. The Problem
- 91. The Problem • Queues contents live in the node where the Queue was declared
- 92. The Problem • Queues contents live in the node where the Queue was declared • A cluster can access the queue from every connected node
- 93. The Problem • Queues contents live in the node where the Queue was declared • A cluster can access the queue from every connected node • Queues are an Erlang process (tied to one core)
- 94. The Problem • Queues contents live in the node where the Queue was declared • A cluster can access the queue from every connected node • Queues are an Erlang process (tied to one core) • Adding more nodes doesn’t really help
- 97. Pieces of the Puzzle • modulo hash exchange (consistent hash works as well) • good ol’ queues
- 98. Sharded Queues
- 99. Sharded Queues
- 100. Sharded Queues
- 101. Sharded Queues
- 102. Sharded Queues • Declare Queues with name: nodename.queuename.index
- 103. Sharded Queues • Declare Queues with name: nodename.queuename.index • Bind the queues to a consistent hash exchange
- 104. Sharded Queues • Declare Queues with name: nodename.queuename.index • Bind the queues to a partitioner exchange • Transparent to the consumer (virtual queue name)
- 105. We need more scale!
- 106. Federated Queues
- 107. Federated Queues • Load-balance messages across federated queues • Only moves messages when needed
- 108. Federating a Queue rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}'
- 109. Federating a Queue rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}' ! rabbitmqctl set_policy --apply-to queues federate-me "^images." '{"federation-upstream-set":"all"}'
- 110. With RabbitMQ we can
- 111. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP
- 112. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using Federation
- 113. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using Federation • Scale up using Sharding
- 114. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using Federation • Scale up using Sharding • Load balance consumers with Federated Queues
- 115. Credits world map: wikipedia.org federation diagrams: rabbitmq.com
- 116. Questions?