RabbitMQ and AMQP with .net client library
- 2. Agenda • Message Broker • RabbitMQ Overview • RabbitMQ Concepts • AMQP • RabbitMQ Workfolw • Message Properties • Publishing Messages • Consuming Messages • Rejecting Messages • Controlling queues
- 4. Message Broker A message broker is an architectural pattern for message validation, transformation, and routing. It mediates communication among applications, minimizing the mutual awareness that applications should have of each other in order to be able to exchange messages, effectively implementing decoupling.
- 5. Message Broker Actions • Route messages to one or more destinations • Transform messages to an alternative representation • Perform message aggregation, decomposing messages into multiple messages and sending them to their destination, then recomposing the responses into one message to return to the user • Interact with an external repository to augment a message or store it • Invoke web services to retrieve data • Respond to events or errors • Provide content and topic-based message routing using the publish– subscribe pattern
- 6. List of message broker software • Amazon Web Services (AWS) Simple Queue Service (SQS) • Apache ActiveMQ • Apache Kafka • Apache Qpid • Celery • Cloverleaf (E-Novation Lifeline) • Comverse Message Broker (Comverse Technology) • Enduro/X Transactional Message Queue (TMQ) • Financial Fusion Message Broker (Sybase) • Fuse Message Broker (enterprise ActiveMQ) • Gearman
- 7. List of message broker software Cont. • HornetQ (Red Hat) • IBM Integration Bus • IBM Message Queues / IBM WebSphere MQ • JBoss Messaging (JBoss) • JORAM • Microsoft Azure Service Bus (Microsoft) • Microsoft BizTalk Server (Microsoft) • NATS (MIT Open Source License, written in Go) • Open Message Queue
- 8. List of message broker software Cont. • Oracle Message Broker (Oracle Corporation) • RabbitMQ (Mozilla Public License, written in Erlang) • Redis An open source, in-memory data structure store, used as a database, cache and message broker. • SAP PI (SAP AG) • Solace PubSub+ • Spread Toolkit • Tarantool, a NoSQL database, with a set of stored procedures for message queues • TIBCO Enterprise Message Service • WSO2 Message Broker
- 10. RabbitMQ RabbitMQ is an open source message broker written in Erlang, currently under the wing of Pivotal Software. It’s based around the AMQP open protocol, with official client libraries in Java, .NET, Erlang, as well as libraries for most other popular programming languages.
- 11. RabbitMQ’s features and benefit • Open source • Platform and vendor neutral • Lightweight • Client libraries for most modern languages • Flexibility in controlling messaging trade-offs— • Plugins for higher-latency environments— • Third-party plugins • Layers of security
- 12. RabbitMQ and Erlang It was written in Erlang, the telco-grade, functional programming language designed at the Ericsson Computer Science Laboratory in the mid-to-late 1980s. Erlang was designed to be a distributed, fault-tolerant, soft real-time system for applications that require 99.999% uptime. As a language and runtime system, Erlang focuses on lightweight processes that pass messages between each other, providing a high level of concurrency with no shared state.
- 13. Message-oriented middleware (MOM) Software or hardware infrastructure that allows for the sending and receiving of messages from distributed systems. RabbitMQ fills this role handily with functionality that provides advanced routing and message distribution, even with wide area network (WAN) tolerances to support reliable, distributed systems that interconnect with other systems easily.
- 14. Loosely Coupled ArchitectureHigh Coupled Architecture Loosely Coupled Architecture
- 15. Loosely Coupled ArchitectureHigh Coupled Architecture Loosely Coupled Architecture
- 19. Message Queuing model • Exchange—The component of the message broker that routes messages to queues • Queue—A data structure on disk or in memory that stores messages • Binding—A rule that tells the exchange which queue the messages should be stored in
- 20. Messaging Queue Model Routing Message Queuing model
- 23. RabbitMQ Concepts • Producer: Application that sends the messages. • Consumer: Application that receives the messages. • Queue: Buffer that stores messages. • Message: Data that is sent from the producer to a consumer through RabbitMQ. • Connection: A connection is a TCP connection between your application and the RabbitMQ broker. • Channel: A channel is a virtual connection inside a connection. When you are publishing or consuming messages or subscribing to a queue, it's all done over a channel.
- 24. RabbitMQ Concepts Cont. • Exchange: Receives messages from producers and pushes them to queues depending on rules defined by the exchange type. A queue needs to be bound to at least one exchange to be able to receive messages. • Binding: A binding is a link between a queue and an exchange. • Routing key: The routing key is a key that the exchange looks at to decide how to route the message to queues. You can think of the routing key as the destination address of a message.
- 25. RabbitMQ Concepts Cont. • AMQP: The Advanced Message Queuing Protocol is the primary protocol used by RabbitMQ for messaging. • Users: It's possible to connect to RabbitMQ with a given username and password. Every user can be assigned permissions such as rights to read, write and configure privileges within the instance. Users can also be assigned permissions to specific virtual hosts. • Vhost, virtual host: A virtual host provides a way to segregate applications that are using the same RabbitMQ instance. Different users can have different access privileges to different vhosts and queues, and exchanges can be created so that they only exist in one vhost.
- 26. RabbitMQ Concepts Cont. • Acknowledgments and Confirms: Acknowledgments and confirms indicate that messages have been received or acted upon. Acknowledgments can be used in both directions; A consumer can indicate to the server that it has received/processed a message, and the server could report
- 27. AMQP Advanced Message Queuing Protocol
- 29. AMQP as an RPC transport • Kicking off the conversation • The Client send a protocol header • The server send Connection.Start command • The client responds with the Connection.StartOK frame • Tuning in to the right channel
- 30. AMQP commands • AMQP uses classes and methods, referred to as AMQP commands, to create a common language between clients and servers. The classes in AMQP define a scope of functionality, and each class contains methods that perform different tasks. (Class)Connection.Start(Method)
- 31. AMQP frame components • When commands are sent to and from RabbitMQ, all of the arguments required to execute them are encapsulated in data structures called frames that encode the data for transmission.
- 33. AMQP frame components • Frame type • Channel number • Frame size in bytes • Frame payload • End-byte marker (ASCII value 206) Frame Header
- 34. Types of frames • The protocol header frame is only used once, when connecting to RabbitMQ. • A method frame carries with it the RPC request or response that’s being sent to or received from RabbitMQ. • A content header frame contains the size and properties for a message. • Body frames contain the content of messages. • The heartbeat frame is sent to and from RabbitMQ as a check to ensure that both sides of the connection are available and working properly.
- 35. Method Frame Header Frame Body Frame Types of frames
- 39. RabbitMQ Workfolw Publish and Consume message
- 40. 1. The producer publishes a message to an exchange. When you create the exchange, you have to specify the type of it. The different types of exchanges are explained in detail later on. 2. The exchange receives the message and is now responsible for the routing of the message. The exchange looks at different message attributes and keys depending on the exchange type. 3. In this case, we see two bindings to two different queues from the exchange. The exchange routes the message to the correct queue, depending on its attributes. 4. The messages stay in the queue until a consumer handles them. 5. The consumer handles the message, thus removing it from the queue. The message flow in RabbitMQ
- 41. Publish-Consumer Steps • Declaring an exchange • Declaring a queue • Binding a queue to an exchange • Publishing a message to RabbitMQ • Consuming messages from RabbitMQ
- 42. Declaring an exchange • Exchanges are AMQP entities where messages are sent. Exchanges take a message and route it into zero or more queues. The routing algorithm used depends on the exchange type and rules called bindings. AMQP 0-9-1 brokers provide four exchange types: • Direct : delivers messages to queues based on a message routing key. In a direct exchange, the message is routed to the queue whose binding key exactly matches the routing key of the message. • Fanout : routes messages to all of the queues that are bound to it. • Topic : performs a wildcard match between the routing key and the routing pattern specified in the binding. • Headers : use the message header attributes to do their routing.
- 43. Declaring a queue • They store messages that are consumed by applications. Queues share some properties with exchanges, but also have some additional properties: • Name • Durable : A durable queue ensures that RabbitMQ never loses the queue • Message TTL: The time a message published to a queue can live before it's discarded.Auto- delete • Auto-expire: The time a queue can be unused before it's automatically deleted. • Max length: How many (ready) messages a queue can hold before it starts to drop them. • Max length bytes: The total body size of ready messages a queue can contain before it starts to • drop them.
- 44. Binding a queue to an exchange • Once the exchange and queue have been created, it’s time to bind them together.
- 45. Publishing a message to RabbitMQ
- 46. Consuming messages from RabbitMQ • To consume messages from a queue in RabbitMQ, a consumer application subscribes to the queue in RabbitMQ by issuing a Basic.Consume command. • The server will respond with Basic.ConsumeOk • The consumer will start receiving messages in the unsurprising form of Basic.Deliver methods and their content header and body frame counterparts
- 47. Writing a message publisher in .Net
- 48. Getting messages from RabbitMQ in .Net
- 51. Message Properties in Header Frame
- 52. Property Type For use by Suggested or specified use content-type short-string Application Specify the type of the message body using mime-types. content-encoding short-string Application Specify whether your message body is encoded in some special way, such as zlib, deflate, or Base64. Message-id/Correlation-id : enables the message to carry data in the header that uniquely identifies it as it flows through the various components in a loosely coupled system. correlation-id short-string Application If the message is in reference to some other message or uniquely identifiable item, the correlation-id is a good way to indicate what the message is referencing. message-id short-string Application A unique identifier such as a UUID that your application can use to identify the message. timestamp timestamp Application An epoch or Unix timestamp value that can be used to indicate when the message was created. Expiration : In declaring a queue, you can pass an x-message-ttl argument along with the queue definition. expiration short-string Application An epoch or Unix timestamp value as a text string that indicates when the message should expire. Message Properties
- 53. Property Type For use by Suggested or specified use delivery-mode octet RabbitMQ A value of 1 tells RabbitMQ it can keep the message in memory; 2 indicates it should also write it to disk. app-id short-string Application Useful for defining the application publishing the messages. user-id short-string Application A free-form string that, if used, RabbitMQ will validate against the connected user and drop messages if they don’t match. type short-string Application A text string your application can use to describe the message type or payload. reply-to short-string Application Can be used to carry a queu name or the routing key a consumer should use when replying to a message implementing an RPC pattern. headers table Both A free-form key/value table that you can use to add additional metadata about your message; RabbitMQ can route based upon this if desired. priority octet RabbitMQ A property for priority ordering in queues. Message Properties Cont.
- 55. Publishing message facets Performance Reliability
- 56. balance between high performance and message safety • How important is it that messages are guaranteed to be en-queued when published? • Should a message be returned to a publisher if it can’t be routed? • If a message can’t be routed, should it be sent somewhere else where it can later be reconciled? • Is it okay if messages are lost when a RabbitMQ server crashes? • Should RabbitMQ confirm that it has performed all requested routing and persistence tasks to a publisher when it processes a new message? • Should a publisher be able to batch message deliveries and then receive confirmation from RabbitMQ that all requested routing and persistence tasks have been applied to all of the messages in the batch? • If you’re batching the publishing of messages that require confirmation of routing and persistence, is there a need for true atomic commits to the destination queues for a message? • Are there acceptable trade-offs in reliable delivery that your publishers can use to achieve higher performance and message throughput? • What other aspects of message publishing will impact message throughput and performance?
- 57. RabbitMQ won’t accept non-routable messages with mandatory set
- 58. Publisher Confirms as a lightweight alternative to transactions
- 59. Using alternate exchanges for unroutable messages
- 60. Batch processing with transactions
- 61. Surviving node failures with HA queues
- 62. Persisting messages to disk via delivery-mode 2 In addition to delivery-mode of 2, for messages to truly survive a restart of a RabbitMQ broker, your queues must be declared as durable when they’re created. delivery-mode: 1 delivery-mode: 2
- 64. Basic.Get Get is acting as Pull message from the queue if there is any messages on it If There is a messages If There is not any messages
- 65. Basic.Consume Consume is acting as push message from the queue to the consumer
- 67. With no-Ack if the socket buffer is not full and the rabbitmq can write to it the rabbitmq will write the new messages but if the ack is required the rabbitmq will not write a new message until the consumer send ack Using no-ack mode for faster throughput
- 68. Controlling consumer prefetching via quality of service settings AMQP specifies the basic.qos method to allow you to limit the number of unacknowledged messages on a channel (or connection) when consuming (aka "prefetch count"). Simple benchmark results for consuming with no QoS set and different prefetch count values
- 69. Using transactions with consumers Transactions don’t work for consumers with acknowledgments disabled. Message velocities when using transactions compared to non- transactional message velocities
- 71. Rejecting messagesRejecting Message Basic.Reject Basic.Nack Dead Letter Exhange
- 72. A consumer can acknowledge, reject, or negatively acknowledge a message. Basic.Nack allows for multiple messages to be rejected at once, whereas Basic.Reject allows just one message to be rejected at a time.
- 73. A rejected message can be routed as a dead-letter message through another exchange. Dead Letter Exchange • The message is rejected (basic.reject or basic.nack) with requeue=false, • The TTL for the message expires; or • The queue length limit is exceeded. If the dead letter exchange is missing then, the messages will be silently dropped.
- 75. Temporary queues • Automatically deleting queues • the queue will only delete itself when there are no more consumers listening to it. • Allowing only a single consumer • An exclusive queue will also automatically be deleted when the channel that the queue was created on is closed • Automatically expiring queues • RabbitMQ will delete the queue if it has gone unused for some length of time.
- 76. Permanent queues • Queue durability • When declaring a queue that should persist across server restarts. • Auto-expiration of messages in a queue • The stale data that should be removed after its usefulness has expired • Maximum length queues • once the queue messages reaches the maximum size, RabbitMQ will drop messages from the front of the queue as new messages are added.
- 77. Queue settings Argument name Purpose x-dead-letter-exchange An exchange to which non-requeued rejected messages are routed x-dead-letter-routing-key An optional routing key for dead-lettered messages x-expires Queue is removed after the specified number of milliseconds x-ha-policy When creating HA queues, specifies the mode for enforcing HA across nodes x-ha-nodes The nodes that an HA queue is distributed across x-max-length The maximum message count for a queue x-message-ttl Message expiration in milliseconds, enforced at the queue level x-max-priority Enables priority sorting of a queue with a maximum priority value of 255
- 78. Thanks