Stream Queues: Filtering.
Gabriele Santomaggio
Distributed Systems | RabbitMQ Team Member | Open Source Addicted
Okay, we spoke about RabbitMQ Streams and Super Streams. Let's see another interesting feature:
Before starting, we need to know some internals
The stream server sends chunks to the clients according to the protocol. Each chunk can contain one or more messages, like:
The server is unaware of the message's context. The client parses the chunk and decodes the messages.
How does the filter work?
RabbitMQ Stream uses Bloom filter to filter the chunks.
The filter is for chunks, which means that the client could receive more data. Given the following example, where there are three chunks:
one: only with messages with the state "NY"
two: with states NY and Alabama"
three: only with messages with the state "Alabama"
Case 1: If the user filters for messages with "Alabama", the server will send chunks 2 and 3.
Case 2: If the user filters for "NY", the server will send chunks 1 and 2.
The filter is server-side. Post-filtering is performed on the client side and removes messages that don't match the filter.
The filter can significantly reduce traffic and improve application performance.
Code
On the client side, it is enough to define the filter for the producer:
Send the messages:
Consumer side:
Server-side filter
Client-side filter
Easy, right 😎?
Feedback
A customer with several streams and many, many consumers says:
"We dropped the download time from about one hour to seconds just by enabling the filter"
I can't share more details about this use case, but it is interesting since it involves streams, super stream, single active consumer and stream filtering. Let's see in the future.
Conclusions
Filtering can speed up your application, and it is easy to use. I used the .NET client, but you can pick your preferred language 😀
Please see also:
[Doc] Stream filtering
Need help? Refer to https://www.rabbitmq.com/contact#consulting for information on community and commercial support.
Speak Italian? Here is a Telegram channel.
Happy RabbitMQ messaging! 🎉