Redis v5 & Streams
2 likes1,035 views
The document details the features and functionalities introduced in Redis v5, which emphasizes the new Streams data structure and its capabilities for handling real-time data, particularly in distributed systems. It discusses the anatomy of streams, message delivery semantics, and the API for producers and consumers, including consumer groups for managing message processing. The document also highlights the problems with traditional methods and the necessity of streams in modern application architectures.
![XADD key [MAXLEN ~ n] <ID | *>
<field> <string> [field string…]
> XADD mystream * foo bar baz qaz
1532556560197-42
Time complexity: O(log n)
See https://redis.io/commands/xadd
The Redis Stream Producer API](https://image.slidesharecdn.com/201810redistlvmeetup-v5streams-181028190617/85/Redis-v5-Streams-29-320.jpg)
![XLEN key - does exactly that, not very interesting.
X[REV]RANGE key <start | -> < end | +>
[COUNT count] - much more interesting :)
Get a single entry (start = end = ID)
SCAN-like iteration on a stream (IDs inc.), but better
Range (timeframe) queries on a stream
What's in the stream "API"](https://image.slidesharecdn.com/201810redistlvmeetup-v5streams-181028190617/85/Redis-v5-Streams-33-320.jpg)
![Yes. No. Maybe.
It can be used for consuming, but that requires the
client constantly polling the stream for new entries.
So generally, no. There's something better for
consumers.
Is X[REV]RANGE the Consumer API?](https://image.slidesharecdn.com/201810redistlvmeetup-v5streams-181028190617/85/Redis-v5-Streams-35-320.jpg)
![XREAD [COUNT count]
STREAMS key [key ...] ID [ID ...]
Somewhat like X[REV]RANGE, but:
â—Ź Supports multiple streams
â—Ź Easier to consume from an offset onwards
(compared to fetching ranges)
â—Ź But it is still polling, so...
The Redis Stream Consumer API](https://image.slidesharecdn.com/201810redistlvmeetup-v5streams-181028190617/85/Redis-v5-Streams-36-320.jpg)
![XREAD [COUNT count] [BLOCK ms]
STREAMS key [key ...] ID [ID ...]
â—Ź Like `BRPOP` (or `BZMINPOP` ;))
â—Ź Supports the special `$` ID, i.e. "new messages
since blockage"
â—Ź What about message delivery semantics?
The Redis Stream Consumer Blocking API](https://image.slidesharecdn.com/201810redistlvmeetup-v5streams-181028190617/85/Redis-v5-Streams-37-320.jpg)
![Like PubSub, it appears to "fire and forget", or
at-most-once delivery for efficient fan-out.
Contrastingly, messages in a stream are stored.
The consumer manages its last read ID, and can
resume from any point.
(And unlike blocking list (and zset :)) operations,
multiple consumers can consume the same stream)
XREAD [BLOCK] message delivery semantics …](https://image.slidesharecdn.com/201810redistlvmeetup-v5streams-181028190617/85/Redis-v5-Streams-38-320.jpg)
![XREADGROUP GROUP
<groupname> <consumername>
[COUNT count] [BLOCK ms]
STREAMS key [key ...] ID [ID ...]
â—Ź consumername is the member's ID
â—Ź groupname is the name of the group
â—Ź The special `>` ID means "new messages", any
other ID returns the consumer's history
Consumers Group API, #1](https://image.slidesharecdn.com/201810redistlvmeetup-v5streams-181028190617/85/Redis-v5-Streams-44-320.jpg)
![XACK <key> <group> <id> [<id> …]
Acknowledges the receipt of messages.
(that's at-least-once message delivery semantics)
Essentially removes them from the PEL.
Observation: consumername is not required, only
an ID, so anyone can `XACK` pending messages.
Consumers Group API, #3](https://image.slidesharecdn.com/201810redistlvmeetup-v5streams-181028190617/85/Redis-v5-Streams-47-320.jpg)
![XPENDING <key> <group>
[<start> <stop> <count>]
[<consumer>]
XCLAIM <key> <group> <consumer>
<min-idle-time>
<id> [<id> …] [MOAR]
CG introspection & handling consumer failures.
Consumers Group API, #4](https://image.slidesharecdn.com/201810redistlvmeetup-v5streams-181028190617/85/Redis-v5-Streams-48-320.jpg)
![XINFO <key>
XDEL <key> <id> [<id> …]
XTRIM <key> [MAXLEN [~] <n>]
Streams API, some loose ends](https://image.slidesharecdn.com/201810redistlvmeetup-v5streams-181028190617/85/Redis-v5-Streams-49-320.jpg)
Redis v5 & Streams
- 1. Redis TLV Meetup v5 & Streams @itamarhaber, October 2018
- 2. Redis v5
- 3. Redis v5 • Released on Wed Oct 17 13:28:26 CEST 2018 • 9.5 years after v1, 15 months of development • Major feature: Streams • Other stuff: – Project Spartacus – Active Defragmentation v2 – LOLWUT – ZPOPMIN/MAX and their blocking variants – Integrated help for subcommands
- 4. Georg Ness Schotter vs LOLWUT (performance) ©VictoriaandAlbertMuseum,London
- 5. ZPOP - youtube.com/watch?v=Xk4avdjdM-E Sorted Sets now support List-like pop operations. â—Ź ZPOPMIN - removes and returns the lowest-ranking member â—Ź ZPOPMAX - same, but for highest-ranking â—Ź BZPOPMIN, BZPOPMAX - blocking variants
- 6. Integrated help for subcommands, e.g.:
- 7. Streams
- 8. A data stream is a sequence of elements. Consider: • Real time sensor readings, e.g. particle colliders • IoT, e.g. the irrigation of avocado groves • User activity in an application • … • Messages in distributed systems In the context of data processing...
- 9. … one in which the failure of a computer you didn't even know existed can render your own computer unusable. Leslie Lamport A distributed system is “
- 10. ... a model in which components located on networked computers communicate and coordinate their actions by passing messages Distributed Computing, Wikipedia Includes: client-server, 3/n-tier, peer to peer, SOA, micro- & nanoservices, FaaS & serverless… A distributed system is “
- 11. There are only two hard problems in distributed systems: 2. Exactly-once delivery 1. Guaranteed order of messages 2. Exactly-once delivery Mathias Verraes, on Twitter An observation “
- 12. Fact #1: you can choose one and only one: • At-most-once delivery, i.e. "fire and forget" • At-least-once delivery, i.e. explicit acknowledgement sent by the receiver Fact #2: exactly-once delivery doesn't exist (unless you change the definition of MSD) Observation: order is usually important (duh) Refresher on message delivery semantics
- 13. Consider the non-exhaustive list at taskqueues.com • 17 message brokers, including: Apache Kafka, NATS, RabbitMQ and Redis • 17 queue solutions, including: Celery, Kue, Laravel, Sidekiq, Resque and RQ (<- all these use Redis as their backend btw ;)) And that's without considering protocol-based architectures, legacy busses etc... This isn't exactly a new challenge
- 14. Streams: Anatomy
- 15. The Log is a storage abstraction that is: • Append-only, can be truncated • A sequence of records ordered by time A Logical Log is: • Based on a logical offset, e.g. time (vs. bytes) • (Therefore time range queries) • Can be made up of data structures (vs. lines) A Stream is not unlike a (logical) log
- 16. A Stream is (also) a storage abstraction, that is basically an ordered, logical log of messages (records). These messages are: â—Ź Made up of: â—‹ Data payload (semi-structured usually) â—‹ Metadata (e.g. identifier) â—Ź Immutable, once created â—Ź Always added to the end of the stream So what is a stream?
- 17. A producer is a software component that adds messages (always to the end of) a stream. A consumer is a software component that reads messages from a stream (and acts on them). It can start reading the messages from any arbitrary offset, or just wait for new ones. The Stream Players: producers and consumers
- 18. A picture I made of a stream 0 1 2 3 4 5 6 7 8 9 10 11 12 13 Producer Consumer 1 position Consumer 2 position "Next" message Message 0
- 19. 1. A component can both be a producer and a consumer. Either for the same, or between different streams. It depends. 2. Multiple stream producers can exist. At least one is usually needed though. 3. A stream can exist without any consumers. That's kind of pointless though. Some observations
- 21. Consider the alternative, i.e. batch processing. Besides fast response times (batch=1): â—Ź Scalable (distributed) design â—Ź Loose coupling of components and faults â—Ź Enables building complex-er pipelines Why (do architects) build stuff on streams?
- 22. … that architects like using, like: ● CQRS ● Event sourcing ● Unified (distributed commit) log ● Microservices ● ... Also, they fit so well with other stuff...
- 23. An abstraction that is useful to work with (in distributed systems). Capable of trivially addressing message ordering. Able to provide (depending on the implementation) ATM and ALO MDS. An enabler for Stream Processing (e.g. Spark Streams, Kafka's Stream Processors). The Stream (for architects) is
- 24. Streams: Redis'
- 25. Necessity is the mother of invention There ain't no such thing as a free lunch The existing (i.e. lists, sorted sets, PubSub) isn't "good enough" for things like: • Log/time series-like data patterns • At-least-once messaging with fan-out Also Disque, listpacks, radix trees & reading Kafka :) Why reinvent hot water (in Redis)? “ “
- 26. â—Ź Sorted Sets? Memory hungry, no `BZMINPOP` (at that time ;)), ordering depends on mutable score and require uniqueness of elements â—Ź Lists? Inefficient access (linear), index (changeable)-based, and only have queue-like blocking operations (single consumer) â—Ź PubSub? Fan-out, sure, but only AMO MDS How do you model "messages" in Redis < v5?
- 27. â—Ź A project by Salvatore Sanfilippo â—Ź Like Redis, but is â—Ź "a distributed, in-memory, message broker" â—Ź Eventually consistent (AP in CAP terms) â—Ź Last updated: Jan 2016 â—Ź Planned to: come back as a Redis module in v6 â—Ź Observation: A Stream "API" can also be built on top of a message broker (see Kafka) Interjection: What is Disque?
- 28. â—Ź A 1st-class citizen, a data structure like any other â—Ź The most complex, implementation-wise â—Ź Stores entries â—Ź Is conceptually made up from 3 APIs: a. Producer b. Consumer c. Consumers Group What is the Redis Stream?
- 29. XADD key [MAXLEN ~ n] <ID | *> <field> <string> [field string…] > XADD mystream * foo bar baz qaz 1532556560197-42 Time complexity: O(log n) See https://redis.io/commands/xadd The Redis Stream Producer API
- 30. Every entry has a unique ID that is its logical offset. The ID is in following format: <epoch-milliseconds>-<sequence> Each ID part is a 64-bit unsigned integer Sequence is for ordering at millisecond scope When user-provided, has to be bigger than latest. When not, max(localtime, latest) is used. The entry ID
- 31. Redis' Stream entries are made up of field-value pairs. Like a Hash. Unlike a Hash, repeating field names in consecutive entries are compressed. Values are not compressed. Yet. (Time series engines often compress values, with values being after all just numbers) The entry itself
- 32. The `MAXLEN` subcommand is for that. The `~` means about, less expensive to use. The stream is capped by the number of entries. Not by time frame. Future regarding that is "yet unclear" - ref: https://stackoverflow.com/questions/51168273/re dis-stream-managing-a-time-frame Side note: capped streams
- 33. XLEN key - does exactly that, not very interesting. X[REV]RANGE key <start | -> < end | +> [COUNT count] - much more interesting :) Get a single entry (start = end = ID) SCAN-like iteration on a stream (IDs inc.), but better Range (timeframe) queries on a stream What's in the stream "API"
- 34. > XRANGE mystream - + 1) 1) 1532556560197-0 2) 1) "foo" 2) "bar" 3) "baz" 4) "qaz" A "real" picture of a stream
- 35. Yes. No. Maybe. It can be used for consuming, but that requires the client constantly polling the stream for new entries. So generally, no. There's something better for consumers. Is X[REV]RANGE the Consumer API?
- 36. XREAD [COUNT count] STREAMS key [key ...] ID [ID ...] Somewhat like X[REV]RANGE, but: â—Ź Supports multiple streams â—Ź Easier to consume from an offset onwards (compared to fetching ranges) â—Ź But it is still polling, so... The Redis Stream Consumer API
- 37. XREAD [COUNT count] [BLOCK ms] STREAMS key [key ...] ID [ID ...] â—Ź Like `BRPOP` (or `BZMINPOP` ;)) â—Ź Supports the special `$` ID, i.e. "new messages since blockage" â—Ź What about message delivery semantics? The Redis Stream Consumer Blocking API
- 38. Like PubSub, it appears to "fire and forget", or at-most-once delivery for efficient fan-out. Contrastingly, messages in a stream are stored. The consumer manages its last read ID, and can resume from any point. (And unlike blocking list (and zset :)) operations, multiple consumers can consume the same stream) XREAD [BLOCK] message delivery semantics …
- 39. A consumer of a stream gets all entries in order, and will eventually become a bottleneck. Or fail. Possible workarounds: • Add a "type" field to each record - that's dumb • Shard the stream to multiple keys - meh • Have the consumer dispatch entries as jobs in queues or messages in a … GOTO 10 The problem with scaling consumers
- 40. Consider the Stream. There needs to be a way for constructing a high-level/pseudo consumer, such that is made up of multiple of its instances running in parallel, each processing a mutually-exclusive subset of the entries. Another, high-level, perspective
- 41. … allow multiple consumers to cooperate in processing messages arriving in a stream, so that each consumers in a given group takes a subset of the messages. Shifts the complexity of recovering from consumer failures and group management to the Redis server Consumer Groups “
- 42. A group picture (via @antirez)
- 43. 1. Members are identified 2. New members get only undelivered messages 3. Each message is delivered to only one member 4. A member can only read its messages 5. A member must explicitly acknowledge the receipt of messages Observation: Big Brother (Redis) is observing you Trivia: this is where most of the effort went into (Consumer) Group membership rules
- 44. XREADGROUP GROUP <groupname> <consumername> [COUNT count] [BLOCK ms] STREAMS key [key ...] ID [ID ...] â—Ź consumername is the member's ID â—Ź groupname is the name of the group â—Ź The special `>` ID means "new messages", any other ID returns the consumer's history Consumers Group API, #1
- 45. XGROUP CREATE <key> <groupname> <id or $> // Explicit creation! // And key must exist XGROUP SETID <key> <id or $> XGROUP DESTROY <key> <groupname> XGROUP DELCONSUMER <key> <groupname> <consumername> Consumers Group API, #2
- 46. One of the internal data structures used. Tracks which member saw which messages. â—Ź When a new message is delivered, a new entry in the list is created â—Ź When an "old" message is delivered, the last delivered timestamp and number of deliveries counter (for it) are updated The Pending Entries List (PEL) is
- 47. XACK <key> <group> <id> [<id> …] Acknowledges the receipt of messages. (that's at-least-once message delivery semantics) Essentially removes them from the PEL. Observation: consumername is not required, only an ID, so anyone can `XACK` pending messages. Consumers Group API, #3
- 48. XPENDING <key> <group> [<start> <stop> <count>] [<consumer>] XCLAIM <key> <group> <consumer> <min-idle-time> <id> [<id> …] [MOAR] CG introspection & handling consumer failures. Consumers Group API, #4
- 49. XINFO <key> XDEL <key> <id> [<id> …] XTRIM <key> [MAXLEN [~] <n>] Streams API, some loose ends
- 51. • Introduction to Redis Streams https://redis.io/topics/streams-intro • The Redis Manifesto https://github.com/antirez/redis/blob/unstable/MANIFESTO • Salvatore's blog posts http://antirez.com/news/114 and http://antirez.com/news/116 • Salvatore's inaugural Streams demo https://www.youtube.com/watch?v=ELDzy9lCFHQ • Salvatore's live demo at Redis Day Tel Aviv 2018 https://www.youtube.com/watch?v=qXEyuUxQXZM • RCP 11 - The stream data type https://github.com/redis/redis-rcp/blob/master/RCP11.md • Reddit discussion https://www.reddit.com/r/redis/comments/4mmrgr/stream_data_structure_for_redis_lets _design_it/ • Hacker News discussion https://news.ycombinator.com/item?id=15384396 • Consumer groups specification https://gist.github.com/antirez/68e67f3251d10f026861be2d0fe0d2f4 • Consumer groups API https://gist.github.com/antirez/4e7049ce4fce4aa61bf0cfbc3672e64d & https://gist.github.com/antirez/4e7049ce4fce4aa61bf0cfbc3672e64d (some) Redis References