Redis Streams - Fiverr Tech5 meetup
2 likes935 views
Redis Streams is an innovative data structure in Redis designed for managing time-series data and providing messaging capabilities similar to Kafka. It supports real-time data processing, allowing multiple producers and consumers to interact with the data stream efficiently while enabling various delivery semantics. Currently, Redis Streams is partially implemented with expected full availability in version 5.0 by April 2018, inviting users to explore its functionality further.
![# Adding entries
redis> XADD <key> <* | id>
[MAXLEN [~] <n>]
<field> <value> [...]
<epoch-milliseconds>-<sequence>
# Stream length
redis> XLEN <key>
(integer) <stream-length>](https://image.slidesharecdn.com/redisstreams-fiverrtech5-180207174319/85/Redis-Streams-Fiverr-Tech5-meetup-18-320.jpg)
![# Iterating
redis> X[REV]RANGE <key>
<start> <stop>
[COUNT <n>]
1) 1) <entry-id>
2) 1) <field1>
2) <value1>
3) ...](https://image.slidesharecdn.com/redisstreams-fiverrtech5-180207174319/85/Redis-Streams-Fiverr-Tech5-meetup-19-320.jpg)
![# [Blocking] read
redis> XREAD [BLOCK <milliseconds>]
STREAMS <key> [...]
<start> [...]
1) 1) <entry-id>
2) 1) <field1>
2) <value1>
3) ...](https://image.slidesharecdn.com/redisstreams-fiverrtech5-180207174319/85/Redis-Streams-Fiverr-Tech5-meetup-20-320.jpg)
Redis Streams - Fiverr Tech5 meetup
- 1. Redis Streams @itamarhaber #Tech5 @Fiverr—FEBRUARY 2018
- 2. Who We Are Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source home and commercial provider of Redis Enterprise (Redise ) technology, platform, products & services. @itamarhaber, Technology Evangelisthello I am
- 3. Redis Streams • 1st class Redis citizens • An abstract data type that is not unlike a log • Designed with time series data in mind • Provide some "Kafkaesque" messaging abilities This session is about
- 4. https://redis.io (emphasis for session context) Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries. Redis has built-in replication, Lua scripting, LRU eviction, transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster. “
- 5. 1. REmote DIctionary Server 2. / rɛdɪs/, pronounced “red-iss” 3. OSS (BSD3), https://github.com/antirez/redis 4. In-memory, but with optional disk persistence 5. By Salvatore Sanfilippo @antirez circa 27/2/09 6. DSL4ADT: A Domain Specific Language (DSL) for Abstract Data Types (ADT) 7. Designed for performance and simplicity Redis is
- 6. Necessity is the mother of invention There ain't no such thing as a free lunch The existing (i.e. lists, sorted sets, PubSub) isn't "good enough" for things like: • Log-like data patterns • At-least-once messaging with fan-out And listpacks, radix trees & reading Kafka :) Why invent yet another Redis thingamajig? “ “
- 7. A storage abstraction that is: • Append-only, can be truncated • A sequence of records ordered by time A Logical Log is: • Based on a logical offset, i.e. time (vs. bytes) • Therefore time range queries • Made up of in-memory data structures, naturally The Log is (hardly a new thing)
- 8. A data stream is a sequence of elements. Consider: • Real time sensor readings, e.g. particle colliders • IoT, e.g. the irrigation of avocado groves • User activity in an application • … • Messages in distributed systems Logging streams of semi-structured data
- 9. A distributed system is a model in which components located on networked computers communicate and coordinate their actions by passing messages Distributed Computing, Wikipedia Includes: client-server, 3/n-tier, peer to peer, SOA, micro- & nanoservices, FaaS & serverless… A side note about Distributed Systems “
- 10. There are only two hard problems in distributed systems: 2. Exactly-once delivery 1. Guaranteed order of messages 2. Exactly-once delivery Mathias Verraes, on Twitter An observation “
- 11. Fact #1: you can choose one and only one: • At-most-once delivery, i.e. "shoot and forget" • At-least-once delivery, i.e. explicit ack Fact #2: exactly-once delivery doesn't exist Observation: order is usually important (duh) Refresher on message delivery semantics
- 12. Consider the non-exhaustive list at taskqueues.com • 17 message brokers, including: Apache Kafka, NATS, RabbitMQ and Redis • 17 queue solutions, including: Celery, Kue, Laravel, Sidekiq, Resque and RQ <- all these use Redis as their backend btw ;) And that without considering protocol-based etc... This isn't exactly a new challenge
- 13. Redis (in general and) Streams (in particular) are: • Everywhere, from the IoT's edge to the cloud • Blazing fast, massive throughput • Usable from all(most) languages and platforms (IoT microcontrollers included) Note: apropos IoT, they are great async buffers So again, why "reinvent hot water"?
- 14. A stream is a sequence of entries (records). It: • Is "sharded" by key ("topic") • Has 1+ producers • Has 0+ consumers • Can provide at-most- or at-least-once semantics • Enables stream processing/real time pipelines (as opposed to batch) Redis Streams "formalism"
- 15. A picture of a stream 0 1 2 3 4 5 6 7 8 9 10 11 12 13 Producer Consumer 1 position Consumer 2 position Next entry ("*")
- 16. Every entry has a unique ID that is its logical offset. The ID is in following format: <epoch-milliseconds>-<sequence> Note: each ID part is a 64-bit unsigned integer An entry also has one or more ordered field-value pairs, allowing for total abstraction (the empty string is a valid field name, good for time series). Entries in the Stream
- 17. Streamz Demo Or how I'm graphing my laptop's CPU and battery temperatures using only bash, iStats, redis-cli, redis-server, docker, grafana & a browser https://github.com/itamarhaber/streams-cpubattmp
- 18. # Adding entries redis> XADD <key> <* | id> [MAXLEN [~] <n>] <field> <value> [...] <epoch-milliseconds>-<sequence> # Stream length redis> XLEN <key> (integer) <stream-length>
- 19. # Iterating redis> X[REV]RANGE <key> <start> <stop> [COUNT <n>] 1) 1) <entry-id> 2) 1) <field1> 2) <value1> 3) ...
- 20. # [Blocking] read redis> XREAD [BLOCK <milliseconds>] STREAMS <key> [...] <start> [...] 1) 1) <entry-id> 2) 1) <field1> 2) <value1> 3) ...
- 21. # And the usual Redis goodness, e.g. TX redis> MULTI ... # Or server-side processing redis> EVAL "return 'Lua Rocks!'" 0 ... # Or your own custom module redis> MODULE LOAD <your-module-here> OK
- 22. A consumer of a stream gets all entries in order, and will eventually become a bottleneck. Possible workarounds: • Add a "type" field to each record - that's dumb • Shard the stream to multiple keys - meh • Have the consumer dispatch entries as jobs in queues… GOTO 10 The problem with scaling consumers
- 23. … allow multiple consumers to cooperate in processing messages arriving in a stream, so that each consumers in a given group takes a subset of the messages. Shifts the complexity of recovering from consumer failures and group management to the Redis server Consumer Groups “
- 24. We are here :) • Groups are named and are explicitly (!) created: XGROUP CREATE temps agg $ • Consumers are also named, and each gets only a subset of the stream: XREAD-GROUP GROUP agg CONSUMER escher-01 STREAMS temps > • XACK/NOACK in XREAD, XCLAIM, XPENDING... Group orientation
- 25. Presently OSS Redis Streams are: • Partially implemented – Existing commands are relatively stable – Some API corners still missing, e.g. XTRIM – Consumer Groups are getting real • A part of the unstable branch • Expected to be GA as v5.0 during April 2018 Up to date status (Jan 26th)
- 26. • From your browser: https://try.redis.io • Or download it: https://redis.io/download • Or clone it: https://github.com/antirez/redis • Or dockerize it: docker run -it redis • Or try Redis Enterprise by https://redislabs.com Next, try Redis yourself!
- 27. • The Redis Manifesto https://github.com/antirez/redis/blob/unstable/MANIFESTO • Salvatore's blog posts http://antirez.com/news/114 and http://antirez.com/news/116 • Salvatore's Streams demo https://www.youtube.com/watch?v=ELDzy9lCFHQ • RCP 11 - The stream data type https://github.com/redis/redis-rcp/blob/master/RCP11.md • Reddit discussion https://www.reddit.com/r/redis/comments/4mmrgr/stream_data_structure_for_redis_lets _design_it/ • Hacker News discussion https://news.ycombinator.com/item?id=15384396 • Consumer groups specification https://gist.github.com/antirez/68e67f3251d10f026861be2d0fe0d2f4 • Consumer groups API https://gist.github.com/antirez/4e7049ce4fce4aa61bf0cfbc3672e64d & https://gist.github.com/antirez/4e7049ce4fce4aa61bf0cfbc3672e64d • Redis Streams and the Unified Log https://brandur.org/redis-streams • Introduction to Redis Streams https://hackernoon.com/introduction-to-redis-streams-133f1c375cd3 References
- 28. Join us next month at Redis Day Tel Aviv Thank you, Questions?