Redis Streams plus Spark Structured Streaming

Editor's Notes

• #3: Agendaa: Takeaway: Use Redis Streams + Spark-Redis + Structured Streaming with Microbatches in Spark to collect and process data stream in real-time at scale
• #4: Call the spark engine with spark-submit in scala directory: spark-submit --class com.redislabs.streaming.ClickAnalysis --jars ./lib/spark-redis-2.4.0-SNAPSHOT-jar-with-dependencies.jar --master local[*] ./target/scala-2.11/redisexample_2.11-1.0.jar 4 datastructures clickstream – redis stream data structure to collect all clicks $ XLEN clickstream Spark 2.3 introduced Structured streaming that does microbatches – collect from streams, such as Redis Streams or Kafka A few messages every few milliseconds. Then Spark runs queries to aggregate and collecting and storing somewhere, such as Redis Trick – Any hash data structure that starts with clicks: belongs to a table called Clicks $ HGETALL clicks:image_1 Fields called image and count are columns in the table Configure Spark SQL so it knows that any field that starts with clicks belongs to a table called Clicks $ HMSET clicks:image_test img test count 10000
• #5: Summarize demo
• #6: Read then go to next slide
• #7: What are the functional blocks: Data ingest-collect all clicks without losing any – Redis Cloud w/Streams  free up to 30 MB Spark-Redis into Spark Process data in real-time – Spark w/Structured Streaming for Microbatches Data Querying – some kind of custom chart, leaderboard, or Grafana  Using Spark SQL with Redis Cloud again
• #11: Cover at high level Connects producers with consumers. May have many of either
• #12: Redis Streams supports both asynchronous communication and look-back queries
• #13: How many have used pubsub, list or sorted sets Pubsub – no lookback queries. All asynchrous List – one list cannot support many consumers. Sorted Sets – solves problem  don’t need copy for each consumer, but have to always poll for data. Can use blocking call, but transforms into a list. For streaming you have to poll.
• #14: Redis Streams manages the life cycle of the streaming data effectively (Example: Consumer groups and their commands XREADGROUP, XACK and XCLAIM ensure every data object is consumed properly) It offers consumers a rich choice of options from to consume the data – they can read from where they left off, or only the new data, or from the beginning The lookback queries are super fast as they are powered by the radix trees Kafka or Kenisis have a timeframe limit. Redis has no timeframe, you can cap by maxlength / size
• #15: If you have different types of consumers …
• #16: Ex: toll booth – can back up – but we can match rate of arrival with rate of departure
• #18: So this is our stream example
• #19: clickstream is the stream key Lets redis create the timestamp img is the field So that’s how data ingest works. Any questions?
• #20: Stop and run query to see latest count scala> sql("select * from clicks").show(); $ DEL clicks:image_test
• #23: Marketing definition from databricks
• #24: With Structured Streaming, Spark pulls data in micro-batches (like a table) Every microbatch has rows and each row is like a dataframe so you can run dataframe operations on these microbatches Windowing – can aggregate for last 30 mins Can also dedupe Go to databricks website to see more I’m doing aggregations in my demo Used to only do batches Now in 2.4 microbatches- batches in miliseconds Now in experimental mode – is continuous processing, get dataframes in microsections Finally can define an Output sink, Or output to console 3 modes: dump everything, append only, or update sync,
• #25: Redis Labs developed and supports this open source library Data Source and Data Sync Redis streams Dump data into Redis Query Redis from Spark All written in Scala
• #26: How do you use Redis as a Datasource? There are 4 steps Connect to Redis db Map Redis Streams keyvaalue pairs to microbatch Query object Run the query in a loop
• #27: Connect to Redis Cloud Move to next sldie
• #28: 2. Interpreting the stream data Clickstream is key name Img is field name
• #29: 3. Defining the query Group by img Count
• #30: 4. Dumping into a ForEachWriter - Custom – see next slide -
• #32: Stop and run query to see latest count scala> sql("select * from clicks").show();
• #33: Any questions? How do you query redis? Can connect ODBC drivers to Spark and can query Redis?
• #34: How to connect to Redis How to map to Redis? Run Query
• #35: 1. 2. Create a table (do it only once). Tell it to use class name org.apache.spark.sql.redis Map to table ‘clicks’
• #36: 3. Run query