Spark meetup - Zoomdata Streaming
0 likes895 views
This document discusses interactive data visualization powered by Spark streaming. It describes how Zoomdata allows users to visualize streaming data in real-time as new data is delivered. The key challenges of streaming data like time, frequency, retention and synchronization are addressed. Zoomdata receives streaming data via Kafka or JMS, processes it using Spark Streaming in a single JVM, and stores the data in buffers like MongoDB. This allows for interactive data visualizations that update in real-time as new streaming data is processed. The document also outlines technologies used, how the system scales out, benefits, and includes a demo of streaming data from Twitter to MemSQL and Solr sinks using Spark Streaming.
Related topics:1 of 14
Downloaded 20 times
More Related Content
Ad
Viewers also liked (20)
Ad
Similar to Spark meetup - Zoomdata Streaming (20)
More from Zoomdata (8)
Recently uploaded (20)
Spark meetup - Zoomdata Streaming
- 1. Interactive Visualization of Data powered by Spark
- 2. Streaming Data @ Zoomdata Visualizations react to new data delivered Users start, stop, pause the stream Users select a rolling window or pin a start time to capture cumulative metrics
- 3. Drivers for Streaming Data Data Freshness Time to Analytic Business Context
- 4. Challenges ● Time ● Frequency ● Retention ● Synchronization ● Order ● Updates
- 5. Addressing the Problem @ Zoomdata Historical Revised Receive Data JMS Kafka Manipulate Stream Single JVM in Memory Spark Streaming Hold Data in Buffer MongoDB Pluggable Interact with Data Custom Code Pluggable
- 6. Technology Cast ● The Stream - Kafka, Kinesis, JMS ● Processing Fabric - Spark Streaming ● Landing Area - MemSQL, Solr, Kudu, Others
- 7. How it looks
- 8. With the rest of the app
- 9. Scale Out
- 10. Benefits ● Contextual Expressiveness with Streaming Data ● Independent scalability (scale-up, scale-around) ● Expressiveness powered by Spark -- using Windowing (dataframe API with stream)
- 11. Side Benefits ● Separation of concerns ● Disaster Recovery, COOP, other Data management concerns ● Restatements ● Options!
- 12. Demo ● Twitter Producer ● Spark Streaming ● MemSQL & Solr Sinks
- 13. Future Work ● Cross Stream Synchronization & Fusion ● On-demand scale out and resource management via Mesos ● Schema Evolution ● Storage Tiering
- 14. Thanks For more information contact: ruhollah@zoomdata.com quan@zoomdata.com