Realtime Distributed Analysis of Datastreams
0 likes1,207 views
This document discusses real-time distributed data analysis and frameworks like Storm. It introduces the lambda architecture for working with big data, which has a speed layer, serving layer, and batch layer. The batch layer stores immutable data and precomputes batch views using Hadoop. The serving layer provides random access to these views using systems like Impala. The speed layer compensates for high-latency batch views using fast, incremental algorithms like Storm. It also describes how Twitter evolved from queues and workers to using Storm for scalable, fault-tolerant real-time analytics.
Realtime Distributed Analysis of Datastreams
- 1. Realtime Distributed Analysis of Datastreams Philipp Nolte – University of Passau – January 2014 1
- 2. Learn Why we need fancy Big Data frameworks. How the lambda architecture looks like. How twitter used to do real-time analytics. Why twitter created Storm. How Storm works. 2
- 3. Limits Imagine a traditional web analytics software: Every page view increments the url’s database row. 3
- 4. First Aid Queue your writes and write in batches. Shard your data: Partition horizontally. 4
- 5. Chronic Issues Fault-tolerance is hard. Applications become more and more complex. You have to do all the work. 5
- 6. New Tools Large scale computation systems such as Hadoop. Scalable databases such as Casandra and Riak. Easy to use frameworks such as Storm and Dempsy. 6
- 7. Lambda Architecture Theoretical, abstract architecture for working with big data. Speed Layer Serving Layer Batch Layer 7
- 8. Goal Compute arbitrary functions on arbitrary data. query = function ( all data ) 8
- 9. Properties Robust and fault-tolerant. Low latency reads and updates. Scalable. Minimal maintenance. 9
- 10. Batch Layer Speed Layer Serving Layer Stores the immutable master dataset. Precomputes arbitrary batch views. Home of batch processing and map reduce systems such as Hadoop. 10 Batch Layer
- 11. Serving Layer Speed Layer Serving Layer Batch Layer Read-only random-access to batch views. Updated by batch layer. Indexes batch views. Home of real-time query systems such as Cloudera Impala for Hadoop. 11
- 12. Speed Layer Speed Layer Serving Layer Batch Layer Compensates for high-latency batch views. Fast, incremental algorithms. More complex because of random-writes. Home of Apache HBase or Storm. 12
- 13. Lambda Architecture Speed Layer Realtime Views Batch Views Data Serving Layer Batch Layer 13 Query
- 14. Available Data Batch View Time Realtime View Discard Realtime View as soon as it is represented in the batch view. Batch View Realtime View 14
- 16. Storm Guaranteed message processing without message brokers. Horizontal scalability. Fault-tolerance. High level of abstraction. Just works. 16
- 20. Know Why we need fancy Big Data frameworks. How the lambda architecture looks like. How twitter used to do real-time analytics. Why twitter created Storm. How Storm works. 20