Deconstructing Lambda architectures
0 likes170 views
The document discusses the lambda architecture, which is designed to handle large volumes of data and queries over that data. It describes the three main layers - batch, speed, and serving layers - that make up the lambda architecture. The batch layer handles batch processing and generates a master dataset. The speed layer handles real-time queries and compensates for latency in the batch layer. The serving layer indexes and exposes views of the data to support ad-hoc queries.
Deconstructing Lambda architectures
- 1. Deconstructing Lambda Architectures Felix Crisan, Co-Founder & CTO, NETOPIA
- 2. Then...
- 3. ...and now
- 4. Meaning of NoSQL 1970 = We have no SQL 1980 = Know SQL 2000 = No SQL! 2005 = Not only SQL 2015 = No, SQL (slide adapted from @markmadsen)
- 5. λ - Architecture - Why? ● Robust & Fault Tolerant system ● Multiple workloads ● Ad-hoc queries ● Scalable (out rather than up) ● Extensible
- 6. λ - Architecture - So? ● In a nutshell - storage is cheap ● Data is generated (retrieved, stored) in time increments ● Data tends to be immutable (especially events)
- 7. Relational Algebra queryresult = f(all data) f is a composite of one or more σ(select), Π (projection), U(reunion), X(product), - (difference - setwise)
- 8. CAP - Pick Any Two Consistency Availability Partition Tolerance No system can be in this region You can find systems here
- 9. λ-Arch - Who/When? ● Nathan Marz ● Oct 2011 ● “How To Beat the CAP theorem” http://nathanmarz. com/blog/how-to-beat-the-cap-theorem. html
- 10. λ-Architecture - What? (Big)Data Source Batch Layer Speed Layer Serving Layer
- 11. λ-Architecture Layer Holds Characteristics Batch Deep Global Truth High Throughput/High Latency Speed Relevant Local Truth Medium Throughput/Low Latency Serving Data for Rapid Retrieval Low Throughput/Low Latency
- 12. Batch Layer ● de-normalized data inputs/master dataset ● append-only ● scalable ● idempotent calculations ● AP
- 13. Usual suspects ● HDFS/Hadoop ● MapReduce (MRv1), YARN (MRv2) ● Spark/SparkSQL ● Hive ● Pig ● ...and others
- 14. Speed/Realtime Layer ● Realtime is actually Near Real Time ● Compensates latency in Batch Layer ● Continuous computation/Limited window
- 15. Usual Suspects ● Storm (http://storm.apache.org/) ● Spark Streaming (http://spark.apache.org/) ● Samza (http://samza.apache.org/) ● S4 (http://incubator.apache.org/s4/) ● MQ (0MQ,RabbitMQ/AMQP etc)
- 16. Serving Layer ● Indexes and exposes views into data ● Sometimes split into Speed Serving and Batch Serving ● Supports ad-hoc queries
- 17. (Non)Usual suspects ● Drill (http://drill.apache.org/) ● ElephantDB (https://github.com/nathanmarz/elephantdb) ● Voldemort (http://www.project-voldemort. com/voldemort/) ● ElasticSearch/Solr/Lucene (https://www. elastic.co/) ● Cassandra sstables (http://cassandra.apache. org/) ● Cloudera Impala (http://www.cloudera.com/ )
- 18. Takeaways ● Immutability ● (P)Re-computation ● Human fault-tolerance ● No One-Size-Fits-All ● YMMV ● ψ-Architectures?