Big Data Architecture
75 likes26,501 views
This document discusses different architectures for big data systems, including traditional, streaming, lambda, kappa, and unified architectures. The traditional architecture focuses on batch processing stored data using Hadoop. Streaming architectures enable low-latency analysis of real-time data streams. Lambda architecture combines batch and streaming for flexibility. Kappa architecture avoids duplicating processing logic. Finally, a unified architecture trains models on batch data and applies them to real-time streams. Choosing the right architecture depends on use cases and available components.
Big Data Architecture
- 1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH Big Data Architectures Guido Schmutz
- 2. Guido Schmutz Working for Trivadis for more than 18 years Oracle ACE Director for Fusion Middleware and SOA Co-Author of different books Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data Member of Trivadis Architecture Board Technology Manager @ Trivadis More than 25 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Twitter: gschmutz
- 3. Agenda 1. Introduction 2. Traditional Architecture for Big Data 3. Streaming Analytics Architecture for Fast Data 4. Lambda/Kappa/Unifed Architecture for Big Data 5. Summary
- 4. Introduction
- 5. Big Data is still “work in progress” Choosing the right architecture is key for any (big data) project Big Data is still quite a young field and therefore there are no standard architectures available which have been used for years In the past few years, a few architectures have evolved and have been discussed online Know the use cases before choosing your architecture To have one/a few reference architectures can help in choosing the right components
- 6. Hadoop Ecosystem – many choices …. Management /Monitoring Core Analytics Workflow/JobUnstructured Data Sources Structured Data Sources SQL on Hadoop SerializationData Storage Security
- 7. Important Properties to choose a Big Data Architecture Latency Keep raw and un-interpreted data “forever” ? Volume, Velocity, Variety, Veracity Ad-Hoc Query Capabilities needed ? Robustness & Fault Tolerance Scalability …
- 8. From Volume and Variety to Velocity Big Data has evolved … and the Hadoop Ecosystem as well …. Past Big Data = Volume & Variety Present Big Data = Volume & Variety & Velocity Past Batch Processing Time to insight of Hours Present Batch & Stream Processing Time to insight in Seconds Adapted from Cloudera blog article
- 9. Traditional Architecture for Big Data
- 10. “Traditional Architecture” for Big Data Data Collection (Analytical) Data Processing Result StoreData Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools Social RDBMS Sensor ERP Logfiles Mobile Machine Batch compute Stage Result Store Query Engine Computed Information Raw Data (Reservoir) = Data in Motion = Data at Rest
- 11. Use Case 1) – Click Stream analysis: 360 degree view of customer Data Collection (Analytical) Data Processing Result StoreData Sources Data Consumer Channel Batch compute Computed Information Raw Data (Reservoir) Result Store Query Engine Reports Analytic Tools Logfiles
- 12. Use Case 2) – Ingest Relational Data into Hadoop and make it accessible Data Collection (Analytical) Data Processing Result StoreData Sources Data Consumer RDBMS Batch compute Computed Information Raw Data (Reservoir) Result Store Query Engine Reports Service Analytic Tools Alerting Tools
- 13. Use Case 2a) – Ingest Relational Data into Hadoop and make it accessible Data Collection (Analytical) Data Processing Result StoreData Sources Data Consumer RDBMS (CDC) Batch compute Computed Information Raw Data (Reservoir) Result Store Query Engine Reports Service Analytic Tools Alerting Tools Channel
- 14. “Hadoop Ecosystem” Technology Mapping Data Collection (Analytical) Data Processing Result StoreData Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools Social RDBMS Sensor ERP Logfiles Mobile Machine Batch compute Staging Result Store Query Engine Computed Information Raw Data (Reservoir) = Data in Motion = Data at Rest
- 15. Apache Spark – the new kid on the block Apache Spark is a fast and general engine for large-scale data processing • The hot trend in Big Data! • Originally developed 2009 in UC Berkley’s AMPLab • Can run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk • One of the largest OSS communities in big data with over 200 contributors in 50+ organizations • Open Sourced in 2010 – since 2014 part of Apache Software foundation • Supported by many vendors
- 16. Motivation – Why Apache Spark? Apache Hadoop MapReduce: Data Sharing on Disk Apache Spark: Speed up processing by using Memory instead of Disks map reduce . . . Input HDFS read HDFS write HDFS read HDFS write op1 op2 . . . Input Output Output
- 17. Apache Spark “Ecosystem” Spark SQL (Batch Processing) Blink DB (Approximate Querying) Spark Streaming (Real-Time) MLlib, Spark R (Machine Learning) GraphX (Graph Processing) Spark Core API and Execution Model Spark Standalone MESOS YARN HDFS Elastic Search NoSQL S3 Libraries Core Runtime Cluster Resource Managers Data / Data Stores
- 18. Use Case 3) – Predictive Maintenance through Machine Learning on collected data Data Collection (Analytical) Data Processing Result StoreData Sources Data Consumer Machine Batch compute Computed Information Raw Data (Reservoir) Result Store Query Engine Reports Service Analytic Tools Alerting Tools DB StagingFile Channel
- 19. “Spark Ecosystem” Technology Mapping Data Collection (Analytical) Data Processing Result StoreData Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools Social RDBMS Sensor ERP Logfiles Mobile Machine Batch compute Stage Result Store Query Engine Computed Information Raw Data (Reservoir) = Data in Motion = Data at Rest
- 20. Traditional Architecture for Big Data • Batch Processing • Not for low latency use cases • Spark can speed up, but if positioned as alternative to Hadoop Map/Reduce, it’s still Batch Processing • Spark Ecosystems offers a lot of additional advanced analytic capabilities (machine learning, graph processing, …)
- 21. Streaming Analytics Architecture for Big Data
- 22. Streaming Analytics Architecture for Big Data aka. (Complex) Event Processing) Data Collection Batch compute Data Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools Social Logfiles Sensor RDBMS ERP Mobile Machine (Analytical) Real-Time Data Processing Stream/Event Processing Result Store Messaging Result Store = Data in Motion = Data at Rest
- 23. Use Case 4) Alerting in Internet of Things (IoT) Data Collection Batch compute Data Sources Channel Data Consumer Analytic Tools Alerting Tools Sensor Machine (Analytical) Real-Time Data Processing Stream/Event Processing Result Store Messaging Result Store
- 24. Use Case 5) Real-Time Analytics on Sensor Events Data Collection Batch compute Data Sources Channel Data Consumer Analytic Tools Sensor Machine (Analytical) Real-Time Data Processing Stream/Event Processing Result Store Messaging Result Store
- 25. Unified Log (Event) Processing Stream processing allows for computing feeds off of other feeds Derived feeds are no different than original feeds they are computed off Single deployment of “Unified Log” but logically different feeds Meter Readings Collector Enrich / Transform Aggregate by Minute Raw Meter Readings Meter with Customer Meter by Customer by Minute Customer Aggregate by Minute Meter by Minute Persist Meter by Minute Persist Raw Meter Readings
- 26. Streaming Analytics Technology Mapping Data Collection Batch compute Data Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools Social Logfiles Sensor RDBMS ERP Mobile Machine (Analytical) Real-Time Data Processing Stream/Event Processing Result Store Messaging Result Store = Data in Motion = Data at Rest
- 27. Streaming Analytics Architecture for Big Data The solution for low latency use cases Process each event separately => low latency Process events in micro-batches => increases latency but offers better reliability Previously known as “Complex Event Processing” Keep the data moving / Data in Motion instead of Data at Rest => raw events are (often) not stored
- 28. Lambda Architecture for Big Data
- 29. “Lambda Architecture” for Big Data Data Collection (Analytical) Batch Data Processing Batch compute Result StoreData Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools Social RDBMS Sensor ERP Logfiles Mobile Machine (Analytical) Real-Time Data Processing Stream/Event Processing Batch compute Messaging Result Store Query Engine Result Store Computed Information Raw Data (Reservoir) = Data in Motion = Data at Rest
- 30. Use Case 6) Social Media and Social Network Analysis Data Collection (Analytical) Batch Data Processing Batch compute Result StoreData Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools Social (Analytical) Real-Time Data Processing Stream/Event Processing Batch compute Messaging Result Store Query Engine Result Store Computed Information Raw Data (Reservoir) = Data in Motion = Data at Rest
- 31. Lambda Architecture for Big Data Combines (Big) Data at Rest with (Fast) Data in Motion Closes the gap from high-latency batch processing Keeps the raw information forever Makes it possible to rerun analytics operations on whole data set if necessary => because the old run had an error or => because we have found a better algorithm we want to apply Have to implement functionality twice • Once for batch • Once for real-time streaming
- 32. „Kappa“ Architecture for Big Data
- 33. “Kappa Architecture” for Big Data Data Collection “Raw Data Reservoir” Batch compute Data Sources Messaging Data Consumer Reports Service Analytic Tools Alerting Tools Social Logfiles Sensor RDBMS ERP Mobile Machine (Analytical) Real-Time Data Processing Stream/Event Processing Result Store Messaging Result Store Raw Data (Reservoir) = Data in Motion = Data at Rest
- 34. Kappa Architecture for Big Data Today the stream processing infrastructure are as scalable as Big Data processing architectures • Some using the same base infrastructure, i.e. Hadoop YARN Only implement processing / analytics logic once Can Replay historical events out of an historical (raw) event store • Provided by either the Messaging or Raw Data (Reservoir) component Updates of processing logic / Event replay are handled by deploying new version of logic in parallel to old one • New logic will reprocess events until it caught up with the current events and then the old version can be de-commissioned.
- 35. „Unified“ Architecture for Big Data
- 36. “Unified Architecture” for Big Data Data Collection (Analytical) Batch Data Processing (Calculate Models of incoming data) Batch compute Result StoreData Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools Social RDBMS Sensor ERP Logfiles Mobile Machine (Analytical) Real-Time Data Processing Stream/Event Processing Batch compute Messaging Result Store Query Engine Result Store Computed Information Raw Data (Reservoir) = Data in Motion = Data at Rest Prediction Models
- 37. Use Case 7) Fraud Detection Data Collection (Analytical) Batch Data Processing (Calculate Models of incoming data) Batch compute Result StoreData Sources Channel Data Consumer Reports Service Analytic Tools Alerting Tools RDBMS Sensor ERP Logfiles Mobile Machine (Analytical) Real-Time Data Processing Stream/Event Processing Batch compute Messaging Result Store Query Engine Result Store Computed Information Raw Data (Reservoir) Prediction Models
- 38. Summary
- 39. Summary Know your use cases and then choose your architecture and the relevant components/products/frameworks You don’t have to use all the components of the Hadoop Ecosystem to be successful Big Data is still quite a young field and therefore there are no standard architectures available which have been used for years Lambda, Kappa Architecture are best practices architectures which you have to adapt to your environment