Data Integration
5 likes3,223 views
This document discusses data integration and architectures for processing both batch and streaming data. It covers topics like data ingestion using tools like Flume, Sqoop and Kafka to move data into data lakes and warehouses. It also discusses batch processing using MapReduce on Hadoop and stream processing using real-time technologies like Kafka and architectures like lambda and kappa for serving queries on both real-time and batch-processed views of the data.
![Batch Processing
Data Lake
Batch
Processing
Pageviews
[url, timestamp]
[url, timestamp]
[url, timestamp]
[url, timestamp]
DBRollups
[url, hour,
count]
[url, hour,
count]
[url, hour,
count]
{url+hour :
count}
{url+hour :
count}
{url+hour :
count}
mapreduce mapreduce Data Analysis](https://image.slidesharecdn.com/dataintegration-170424084417/85/Data-Integration-19-320.jpg)
Data Integration
- 2. Contents Introduction1 2 Data Ingestion 3 Data Processing 4 Data Architectures 5 Workshop
- 5. Relational DBs Log filesSearch indexes NoSQL DBs Message queueMonitoring Data Sources
- 6. Data Warehouse ETL ETL ETL ETL Data Warehouse Ingestion
- 7. Sink Source . . . .Transform Load Extract
- 8. 1990 Data Warehousing - Drop relational assumption - Programmability - Open Source 2008 Hadoop + MapReduce - Batch → Real-time - Daily → Continous 2015 Kafka + Streaming data
- 9. 2. Data Ingestion From ETL to ELT: Flume, sqoop, kafka
- 10. sqoopflume Data Lake Kafka Producer Kafka Producer Kafka Consumer Data Lake Ingestion Kafka
- 11. Channel Channel Processor Interceptor #1 Interceptor #N SinkSource Flume Agent Apache Flume Avro Thrift Kafka Exec JMS Spool dir Twitter Netcat Syslog HTTP HDFS Kafka Hive Logger Avro Thrift IRC HBase Elastic
- 14. Data Pipeline Problem Metrics Pub/Sub A publish/subscribe System
- 16. Apache Kafka Broker 1 Broker 2 Broker 3 Kafka Cluster ● ● ● ●
- 17. Consumer Kafka as reliable Flume channel Flume + Kafka Source Sink Channel Producer Flume as kafka producer/consumer
- 19. Batch Processing Data Lake Batch Processing Pageviews [url, timestamp] [url, timestamp] [url, timestamp] [url, timestamp] DBRollups [url, hour, count] [url, hour, count] [url, hour, count] {url+hour : count} {url+hour : count} {url+hour : count} mapreduce mapreduce Data Analysis
- 20. Stream Processing Real Time Technologies Data Source flume Kafka producer Events / DB writes Process Stream Event Stream Output Stream
- 22. Data Lake Batch Processing Data Processing Architecture Data Source flume Kafka producer Data Analysis
- 23. Data Lake Batch Processing Stream Processing Data Processing Architecture Data Source flume Kafka producer Data Analysis
- 24. Lambda Architecture Serving Layer New Data Stream Batch Views Real-Time Views Partial Aggregate Partial Aggregate Partial Aggregate Real-Time Data Bath LayerPrecompute Views (MapReduce)Batch Processing Real-Time Layer Increment Views Stream Processing Process Stream Merged View query merge
- 25. Data Lake Batch Processing Stream Processing Data Processing Architecture Data Source flume Kafka producer Serving Layer Data Analysis
- 26. Kappa Architecture Serving Layer query Serving DB Output Table n Output Table n+1 Stream Processing System Job Version n Job Version n+1 Data Storage 1 New Data Stream 2 3 .. Where everything is a stream Real-Time Layer query
- 27. 4. Workshop