Kafka to Hadoop Ingest with Parsing, Dedup and other Big Data Transformations
Download as PPTX, PDF1 like1,468 views
The document provides an overview of Apache Apex, a platform for developing scalable and fault-tolerant distributed applications integrated with Hadoop. It covers data ingestion processes, specifically detailing a Kafka ETL use case that involves consuming, processing, and storing data. Additionally, the document highlights key features of Kafka as a distributed messaging system and presents resources for further learning and exploration of Apache Apex.
Kafka to Hadoop Ingest with Parsing, Dedup and other Big Data Transformations
- 1. © 2016 DataTorrent Chaitanya Chebolu Committer, Apache Apex Engineer, DataTorrent Sep 14, 2016 Data Ingestion - Kafka ETL
- 2. © 2016 DataTorrent Agenda 2 • Introduction about Apache Apex (Architecture, Application, Native Hadoop Integration) • What is Data Ingestion • Use Case : Kafka ETL • Brief about Kafka • Kafka ETL App • Kafka ETL Demo
- 3. © 2016 DataTorrent3 Apache Apex • Platform and runtime engine that enables development of scalable and fault-tolerant distributed applications • Hadoop native (Hadoop >= 2.2) No separate service to manage stream processing Streaming Engine built into Application Master and Containers • Process streaming or batch big data • High throughput and low latency • Library of commonly needed business logic • Write any custom business logic in your application
- 4. © 2016 DataTorrent4 Apex Architecture
- 5. © 2016 DataTorrent5 An Apex Application is a DAG (Directed Acyclic Graph) A DAG is composed of vertices (Operators) and edges (Streams). A Stream is a sequence of data tuples which connects operators at end-points called Ports An Operator takes one or more input streams, performs computations & emits one or more output streams ● Each operator is USER’s business logic, or built-in operator from our open source library ● Operator may have multiple instances that run in parallel
- 6. © 2016 DataTorrent6 Apex - Native Hadoop Integration • YARN is the resource manager • HDFS used for storing any persistent state
- 7. © 2016 DataTorrent What is Data Ingestion? 7 • Data Ingestion A process of obtaining, importing, and analyzing data for later use or storage in a database • Big Data Ingestion Discovering the data sources Importing the data Processing data to produce intermediate data Sending data out to durable data stores
- 8. © 2016 DataTorrent Use Case: Kafka ETL 8 • Consuming data from Kafka • Processing data to produce intermediate data • Writing the processed data to HDFS
- 9. © 2016 DataTorrent Brief about Kafka 9 ● Distributed Messaging System. ● Data Partitioning Capability. ● Fast Read and Writes. ● Basic Terminology ○ Topic ○ Producer ○ Consumer ○ Broker
- 10. © 2016 DataTorrent Kafka ETL App 10 Kafka Parser Dedup Transform Formatter HDFS
- 11. © 2016 DataTorrent Kafka ETL Demo 11 Demo
- 12. © 2016 DataTorrent Resources 12 • http://apex.apache.org/ • Learn more: http://apex.apache.org/docs.html • Subscribe - http://apex.apache.org/community.html • Download - http://apex.apache.org/downloads.html • Follow @ApacheApex - https://twitter.com/apacheapex • Meetups – http://www.meetup.com/pro/apacheapex/ • More examples: https://github.com/DataTorrent/examples • Slideshare: http://www.slideshare.net/ApacheApex/presentations • https://www.youtube.com/results?search_query=apache+apex • Free Enterprise License for Startups - https://www.datatorrent.com/product/startup-accelerator/