Kafka and Hadoop at LinkedIn Meetup
Download as PPTX, PDF15 likes3,914 views
The document discusses several methods for getting data from Kafka into Hadoop, including batch tools like Camus, Sqoop2, and NiFi. It also covers streaming options like using Kafka as a source in Hive with the HiveKa storage handler, Spark Streaming, and Storm. The presenter is a software engineer and former consultant who now works at Cloudera on projects including Sqoop, Kafka, and Flume. They also maintain a blog on these topics and discuss setting up and using Kafka in Cloudera Manager.
Kafka and Hadoop at LinkedIn Meetup
- 1. Kafka & Hadoop Gwen Shapira / Software Engineer
- 2. 2©2014 Cloudera, Inc. All rights reserved. • 15 years of moving data around • Formerly consultant • Now Cloudera Engineer: – Sqoop Committer – Kafka – Flume About Me
- 3. 3©2014 Cloudera, Inc. All rights reserved. There’s a book on that!
- 4. 4©2014 Cloudera, Inc. All rights reserved. We are also blogging
- 5. 6 Getting Data from Kafka to Hadoop There are only bad options. It's about finding the best one. ©2014 Cloudera, Inc. All rights reserved.
- 6. 7 Batch ©2014 Cloudera, Inc. All rights reserved.
- 7. 8©2014 Cloudera, Inc. All rights reserved. Camus
- 8. 9©2014 Cloudera, Inc. All rights reserved. Camus ZooKeeper Setup Topic Offsets ProcessesHDFSOtherSystems Task Task Task In process Avro Files In process Avro Files Audit Counts Clean Up Kakfa B A C D F G H I E
- 9. 10©2014 Cloudera, Inc. All rights reserved. Sqoop2 From (RDBMS, HDFS, Hive, Hbase) To (RDBMS, HDFS, Hbase, Hive Kafka) Engine (Webserver, Rest API, Repository, MapReduce) Client
- 10. 11©2014 Cloudera, Inc. All rights reserved. NiFi!
- 11. 12 Mappers HiveKa = Hive + Kafka Hive Storag e Handle r KafkaInputFor mat. getSplits() Kafka Get topic, partitions and offsets MapReduc e Setup Mappers Mappers KafkaRecordRea der Get data Avro SerDe Kafka Kafka
- 12. 13Click to enter confidentiality information
- 13. 14Click to enter confidentiality information
- 14. 15 Streaming ©2014 Cloudera, Inc. All rights reserved.
- 15. 16©2014 Cloudera, Inc. All rights reserved. Flume + Kafka = Flafka
- 16. 17 Sources Interceptors Selectors Channels Sinks Flume Agent How does work? Twitter, logs, webserver, Kafka… Mask, re-format, validate… DR, critical Memory, file, Kafka HDFS, Hbase, Solr, Kafka
- 17. 18 But I just want to get data from Kafka to Hbase / HDFS ©2014 Cloudera, Inc. All rights reserved.
- 18. 19 Channels Sinks Flume Agent Kafka Channel Kafka! HDFS, Hbase, Solr
- 19. 20 Kafka Channel Sources Interceptors Selectors Channels Flume Agent Twitter, logs, webserver, Kafka… Mask, re-format, validate… DR, critical Memory, file, Kafka
- 20. 21©2014 Cloudera, Inc. All rights reserved. SparkStreaming Single Pass Source RawInput DStream RDD Source RawInput DStream RDD RDD Filter Count Print Source RawInput DStream RDD RDD RDD Single Pass Filter Count Print Pre-first Batch First Batch Second Batch
- 21. 22©2014 Cloudera, Inc. All rights reserved. Storm Spout Source Split words bolts Split words bolts Spout Split words bolts Split words bolts Count Count Count Spout Layer Fan out Layer 1 Shuffle Layer 2
- 22. 23©2014 Cloudera, Inc. All rights reserved. Retro Thoughts
- 23. 24©2014 Cloudera, Inc. All rights reserved. • Data often has schema • At least it should • Kafka is unaware – which is good • Need capability to figure out schema for events • Without including it in every event Schema
- 24. 25©2014 Cloudera, Inc. All rights reserved. Kafka in Cloudera Manager
- 25. Questions?
Editor's Notes
- #3: This gives me a lot of perspective regarding the use of Hadoop
- #7: https://gist.github.com/gwenshap/9699072
- #9: Batch MapReduce job. Exactly once semantics. Run once every X minutes.
- #10: A - The setup stage fetches broker urls and topic information from ZooKeeper. B - The setup stage persists information about topics and offsets in HDFS for the tasks to read. C - The tasks read the persisted information from the setup stage. D - The tasks get events from Kakfa. E - The tasks write data to a temp location in HDFS in the format defined by the user defined decoder, in this case Avro formatted files. F - The tasks move the data in the temp location to a final location when the task is cleaning up. G - The task writes out audit counts on its activities. H - A clean up stage reads all the audit counts from all the tasks. I - The clean up stage reports back to Kakfa what has been persisted.
- #17: Kafka source + sink for Flume
- #18: Does not require programming.
- #20: Does not require programming.
- #21: Does not require programming.
- #22: MicroBatch stream processing framework. Basically Spark code executed in a slightly different context and some windowing functions.
- #23: Stream processing framework. Quite popular. Can be event-based or micro-batching (with Trident). Requires low level awareness of API.