Data pipeline with kafka
10 likes3,041 views
The document discusses the role of Apache Kafka in building data pipelines, highlighting its capabilities such as data ingestion, processing, and providing insights. It addresses various aspects of data flow, including Kafka's producer-consumer model, Kafka Connect API, and its application in handling real-time data. Additionally, it explores practical use cases of Kafka in data-driven product development and the benefits and challenges associated with its integration.
Data pipeline with kafka
- 1. Data Pipeline with Kafka Dr. Mole T.Y. WONG @ HK OSCON 2018 2018 / 06 / 16 - 17 1
- 2. whoami 2
- 3. 深入了解用戶行為，洞悉可行的改善方法 Understand our users. Provide actionable insights. Why HK01 Data Team | About
- 4. 以數據驅動產品方向 Data driven: steer our product direction. How HK01 Data Team | About
- 5. 數據：定義、收集、處理、洞見 Data: definition, ingress, process, insight. What HK01 Data Team | About
- 6. 6
- 7. Data-Driven Product Development 7 Browser - Page A Fashion Article Car Article Browser - Page B Fashion Article Stock Article Fashion Article Click eventClick event Traffic Source Analysis Click-Through Rate VS Pageview
- 8. Machine Learning Products 8 Collaborative filtering Image source: wikipedia User Reading History NLP Content-based Clustering Personalized Recommendation Feed
- 9. Outline ● Data pipeline - what is it? ● Kafka - roles in a data pipeline ● Other use cases of Kafka 9
- 10. API Gateway Kinesis S3 Redshift Spectrum AWS EMR WEB APP Apache Airflow Tracker Typical Data Pipeline Setup Metabase
- 11. Different Aspects of a Data Pipeline 11 WEB APP Tracker Data Ingress JS Library (WEB) Native Library (APP) Google Analytics Mixpanel Matomo (Piwik) Data Tracker ● Nature ○ Lightweight ○ Programmable ● Capability ○ Page view / Screen view ○ Custom events ○ Device identification ○ Session management
- 12. Infrastructure - AWS Kinesis - Google Pub/Sub - Apache Kafka API Gateway KinesisWEB APP Different Aspects of a Data Pipeline Tracker Data Infrastructure ● Main Roles ○ Buffering ○ Routing ○ Writing ● Characteristics ○ Multiple producers ○ Multiple consumers ○ Batch / Real-time Data Ingress JS Library (WEB) Native Library (APP) Google Analytics Mixpanel Matomo (Piwik) S3
- 13. Pre-processing Cleansing, Transformation, Data Warehousing S3 Redshift Spectrum AWS EMR Apache Airflow Different Aspects of a Data Pipeline Pre-processing ● Main Roles ○ Avoid direct querying raw data ○ Cleansing ○ ETL - Extract, Transform, Load ○ Scheduling ● Characteristics ○ Defining data sets ○ Time-frame-based queries
- 14. Pre-processing Cleansing, Transformation, Data Warehousing S3 AWS EMR Apache Airflow Application Dashboard, Reporting, Recommendation Engine, etc Redshift Spectrum Metabase Different Aspects of a Data Pipeline Application ● Main Roles ○ KPI VS Exploration ○ Operators VS Data Scientists ○ Planned VS Ad-hoc queries ● Characteristics ○ Production-grade data ○ Fast is a must
- 15. What is Kafka? https://kafka.apache.org/ Main Contributor: Gene NG 15
- 16. API Gateway WEB APP Tracker Metabase Data Pipeline with Kafka What is Kafka? Kafka Connect API Optional: data persists in S3 Kafka Connect API
- 17. API Gateway WEB APP Tracker Metabase Data Pipeline with Kafka What is Kafka? Kafka Connect API Optional: data persists in S3 Kafka Connect API
- 18. What is Kafka - terminology 18 Basics: Producer-Consumer Model while(1) { var e = produce_event() producer.produce(e) } Producer while(1) { var m = consumer.poll() consume_msgs(m) } Consumer
- 19. What is Kafka - terminology 19 Data Source Data Sink Kafka Connect API Kafka Connect API Connect API - For database / data source - Wrapped consumer & producer code - Nice thing: config file only!
- 20. What is Kafka - terminology 20 Data Source Data Sink Kafka Connect API Kafka Connect API Connect API - common connectors JDBC - MySQL, PgSQL S3 HDFS ElasticSearch
- 21. Kafka Connect 21 Data Source 1 Data Source 2 Data Source 3 Topic 1 Topic 2 Topic 3 Data Topic Model ● One-to-one (most common) Feature ● Autonomous ○ Loads data from sources whenever changes occur ● Storage ○ Writes data to the hosted HDD ○ Optional: sync data to S3
- 22. Kafka Connect - Source Property File Source: https://github.com/confluentinc/kafka-connect-jdbc/blob/master/config/source-quickstart-sqlite.properties 22 name=test-source-sqlite-jdbc-autoincrement connector.class=io.confluent.connect.jdbc.JdbcSourceConnector tasks.max=1 connection.url=jdbc:sqlite:test.db mode=incrementing incrementing.column.name=id topic.prefix=test-sqlite-jdbc- 1 2 3 4 5 6 7
- 23. Kafka Connect - Source Property File Source: https://github.com/confluentinc/kafka-connect-jdbc/blob/master/config/source-quickstart-sqlite.properties 23 name=test-source-sqlite-jdbc-autoincrement connector.class=io.confluent.connect.jdbc.JdbcSourceConnector tasks.max=1 connection.url=jdbc:sqlite:test.db mode=incrementing incrementing.column.name=id topic.prefix=test-sqlite-jdbc- 1 2 3 4 5 6 7 Topic naming convention - Prefix, and - DB table name How it works: - Each table implies one topic.
- 24. Kafka Connect 24 App Traffic Web traffic Data Source X All traffic Topic X Data Topic Model ● One-to-one (most common) ● Many-to-one
- 25. Kafka Connect 25 App Traffic Web traffic All traffic Schema-less ● Practically, you can write any types of data to the topic ● Most common choice is Avro Btw, Avro is an open-source library for schema specification and data serialization.
- 26. Kafka Connect 26 Data Source X Data Source Y Topic 1 Topic 3 Data Topic Model ● One-to-one (most common) ● Many-to-one ● One-to-many (most rare) Topic 2
- 27. Kafka Connect 27 Data Source X Topic 1 A more practical approach ● Use the same truth / data ● Consume by multiple guys! Consumer A Consumer B
- 28. Takeaway Messages ● Producers and consumers are actors ○ Push data to or pull data from Kafka ● Connect API automates the above actions ○ Work nicely with databases 28
- 29. 29 Data Pipeline Use Cases
- 30. Kafka as a data pipeline - data resiliency 30 Data Sink Kafka Connect API Kafka Internal - consumer’s state Consumer Topic Current Topic Position Your last-read position Lag behind by hello_world foobar 1080 1000 80 Kafka keeps track on consumer’s state: - A consumer can always resume work-in-progress - New consumer can start fresh!
- 32. Kafka as a data pipeline - Replace ETL 32 .filter() .map() .reduce() Function Use Case filter() Cleansing map() Transformatio n reduce() Aggregation Data Source Data Sink Kafka Connect API Kafka Connect API Kafka Stream API E L T .filter() .map() .reduce() JAVA Or Scala
- 34. Kafka - Streaming Example Code Source: https://kafka.apache.org/11/documentation/streams/tutorial 34 A New Topic is Created!
- 35. API Gateway WEB APP Tracker Metabase Data Pipeline with Kafka v2 Kafka - Replacing ETL Kafka Connect API Optional: data persists in S3 Kafka Connect API Kafka Stream API
- 36. 36 Experimenting Kafka in HK01
- 37. Experimenting Kafka in HK01 37 Metabase Kafka Connect API Kafka Connect API Kafka Stream API Apache Airflow External Services 1. Fetch data from an external service every hour.
- 38. Experimenting Kafka in HK01 38 Metabase Kafka Connect API Kafka Connect API Kafka Stream API Apache Airflow External Services 2. When data arrives at S3, Kafka takes it in.
- 39. Experimenting Kafka in HK01 39 Metabase Kafka Connect API Kafka Connect API Kafka Stream API Apache Airflow External Services 3. Stream API counts the number of new users using certain services.
- 40. Experimenting Kafka in HK01 40 Metabase Kafka Connect API Kafka Connect API Kafka Stream API Apache Airflow External Services 4. Connect API automatically updates the MySQL table. Metabase can display the updates.
- 41. Experimenting Kafka in HK01 41 Will display live dashboard during the talk
- 43. Message Queue | Source: https://www.confluent.io/blog/stream-data-platform-1/ 43 Highly-coupled: application & storage
- 44. Message Queue | Source: https://www.confluent.io/blog/stream-data-platform-1/ 44 As a message queue (MQ): - Pub/Sub - Transformation - Roles; clear that who are the sources and the sinks, respectively
- 45. Other Use Cases | Source: https://kafka.apache.org/uses 45 Things that we didn’t explore ● Logs aggregation ● Database log compaction ● Event sourcing
- 46. 46 Key Takeaways Pros 1. Kafka simplifies your ETL tasks. 2. Kafka unitifies your data storage. 3. Kafka gives your other possibilities.
- 47. 47 Key Takeaways Cons 1. Ops problems - scalability, HA, Zookeeper, etc. 2. Learning curve is *STEEP*.
- 48. We Love to Share Mole Wong Data Pipeline with Apache Kafka Day 1 17:40 Conference Hall 4-5 Ivan Ha React Async Rendering - Paradigm Shift After React Fiber Day 2 15:10 Conference Hall 6 Sunday Ku Video.js with HLS Day 2 12:30 Conference Hall 4-5