Apache kafka
Download as PPTX, PDF6 likes8,117 views
The document provides an introduction to Apache Kafka and its stream processing capabilities, emphasizing its advantages for real-time insights and data management. It discusses the architecture of Kafka, including producers, consumers, and the use of Kafka Connect for integrating various data sources. Additionally, it contrasts streaming and batch processing, highlighting the scalability and flexibility of stream processing applications.
Apache kafka
- 1. Apache Kafka Stream processing made easy
- 3. Streams 101 An introduction to stream processing
- 4. Transformation of a stream of data fragments into a continuous flow of information
- 5. Stream Processing Get real-time insights Lower processing latency Easier to test Easier to maintain Easier to scale Different way of thinking At-least-once vs exactly- once Time + -
- 6. Every company is already doing stream processing (more or less ... )
- 7. A Stream Key 1 -> value 1 Key 2 -> value 2 Key 1 -> value 3 ...
- 8. A Table +-------+---------+ | Key 1 | value 3 | | Key 2 | value 2 | +-------+---------+
- 9. A Table through time +-------+---------+ | Key 1 | value 1 | +-------+---------+ Timestamp 1 +-------+---------+ | Key 1 | value 1 | | Key 2 | value 2 | +-------+---------+ Timestamp 2 +-------+---------+ | Key 1 | value 3 | | Key 2 | value 2 | +-------+---------+ Timestamp 3 Timestamp ...
- 10. Let’s remove the redundancy
- 11. A Table through time as SETs SET(key1 -> value1) Timestamp 1 SET(key2 -> value2) Timestamp 2 SET(key 1 -> value3) Timestamp 3 Timestamp ...
- 12. SET(key1 -> value1) SET(key2 -> value2) SET(key1 -> value3) Changelog key1 -> value1 key2 -> value2 key1 -> value3 Stream
- 13. Tables are materialized views of streams
- 14. Why is this important?
- 15. Events used to manipulate core data. Today events are our core data Daan Gerits, 2012
- 16. Every stream process app is a combination of state and streams
- 17. Streaming vs batch is like agile vs waterfall but then for data.
- 18. Kafka A Stream Processing Platform
- 20. Kafka Platform Streams and Connect apps are just (java) apps Streams and Connect are libraries Can be deployed like any other (java) app Multiple instances of the same app can be launched Use tools like Mesos, kubernetes, Docker Swarm, ...
- 21. Batch Microbatch Flink Kafka Spark Storm / Heron Event
- 22. Build apps, not Jobs
- 24. Kafka Engine A message broker with a twist
- 25. Kafka Engine Producer ConsumerTopic message message Producer Producer Consumer Consumer
- 26. Kafka Engine Producer ConsumerTopic message message Producer Producer Consumer Consumer
- 27. Messages Contain byte arrays Have a Timestamp Key Value
- 28. Topics Are more like datastores Uses disk instead of memory Retains the messages Are partitioned and replicated
- 29. Wait?? … Disk??
- 30. Sequential disk access is fast* * Don’t believe me? Read http://kafka.apache.org/documentation#persistence
- 31. Producer Puts messages onto kafka Determines the partition to write to Can be implemented in many, many languages
- 32. Consumer Gets messages from kafka Can be grouped into Consumer Groups Allows for round robin message delivery Enables scaling of consumers Have a persisted offset per Consumer Group Stored in Zookeeper Or in Kafka
- 33. Kafka Engine Producer Consumer Topic Partition B Producer Consumer Topic Partition A
- 34. 100 000 msg/sec On a barely tweaked, 3 node cluster
- 35. 2 000 000 msg/sec On a heavily tweaked cluster https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
- 36. Kafka Connect Getting data in and out
- 37. A Simple and scalable way to get data in and out of topics
- 38. Kafka Connect Datasource Topic Kafka Connect Topic Datasink Kafka Connect Or
- 40. Kafka Connect MySQL ⬢ Salesforce ⬢ Redis ⬢ MQTT ⬢ InfluxDB ⬢ RethinkDB ⬢ HBase ⬢ Solr ⬢ Couchbase ⬢ Elasticsearch ⬢ Hazelcast ⬢ Google PubSub ⬢ HDFS ⬢ S3 ⬢ Splunk ⬢ Spooldir ⬢ JDBC ⬢ Syslog ⬢ Cassandra ⬢ Vertica ⬢ DB2 ⬢ Goldengate ⬢ Jenkins ⬢ PredictionIO ⬢ JMS ⬢ Twitter ⬢ Attunity ⬢ MSSQL ⬢ Postgres ⬢ DynamoDB ⬢ IRC ⬢ Kudu ⬢ Ignite ⬢ MongoDB ⬢ Bloomberg Ticker ⬢ FTP
- 41. Kafka Streams Processing streaming data
- 43. Kafka Streams KStream for a stream of data KTable to keep the latest value for each key KTable state is distributed across app instances Transform from streams to tables and tables to streams Choose which field to use as “timestamp”
- 44. TOPIC A TOPIC B TOPIC C Kafka Connect App Kafka Streams App Kafka Streams App Kafka Connect App TOPIC C TOPIC B TOPIC A
- 45. So how do you build solutions with this?
- 47. TOPIC A TOPIC B TOPIC C Sales JDBC Kafka Connect Top Products Ranker Emailer TOPIC C TOPIC B TOPIC A Low Stock Notifier Kafka Connect App Slack Poster
- 48. Proposal