Now You See Me, Now You Compute: Building Event-Driven Architectures with Apache Kafka | Strata New York 2019
10 likes1,641 views
The document discusses building event-driven architectures using Apache Kafka, emphasizing the shift from traditional data handling to real-time event streaming for modern business needs. It outlines the key functionalities of an event streaming platform, including publishing, storing, and processing events, and highlights the benefits of high throughput and low latency. The document also addresses the importance of integrating real-time data with historical context for enhanced decision-making across various industries.
Now You See Me, Now You Compute: Building Event-Driven Architectures with Apache Kafka | Strata New York 2019
- 1. 1 Now You See Me, Now You Compute Building Event-Driven Architectures with Apache Kafka® Michael G. Noll Technologist, Office of the CTO, Confluent @miguno
- 3. 33 The world is changing.
- 4. 44 The New Business Reality Past Technology was a support function Innovation required for growth Running the business on yesterday’s data was “good enough” Today Technology is the business Innovation required for survival Yesterday’s data = failure. Modern, real-time data infrastructure is required.
- 5. 5 The Rise Of Event Streaming 60%Fortune 100 Companies Using Apache Kafka
- 7. 77 The world is changing. Then Hardware product Up-front purchase Opaque No data Now Hardware, Software, and Global Internet Service On-demand Real-time visibility Built on a foundation of data Transportation
- 9. 99 This transformation is happening everywhere
- 10. 1010 Banking
- 11. 1111 Retail
- 13. 1313 Cloud Machine Learning Mobile Event Streaming Rethink Decision Making Rethink User Experience Rethink Data Rethink Data Centers
- 14. 1414 Do you see me? Or: Would you blindly cross the street with traffic information that is 5 minutes old?
- 15. 1515 Transportation ETA Real-time sensor diagnostics Driver-rider match Banking Fraud detection Trading and risk systems Mobile applications / customer experience Retail Real-time inventory Real-time POS reporting Personalization Entertainment Real-time recommendations Personalized news feed In-app purchases
- 16. 1616 This is a fundamental paradigm shift... Infrastructure as code Data as continuous stream of events Future of the datacenter Future of data Cloud Event Streaming
- 18. 1818 Two Problems in Application Infrastructure What’s the state of the world? What’s happening in the world? Solution: Databases Solution: Messaging, RPC, ETL, etc.
- 19. 1919 ETL/Data Integration Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency)
- 20. 2020 ETL/Data Integration Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency) Transient MessagesStored records
- 21. 2121
- 22. 2222 ETL/Data Integration Messaging Transient MessagesStored records ETL/Data Integration MessagingMessaging Batch Expensive Time Consuming Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency) Event Streaming Paradigm High Throughput Durable Persistent Maintains Order Fast (Low Latency) Replay
- 23. 2323 To rethink data as neither stored records nor transient messages, but instead as a continuously updating Stream of Events Event Streaming Paradigm
- 24. 24 An Event records the fact that something happened 24 A good was sold An invoice was issued A payment was made A new customer registered
- 25. 25 A Stream represents history as a sequence of Events 25
- 26. 26 Events change the way we think 26 Monolithic Approach ● a database ● a variable ● a singleton ● an RPC Event-First Approach ● an event ● a stream ● a ‘data’ flow ● a stream processor Orders Service Payments Service Customers Service Orders Service Order Validation Service Tax ServiceEmail Notification ServiceDB request response event streams
- 27. 27 An Event Streaming Platform gives you three key functionalities 27 Publish & Subscribe to Events Store Events Process & Analyze Events
- 28. 2828
- 29. 2929 Event Streaming Platform Universal Event Pipeline Data Stores Logs 3rd Party Apps Custom Apps/Microservices ✓ Real-time but also persistent ✓ Elastic, scalable, reliable ✓ High throughput, low latency ✓ All apps and systems can now speak to each other for a complete view of data
- 30. 3030 Data Stores Logs 3rd Party Apps Custom Apps/Microservices Real-Time Inventory Real-Time Fraud Detection Real-Time Customer 360 Machine Learning Models Real-Time Data Transformation ... Event-Driven Apps, with Historical Context Universal Event Pipeline Event Streaming Platform ✓ Real-time but also persistent ✓ Elastic, scalable, reliable ✓ High throughput, low latency ✓ All apps and systems can now speak to each other for a complete view of data
- 31. 3131 Event-Driven App (Location Tracking) Only Real-time Events Messaging Queues and Event Streaming Platforms can do this Contextual Event-Driven App (ETA) Real-time combined with stored data Only Event Streaming Platforms can do this Where is my driver? When will my driver get here? Where is my driver? When will my driver get here? Why Combine Real-time With Historical Context? 2 min
- 32. 3232 The Event Streaming Platform is the Central Nervous System for today’s enterprises
- 33. 3333 Event Streaming Architectures How to Build With Kafka
- 34. 34 is a distributed event streaming platform Publish & Subscribe to Events Store Events Process & Analyze Events
- 35. 3535 01 Stream your data in real-time as Events 02 Store your Event Streams 03 Process & Analyze your Events Streams
- 36. 3636 01 Stream your data in real-time as Events From apps, microservices Use a Kafka producer client from your favorite language … and many more From/to other systems Use Kafka Connect plus a Connector for your system … and many more
- 37. 37 From apps, microservices: producer example Python App network write … and more
- 38. 38 From/to other systems: Kafka Connect and more Tip: Great option to gradually move workloads to Kafka while keeping production running!
- 39. 39 Kafka Connect ● Deployed standalone (development) or as a distributed cluster (production) ● Elastic service that works on bare-metal, VMs, containers, Kubernetes, ... ● The individual ‘Connector’ determines delivery guarantees, e.g., exactly-once VM VM
- 40. 40 Single Message Transforms for real-time ETL Ingress: modify an Event before storing ● Obfuscate sensitive information, e.g. PII ● Add origin of event for lineage tracking ● Remove unnecessary data fields ● … and more Egress: modify an Event on its way out ● Route high-priority events to faster stores ● Direct events to different Elasticsearch indexes ● Cast data types to match destination ● … and more { user: ab123, gender: female, ip: 1.2.3.95 } { user: ab123, ip: 1.2.3.XXX }
- 41. 41 Where SMTs live (ingress example) Data Source Kafka Connect SMT1 Converter transform serializes Source Connector generates events ... SMTn 10101 01010
- 42. 4242 Confluent Hub Discover Connectors, SMTs, and converters confluent.io/hub Easy installation Documentation, support, etc.
- 43. 43 02 Store your Event Streams 43 Kafka Cluster VM Storage is Distributed Scalable Reliable Durable Performant
- 44. 44 Topics PartitionsMessages / sec Brokers 10,000,000 25,000 1,000,000 1,500 Topics PartitionsMessages / sec Brokers 250,000 500 25,000 25 Topics PartitionsMessages / sec Brokers 1 5 300 3 Kafka scales from S to XXL
- 45. 4545 Event Streaming Paradigm Highly Scalable Durable Persistent Maintains Order Fast (Low Latency) Kafka = Source of Truth, stores every article since 1851 Denormalized into “Content View” Normalized assets (images, articles, bylines, etc.) https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/ Store your Events as long as you want
- 46. 46 Secure your Event Streams Authentication Data Confidentiality Authorization
- 47. 47 Achievement Data Unlocked: All Your Data Now Available as Streams of Events
- 48. 48 Consumer Bob Consumer Dina Reads Offset = 3 Offset = 7 Producer Alice Writes 91 2 3 4 5 6 7 8 Independent access to Event Streams
- 49. 49 03 Process & Analyze your Events Streams 49 With separate frameworks … and more With Streaming SQL KSQL streams With apps, microservices … and more Kafka consumer clients or
- 50. 50 CREATE STREAM fraudulent_payments AS SELECT * FROM payments WHERE fraudProbability > 0.8 ● You write only SQL ● No Java, Python, or other boilerplate to wrap around it! ● Create KSQL User Defined Functions in Java when needed ● All you need is Kafka KSQL
- 51. 51 Stream Processing with KSQL 4 Headless1 UI 2 CLI ksql> 3 API POST /query Pick your favorite interface
- 52. 52 Where KSQL lives VM network read/write Elastic & Scalable Fault-tolerant Exactly-once Kafka security Aggregations Windowing Streams & Tables KSQL Cluster
- 53. 53 Stream Processing with KSQL Stream 01 Stream 02 Stream 03 Table Process event streams to create new, continuously updated streams or tables QueryQuery Streaming Query CREATE TABLE OrderTotals AS SELECT * FROM ... EMIT CHANGES
- 54. 54 Stream Processing with KSQL Query tables in Kafka from other apps, similar to a relational database Table QueryQuery Pull Query SELECT * FROM OrderTotals WHERE region = ‘Europe’ Result Upcoming feature (KLIP-8)
- 55. 55 Query tables in Kafka from other apps, similar to a relational database Other Applications (Java, Go, Python, etc.) can directly query tables Result request-response via network (KSQL REST API) Table SELECT * FROM OrderTotals WHERE region = ‘Europe’ Stream Processing with KSQL Upcoming feature (KLIP-8)
- 56. 56 KSQL integrates with Kafka Connect Simplifies event streaming between Kafka and other systems CREATE SOURCE CONNECTOR my-postgres-jdbc WITH ( connector.class = "io.confluent.connect.jdbc.jdbcSourceConnector", connection.url = "jdbc:postgresql://dbserver:5432/my-db", ...); Upcoming feature (KLIP-7) controls controls
- 57. 57 KSQL example use case Creating an event-driven dashboard from a customer database customers table Kafka Connect is streaming change events Results are continuously updating Elasticsearch Aggregations are computed in real-time
- 58. 58 Kafka Streams ● You write standard Java or Scala applications to process your events ● The Kafka Streams library makes these applications: elastic, scalable, fault-tolerant, and more ● All you need is Kafka streams
- 59. 59 <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-streams</artifactId> <version>2.3.0</version> </dependency> Add as dependency to your Java/Scala app Writing a Kafka Streams application
- 60. 60 KStreams Application App instance 1 ... App instance n VM network read/write Elastic & Scalable Fault-tolerant Exactly-once Kafka security Aggregations Windowing Streams & Tables Where your Kafka Streams apps live
- 61. 61 Process event streams to create new, continuously updated streams or tables Orders Inventory Shipping Stream Processing with Kafka Streams apps Frontend Event-driven apps and services communicate through Kafka Reporting New apps can easily be added by tapping into existing event streams
- 62. 62 App instance 1 ... App instance n ResultTable Query your application’s tables and state from other apps Other Applications (Java, Go, Python, etc.) can directly query tables request-response via network (e.g. REST API) Stream Processing with Kafka Streams apps Reporting App
- 63. 63 is a distributed event streaming platform Publish & Subscribe to Events Store Events Process & Analyze Events
- 64. 6464 Where to go from here for more details on event-driven architectures with Kafka
- 65. 65 THANK YOU @miguno michael@confluent.io cnfl.io/meetups cnfl.io/blog cnfl.io/slack