Jay Kreps | Kafka Summit NYC 2019 Keynote (Events Everywhere) | CEO, Confluent
3 likes2,166 views
The document discusses the transformative impact of event streaming and data as a continuous flow on various industries, such as banking and retail, leveraging technologies like cloud, machine learning, and mobile applications. It introduces the event streaming paradigm, emphasizing its advantages over traditional data management methods, and details the functionality of the Confluent platform as a solution for managing real-time event data. The document highlights the widespread adoption of Kafka in production environments across many companies, noting its significance in enhancing customer experiences and operational efficiency.
Jay Kreps | Kafka Summit NYC 2019 Keynote (Events Everywhere) | CEO, Confluent
- 1. 1 Events Everywhere Jay Kreps CEO, Confluent
- 2. 22 The world is changing.
- 3. 33 The world is changing. Then Hardware product Up front purchase Opaque No data Now Hardware, Software & Global Internet Service On-demand Real-time visibility Built on a foundation of data Transportation
- 5. 55 This transformation is happening everywhere...
- 6. 6 Banking
- 7. 7 Retail
- 10. 1010 Transportation ETA Real-time sensor diagnostics Driver-rider match Banking Fraud detection Trading and risk systems Mobile applications / customer experience Retail Real-time inventory Real-time POS reporting Personalization Entertainment Real-time recommendations Personalized news feed In-app purchases
- 11. 1111
- 12. 1212 This is a fundamental paradigm shift... Infrastructure as code Data as a continuous stream of events Future of the datacenter Future of data Cloud Event Streaming
- 14. 1414 Two Problems in Application Infrastructure What’s the state of the world? What’s happening in the world? Solution: Databases Solution: Messaging, RPC, ETL, etc.
- 15. 1515 ETL/Data Integration Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency)
- 16. 1616 ETL/Data Integration Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency) Transient MessagesStored records
- 17. 1717
- 18. 1818 ETL/Data Integration Messaging Transient MessagesStored records ETL/Data Integration MessagingMessaging Batch Expensive Time Consuming Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency) Event Streaming Paradigm High Throughput Durable Persistent Maintains Order Fast (Low Latency)
- 19. 1919 To rethink data as not stored records or transient messages, but instead as a continually updating stream of events Event Streaming Paradigm
- 20. 20
- 21. 21 Event Streaming Platform Universal Event Pipeline Data Stores Logs 3rd Party Apps Custom Apps/Microservices
- 22. 22 Universal Event Pipeline Data Stores Logs 3rd Party Apps Custom Apps/Microservices Real-Time Inventory Real-Time Fraud Detection Real-Time Customer 360 Machine Learning Models Real-Time Data Transformation ... Contextual Event-Driven Apps Event Streaming Platform
- 23. 2323 Building an Event Streaming Platform with Kafka
- 24. 24 Consumer A Consumer B Reads Offset = 3 Offset = 6 Producers Writes 91 2 3 4 5 6 7 8
- 25. 25
- 26. 26
- 27. 27 Kafka Connect and Connector Ecosystem
- 28. 28 Stream Processing with KSQL Stream 01: Stream 02: Stream Table Processing streams of data to create more streams or tables QueryQuery Continuous Query
- 29. 29 KSQL: a few use cases A big step towards a streaming-first world: ● Real-time monitoring and analytics ● Streaming ETL, not Batch ETL ● Application development
- 30. 30
- 31. 31 Securing your event streams Authentication Data Confidentiality Authorization
- 32. 32 Topics PartitionsMessages / sec Brokers 10000000 25000 1000000 1500 Topics PartitionsMessages / sec Brokers 250000 500 25000 25 Topics PartitionsMessages / sec Brokers 1 5 300 3 Operations
- 33. 33 Confluent Products: Software and SaaS Confluent Platform The Enterprise Distribution of Apache Kafka Self-Managed Software Deploy on any platform, on-prem or cloud VM Confluent Cloud Apache Kafka Re-engineered for the Cloud Fully-Managed Service Available on the leading public clouds
- 34. 34 Announcing Confluent Platform 5.2 Accelerate Development ● Free forever Confluent Platform on single broker ● Librdkafka 1.0 ● New KSQL expressions Operate at Scale ● Control Center ○ Schema management ○ Dynamic broker config ○ Multi-cluster KSQL/Connect ○ Improved scalability Enable Hybrid Streaming ● Schema Registry replication from Confluent Platform to Confluent Cloud Apache Kafka 2.2.0
- 35. 3535 This is happening now.
- 36. 363636 Kafka is in production in tens of thousands of companies
- 37. 3737 70% 80% 90%10% 20% 50% 60%40%30% 60% of the Fortune 100 Now
- 38. 3838 Every customer experience Every bank transaction Every purchase Every car ride Every log in Events Everywhere
- 39. 39