Building Event-Driven Applications with Apache Kafka & Confluent Platform
3 likes2,538 views
The document discusses building event-driven applications with Apache Kafka and the Confluent platform, highlighting the challenges of integrating microservices. It explores various integration techniques such as using databases and RPC, emphasizing the advantages of event-driven architectures, particularly through logs and events. Additionally, it presents new features in Confluent Platform 5.2 aimed at enhancing usability and management in event-driven applications.
Building Event-Driven Applications with Apache Kafka & Confluent Platform
- 1. @gamussa | #eventdriven | @ConfluentINc Building Event-Driven Apps with Apache Kafka & Confluent Platform
- 2. @gamussa | #eventdriven | @ConfluentINc 2
- 3. @gamussa | #eventdriven | @ConfluentINc 3 I build highly scalable Hello World apps @kennybastani
- 9. @gamussa | #eventdriven | @ConfluentINc 9 Normal Applications (i.e., monoliths)
- 10. @gamussa | #eventdriven | @ConfluentINc 10 Monoliths are hard to think about
- 11. @gamussa | #eventdriven | @ConfluentINc 11 Monoliths are hard to change
- 13. @gamussa | #eventdriven | @ConfluentINc 13 Re- integration?
- 14. @gamussa | #eventdriven | @ConfluentINc 14 Re- integration
- 16. @gamussa | #eventdriven | @ConfluentINc 16 There are no good ways to integrate microservices
- 17. @gamussa | #eventdriven | @ConfluentINc 17 There are no-good ways to integrate microservices
- 18. @gamussa | #eventdriven | @ConfluentINc 18 Filesystem
- 19. @gamussa | #eventdriven | @ConfluentINc 19
- 20. @gamussa | #eventdriven | @ConfluentINc 20 Database
- 21. @gamussa | #eventdriven | @ConfluentINc 21
- 22. @gamussa | #eventdriven | @ConfluentINc 22 Integrating Microservices through the database ● “I have a database and I know how to use it.” ● Eventually causes services to co-mingle. ● Violation the «bounded context» ● Great to use inside a service boundary! ● Terrible for sharing data or negotiating change.
- 23. @gamussa | #eventdriven | @ConfluentINc 23 RPC
- 24. @gamussa | #eventdriven | @ConfluentINc 24
- 25. @gamussa | #eventdriven | @ConfluentINc 25 Integrating microservices via RPC ● Avoids problems of database integration ● Feels natural ● Aligns with the request/response paradigm ● Problem: cascading failures ● Question: how do you debug this system? ● Answer: you build a log. 🤔
- 26. @gamussa | #eventdriven | @ConfluentINc 26 Log4j type of log? 🤔
- 27. @gamussa | #eventdriven | @ConfluentINc 27 Events?
- 28. @gamussa | #eventdriven | @ConfluentINc 28 What’s an event?
- 29. @gamussa | #eventdriven | @ConfluentINc A shared narrative describing the evolution of the business over time 29
- 30. @gamussa | #eventdriven | @ConfluentINc A combination of: Notification State transfer 30
- 31. @gamussa | #eventdriven | @ConfluentINc Also, events are immutable. 31
- 32. @gamussa | #eventdriven | @ConfluentINc 32 Core abstraction ● DB - table ● Hadoop - file ● Kafka - ?
- 33. @gamussa | #eventdriven | @ConfluentINc 33 LOG
- 34. @gamussa | #eventdriven | @ConfluentINc 34 A Rich Microservices Application
- 35. @gamussa | #eventdriven | @ConfluentINc 35
- 36. @gamussa | #eventdriven | @ConfluentINc 36 Ratings Service operating on the event stream Order Requested ratings raw-rating Ratings Service KStreams API used to count and sum ratings as they arrive Browser Webserver KAFKA ratings counts ratings sums ratings
- 37. @gamussa | #eventdriven | @ConfluentINc 37 Schema Management Order Requested ratings raw-rating Ratings Service Browser Webserver KAFKA ratings counts ratings sums ratings Schema Registry KStreams API used to count and sum ratings as they arrive
- 38. @gamussa | #eventdriven | @ConfluentINc 38 Access legacy relation data Order Requested ratings raw-rating Ratings Service Browser Webserver KAFKA ratings counts ratings sums ratings Schema Registry Connect Movies KAFKA KStreams API used to count and sum ratings as they arrive
- 39. @gamussa | #eventdriven | @ConfluentINc 39 Materialize tables inside the app Order Requested ratings raw-rating Ratings Service Browser Webserver KAFKA ratings counts ratings sums ratings Schema Registry Connect Movies KAFKA Lookup table created inside the Rating Service Movies
- 40. @gamussa | #eventdriven | @ConfluentINc 40 Create a new table, persist it to the log Order Requested ratings raw-rating Ratings Service Browser Webserver KAFKA ratings counts ratings sums ratings Schema Registry Connect Movies KAFKA Movies rated-movies rated-movies Writable table created for rated movies
- 41. @gamussa | #eventdriven | @ConfluentINc 41 Create a new table, persist it to the log Order Requested ratings raw-rating Ratings Service Browser Webserver KAFKA ratings counts ratings sums ratings Schema Registry Connect Movies KAFKA Movies rated-movies rated-movies TRANSACTION
- 42. @gamussa | #eventdriven | @ConfluentINc 42 Hydrate a materialized view Order Requested ratings raw-rating Ratings Service Browser Webserver KAFKA ratings counts ratings sums ratings Schema Registry Connect Movies KAFKA rated-movies an embedded database CDC
- 43. @gamussa | #eventdriven | @ConfluentINc 43 What’s new in Cp 5.2?
- 44. @gamussa | #eventdriven | @ConfluentINc 44 Message browser enhancements Why Makes reading messages easier for application development & troubleshooting Features ● Feed auto-load, scroll, pause, and resume ● Seek to specific offsets (per partition) ● Card & table formats ● Message metadata
- 45. @gamussa | #eventdriven | @ConfluentINc 45 KSQL UI enhancements Why Improves KSQL usability & overall UX Features ● Feed pause & resume ● Card & table formats ● Clearer query status ● Improved error messaging ● Output metadata
- 46. @gamussa | #eventdriven | @ConfluentINc 46 Schema management Why Enables schema editing & policy management Features ● Create and edit schemas ● Validate schemas against compatibility policy ● Manage compatibility policy
- 47. @gamussa | #eventdriven | @ConfluentINc 47 Dynamic broker configuration Why Helps operators manage their brokers Features ● Change broker configs that don’t require restarts
- 48. @gamussa | #eventdriven | @ConfluentINc 48 License management Why Makes managing the subscription license easier and prevents unexpected expirations Features ● License checking ● License submission ● Pending expiration alerts (3mo out, 1mo out, weekly for the last month, daily for the last week) ● Expired license notice
- 49. @gamussa | #eventdriven | @ConfluentINc 49 Multi-cluster KSQL Why Runs queries on 1+ KSQL clusters Features ● Many-to-1 mapping for KSQL-to-Kafka Configuration ● confluent.controlcenter.ksql.<name of cluster>.url = <list of urls> ● confluent.controlcenter.ksql.<name of cluster>.advertised.url = <list of urls>
- 50. @gamussa | #eventdriven | @ConfluentINc 50 Multi-cluster Connect Why Run, monitor, and manage connectors on 1+ Connect clusters Features ● Many-to-1 mapping for Connect-to-Kafka Configuration ● confluent.controlcenter.connect.<name>. cluster = <list of urls>
- 51. @gamussa | #eventdriven | @ConfluentINc 51 My personal favorite
- 52. @gamussa | #eventdriven | @ConfluentINc 52 Kafka-topics use AdminClient API
- 53. CP 5.2 is Free forever on a single Kafka broker
- 55. 55