Stateful Microservices with Apache Kafka and Spring Cloud Stream with Jan Svoboda | Kafka Summit London 2022
0 likes629 views
The document discusses the implementation of stateful microservices using Apache Kafka and Spring Cloud Stream, highlighting core concepts such as event sourcing, decoupling through CQRS, and scaling techniques. It includes a demo application, a bookmarks manager, to illustrate event-driven microservices principles and the architecture involved. The document emphasizes the benefits of transitioning from monolithic architectures to event-driven systems, as well as the use of Kafka Streams for data management.
Stateful Microservices with Apache Kafka and Spring Cloud Stream with Jan Svoboda | Kafka Summit London 2022
- 1. Stateful Microservices with Apache Kafka and Spring Cloud Stream Jan Svoboda, Solution Engineer jsvoboda@confluent.io
- 2. Agenda 2 Basics Recap Motivation for Event Driven Microservices Event Types EDM Components Demo Application Use Cases Application Architecture Stateful Microservices Concepts Event Sourcing using State Store Microservices decoupling (CQRS) State Store Scaling Smart Load Balancing
- 3. Jan Svoboda https://www.linkedin.com/in/svobodajan/ • Confluent, Solutions Engineer 2020 - now • Event Streaming, Event-Driven Microservices • Apache Kafka, Kafka Streams • Pivotal, Platform Architect, 2016 - 2020 • PaaS, Microservices • Cloud Foundry, Spring, Rabbit MQ • IBM, Solutions Architect, 2011 - 2016 • DevOps, Agile • Bluemix, Rational
- 4. Basics Recap Motivation for Kafka and Microservices
- 5. What is your Microservices Journey? 5 James Watters (Pivotal SVP), Kafka Summit 2019 Keynote, https://youtu.be/9I3CDfHKfNY?t=582 ● Stage 1 -> Strangling the Monolith ○ DB based integration ● Stage 2-> Provide Independent Data Sources ○ REST based integration ● Stage 3 -> Build Event Driven Microservices ○ Kafka based integration
- 6. What is in the Event Message? 6 Martin Fowler (Thoughtworks), 7th Feb 2017, https://martinfowler.com/articles/201701-event-driven.html ● Event Notification ○ simple notification of change in its domain ○ no response required ● Event Carried State Transfer ○ contains details of the changed data ○ repeated data ● Event Sourcing ○ contains change to the state of a system ○ rebuild state by reprocessing changes ● Command Query Responsibility Segregation ○ separate structures for reading and writing ○ combined with previous message patterns Neil Avery,, 31st Jan 2019, https://www.confluent.io/blog/journey-to-event-driven-part-1-why-e vent-first-thinking-changes-everything/
- 7. Event Driven Microservices Components 7 Apache Kafka Broker for EDM events Kafka Streams Stream Processing API RocksDB Persistence layer for Kafka Streams Spring Cloud Stream Stream abstraction layer Kafka Streams Binder SCS binding for Kafka topics
- 8. Demo Application Use Cases and Architecture
- 9. Demo Application - Use Cases 9 Simple Bookmarks Manager to demonstrate Event Driven Microservices principles. As a user I can: ● save a bookmark ● read list of my saved bookmarks ● change bookmark URL ● delete bookmark Code available at https://github.com/griga23/bookmarksDemo
- 10. Demo Application - Architecture and Dataflow • Spring Boot (2 jars) • Apache Kafka (2 topics) • Spring Cloud Streams + Kafka Streams Binder (RocksDB as a state store) Jan:Google,http://google.com Jan:Google,http://google.com JanGoogle: Google, http://google.com JanGoogle,http://google.com
- 11. Stateful Microservices Concepts Getting hands dirty...
- 12. 1. Event Sourcing using State Store My Data Liberation ● From Data sourcing to Event sourcing ● From synchronous Request/Reply to asynchronous Publish/Subscribe ● Store key/values in Kafka Streams State Store ● Kafka Streams Interactive Queries for data lookup https://docs.confluent.io/platform/current/streams/developer-guide/interactive-queries.html Spring Cloud Stream components ➔ StreamBridge class ➔ Java Lambda Functions for aggregating data to the Materialized State Store ➔ InteractiveQueryService class
- 13. 2. Microservices Decoupling (CQRS) Command Query Responsibility Segregation ● Pattern separates read and update operations for a data store. ● Implementing CQRS can maximize performance, scalability, and security. ● Allows a system to better evolve over time and prevents update commands from causing merge conflicts at the domain level. https://docs.microsoft.com/en-us/azure/ architecture/patterns/cqrs Kafka Streams State Store ➔ Change-log compacted topic ➔ RocksDB state.dir, applicationId, checkpoint ➔ Standby Replica
- 14. 3. Scaling – Stateful Microservices • Multiple State Stores with different partitions • HTTP REST calls to provide State Stores inter communication Jan:Google,http://google.com Jan:Yahoo,http://yahoo.com JanGoogle:Google,http://google.com JanYahoo:Yahoo,http://yahoo.com JanGoogle:Google,http://google.com JanYahoo:Yahoo,http://yahooo.com https://docs.confluent.io/platform/current/streams/developer-guide/interactive-queries.html#querying-remote-state-stores-for-the-entire-app Kafka Streams State Store ➔ configuration application.server ➔ methods getAllHostsInfo, KeyQueryMetadata, getHostInfo
- 15. 4. Smart Load Balancing – Routing to the Right State Store https://medium.com/bakdata/queryable-kafka-topics-with-kafka-streams-8d2cca9de33f • Spring Cloud Gateway for incoming HTTP traffic to redirect to proper State Store • State Store Host Awareness Jan:Google,http://google.com Jan:Google JanYahoo:Yahoo,http://yahoo.com Kafka Streams State Store ➔ getHostInfo