Redis and Kafka - Simplifying Advanced Design Patterns within Microservices Architectures (Allen Terleto, RedisLabs) Kafka Summit 2020
2 likes5,635 views
The document discusses advanced microservices design patterns, focusing on how to effectively utilize Kafka and Redis in microservices architectures. Key concepts include characteristics of microservices, the challenges of data management, and various design patterns like event sourcing and CQRS. It also highlights the need for improved communication and collaboration within teams to enhance time-to-market.
Redis and Kafka - Simplifying Advanced Design Patterns within Microservices Architectures (Allen Terleto, RedisLabs) Kafka Summit 2020
- 1. Redis and Kafka Advanced Microservices Design Patterns Simplified ALLEN TERLETO SENIOR DIRECTOR, CUSTOMER SOLUTIONS AND FIELD ENGINEERING https://www.linkedin.com/in/allenterleto/ AUGUST 2020
- 2. 2 Explore demand and goals for Microservices 1 Microservices Architecture Define the characteristics of a Microservice 2 Quick introduction of Kafka and Redis 3 Advanced Microservices Architecture Patterns 4 +
- 4. What is holding back organizations from improving time-to-market? 4 Some Manager: We need more communication on and between our teams Jeff Bezos: “No. Communication is terrible”
- 5. Software Evolution from Monolithic to Microservices 5 Source: https://www.martinfowler.com/microservices/
- 6. Characteristics of Microservices Architecture 6 Source: https://www.martinfowler.com/microservices/ 1. Componentization via Services 2. Organized around Business Capabilities 3. Smart endpoints & dumb pipes 4. Decentralized Governance 5. Decentralized Data Management 6. Products not Projects 7. Infrastructure Automation 8. Design for failure 9. Evolutionary Design API Gateway Microservice Message Broker Microservice Microservice
- 7. There’s a lot to this.. so where do we begin? 7
- 8. There’s a lot to this.. so where do we begin? 8
- 9. Open Source In-Memory Database supporting a variety of data structures and data models optimized for real-time and high-throughput use-cases Open Source Stream Processing Platform supporting distributed messaging publish-subscribe data feeds for near- real-time and high-throughput use-cases 9
- 10. Open Source Redis Modules 10 For more info check out: https://redislabs.com/community/redis-modules-hub/
- 11. Beautiful Chaos known as Microservices Architecture 12
- 12. Design Pattern - Bounded Context | Domain Driven Design 13 RedisAIRediSearch risk:profile input:tensors output:tensors RedisBloom Authenticate Digital Identity user:profile behavioral:profile transaction:profile geolocation trusted:devices Probabilistic Fraud Detection Checkpoint Transaction Risk Scoring user:transcations Approve | Decline Payment Solution • Practice Domain-Driven Design (DDD) principles • Simplify complex domain models by dividing them into separate Bounded Contexts; manageable by small teams • Choose optimal data model based on each microservice’s data access patterns and SLAs (RediSearch, RedisBloom, RedisAI, etc.) Challenge • Efficient communication between software developers and domain experts • Smaller autonomous teams are more productive • Common terminology to establish requirements, acceptance criteria, test cases, etc. Bounded Context Bounded Context
- 13. Design Anti-Pattern – Two-Phase Commit Transactions 15 REDIS STREAMS API Gateway Microservice Microservice write write Challenge • Each microservice can have its own database however some business transactions will span multiple Business Contexts • In a Microservices Architecture, the Two-Phase Commit protocol is an anti-pattern, since one or more out of the hundreds or thousands of microservices can, at any time, become unavailable commit commitSolution • Embrace eventual consistency and change mental model from conforming to traditional ACID constraints • Apply appropriate design patterns – Message Broker, Saga, CQRS, Message Relay, Outbox, etc.
- 14. Design Pattern – Publish-Subscribe Message Broker 16 Kafka Message Broker RediSearch RedisBloom API Gateway Authenticate Digital Identity Probabilistic Fraud Detection Checkpoint Challenge • Two-Phase Commit protocol is an anti-pattern • Transactions will span multiple business contexts • Microservices can become unavailable at anytime Solution • Embrace event-driven architecture principles • Inter-service communication (events) asynchronously flow through a publish-subscribe Message Broker • Zero, one, or more microservices can subscribe to events that are relevant to their Business Context; each at their own pace • Microservices, recently recovered from unavailability, can catch up to all the events they missed while unavailable publish subscribe subscribe
- 15. Design Pattern – Choreography-based Saga (State Machine) 17 Kafka Message Broker RedisAIRedisBloom Probabilistic Fraud Detection Checkpoint Transaction Risk Scoring Approve | Decline Payment YesNo YesNo Challenge • Transactions will span multiple Business Contexts and the Two-Phase Commit protocol is an anti-pattern • Microservices can include decision-logic that impacts which downstream services are in the transactional scope Solution • Saga is a sequence of local transactions • Choreography-based Sagas are not centrally coordinated • Has roots in Finite-State Machine • Each local transaction updates its database and publishes an event to trigger the next local transaction • Sagas allow for unavailability along their Chain-of-Responsibility, rollback compensation for failures, and can replay the entire transaction
- 16. Design Pattern – Transactional Outbox | Message Relay 18 RediSearch Kafka Message Broker Authenticate Digital Identity RedisGears user:profile behavioral:profile transaction:profile geolocation trusted:devices Challenge • Writing atomically to a database and a message broker can add unnecessary code complexity • Message Broker unavailability would force decisions about each microservice’s trade-offs between availability and consistency Solution • Separate the concern of managing inter-service communication by using an event-driven Message Relay • RedisGears can be used to easily implement the Transactional Outbox pattern by listening for events within Redis databases and relaying event notifications (write-behind pattern) to a Message Broker for downstream inter-service communication • Redis-based microservices would avoid code complexity, concerns with Message Broker unavailability, and performance overhead
- 17. Design Pattern – Capturing Telemetry 19 RedisGears RedisTimeSeries Analytics Dashboard RediSearch user:profile behavioral:profile transaction:profile geolocation trusted:devices RedisAI risk:profile input:tensors output:tensors RedisBloom user:transcations RedisGears RedisGears RedisTimeSeries Challenge • Telemetry is built on three pillars – metrics, logs, and tracing • Metrics are used for holistic observability of a distributed-system in real time • Metrics and tracing visualization often use an underlying data model called “time-series” Solution • RedisTimeSeries has a native time-series data model supporting high-volume inserts, low- latency reads, aggregated queries, search-by labels, downsampling, retention, indexing, etc. • Metrics can be collected at various levels – KPIs, application SLAs, infrastructure utilization, etc.
- 18. How can developers visualize time-series telemetry data? 20 Download for free: https://redislabs.com/redisinsight/
- 19. How can business analysts visualize time-series telemetry data? 21 For more info check out: https://github.com/mikhailredis/redis-pop-up-store and https://github.com/RedisTimeSeries/grafana-redis-datasource
- 20. Design Pattern – Event Sourcing 22 RedisAI Analytics Dashboard Redis Streams risk:profile input:tensors output:tensors RedisBloom user:transcations RedisGears RedisGears Kafka Message Broker Challenge • Observability, tracing, and auditing of Choreography-based Sagas is difficult across a decoupled chain of microservices Solution • Event Sourcing implies recording every microservice’s state change as an immutable event; acting as a source of truth • It is composed of a Message Broker and Event Store • Based on the ordered sequence of events persisted in the event store, a microservice can replay/rebuild its system state by reprocessing recorded events at any time • Redis Streams is an immutable in-memory append-only log data structure perfectly suited and used as an Event Store • Visualizing a stream of events can provide real-time observability of the system’s state and instant recovery/replay
- 21. How can we visualize a stream of events? 23
- 22. Design Pattern – CQRS (Command Query Responsibility Segregation) 24 API Gateway Approve | Decline Payment Payment History RedisCDC Redis Enterprise readwrite Challenge • Each microservice should choose an optimal data model for its unique data access patterns and SLAs • There is no single data model optimally suited for all needs Solution • CQRS implies maintaining separate data structures for writing (command) and reading (query) information • Allows a microservice with extreme durability SLAs to use a write-optimized database, while another microservice that requires access to the same data uses a read-optimized database • Managing eventual consistency between models is commonly handled using the Transaction Log Tailing pattern (a.k.a. Change-Data-Capture) or Message Relay pattern (covered earlier) • RedisCDC provides a seamless drop-in-solution to keep heterogenous databases in-synch with Redis
- 23. Design Pattern – Shared Data (Microservice Level) 25 API Gateway Approve | Decline Payment Payment History RedisCDC Redis Enterprise readwrite Clearing & Settlement read Challenge • Multiple microservices need to query the same data which could be sourced from multiple downstream microservices Solution • Implement the CQRS pattern to build a view-only database composed of replicated state- change events from one or more source databases • Redis is optimally suited as a view-only database, or microservices-cache, because it inherently guarantees atomicity for each operation, across multiple clients, and provides sub-millisecond latency at scale
- 24. Design Pattern – Shared Data (Global Level) 26 Redis Enterprise user:session oauth:token Kafka Message Broker RediSearch RedisBloom API Gateway Authenticate Digital Identity Probabilistic Fraud Detection Checkpoint Challenge • Session state, client authentication tokens, and/or other global data may need to be shared across hundreds of microservices • A shared database, or ephemeral cache, would couple all microservices together • A global database could become a single-point of failure user:transcations user:profile behavioral:profile transaction:profile geolocation trusted:devices Solution • API Gateway deploys its own isolated and highly-available database to store global data • Redis is used across thousands of microservices architectures in production to manage session state and client authentication tokens
- 25. Retrospective 27 Challenges • There’s more to microservices than breaking off chunks of code from a monolith • Data Management quickly becomes a pain point if best practices are not considered • Decentralization comes with trades-offs to complexity, technology sprawl, observability, etc. Solutions • Microservices design patterns help avoid complexity at scale • Redis supports multiple models optimally suitable to simplify their implementation • Kafka and Redis help turn Microservices Architecture’s beautiful chaos into controlled chaos
- 26. Check out my interview with Chris Richardson from 28 https://www.youtube.com/watch?v=q5Z8-cwGBXQ
- 27. Free Redis Microservices e-book for Developers 29 https://redislabs.com/docs/redis-microservices-for-dummies/
- 28. Check out these free resources to learn more about Redis 30 • http://redis.io/ • https://university.redislabs.com/ • https://docs.redislabs.com/latest/rs/ • https://www.youtube.com/c/RedisUniversity