Brown bag eventdrivenmicroservices-cqrs
Download as PPTX, PDF0 likes197 views
Event driven microservices and CQRS were discussed. Event sourcing was presented as a solution to data consistency challenges with microservices and NoSQL databases. CQRS with materialized views was proposed to handle queries in an eventually consistent manner from the event store in event sourced applications. The benefits of this approach include easier data consistency, built-in audit log, and supporting analytics; while drawbacks include complexity and code duplication.
Brown bag eventdrivenmicroservices-cqrs
- 1. Event driven Microservices (μs) and CQRS Vikash Kodati 8/10/2016
- 2. AGENDA 4/6/2016 T-MobileConfidential2 • Why Event driven Microservice (μs) • Overview of event sourcing • Implementing Queries in an event sourced application
- 3. • Let’s imagine you are building a large, complex application, e.g., an online store 6/13/2016 T-MobileConfidential3
- 4. Monolithic Architecture 6/13/2016 T-MobileConfidential4 Store front UI Module WAR Catalog Module Reviews Module Orders Module Browser Client App HTML REST/JSON SQL Database
- 5. Limitations 6/13/2016 T-MobileConfidential5 • Monoliths = Trouble • Very intimidating to change things • State of art = deploy many times in a day = difficult with Monolith • The large the app = slower the IDE = slower container • Obstacle to development and ties to a technology choices at the start of the project. • RDBMS issues around Scale, Schema updates, OR mismatch • Don’t deal well with unstructured data
- 6. Microservice Architecture 6/13/2016 T-MobileConfidential6 Browser Mobile Device Store Front UI API Gateway Catalog Service Review Service Order Service …. Service Catalog Database Review Database Order Database …. Database HTML Rest Rest Rest
- 7. NoSQL db options 6/13/2016 T-MobileConfidential7 • Avoids the limitations of RDBMS • Text Search Solr/CloudSearch • Social (graph) data Neo4J • Highly distributed and available database Cassandra.
- 8. Drawbacks 6/13/2016 T-MobileConfidential8 • Complexity of developing a distributed system • Implementing inter-process Communication • Handling partial failure • Complexity of implementing business transactions that span multiple databases (without 2pc) • Complexity of testing a distributed system • Complexity of deploying and operating a distributed system • Managing the development and deployment of features that span multiple services
- 9. Issues to address 6/13/2016 T-MobileConfidential9 • How to deploy the services? • How do the services communicate? • How do client of application communicate with the services? • How to partition the system into services? • How to deal with distributed data management problems?
- 10. #1: SQL + Text search engine 6/13/2016 T-MobileConfidential10
- 11. #2: Update 2 entities in NoSQL db 6/13/2016 T-MobileConfidential11
- 12. #3: Indexing in Cassandra 6/13/2016 T-MobileConfidential12
- 14. 6/13/2016 T-MobileConfidential14 How to deal with distributed data management problems?
- 15. Solution: Event-Based Architecture 4/6/2016 T-Mobile Confidential15 • Components (e.g., services) publish events when state changes • Components subscribe to events • Maintain eventual consistency across multiple aggregates • Synchronize replicated data
- 16. #1: Shared Databases (today’s world) 6/13/2016 T-MobileConfidential16 Order Service Customer Service ….Service Tight Coupling Customer table Credit Limit Order table Order total ….. The Database Simple and ACID
- 17. #2: Database per services 6/13/2016 T-MobileConfidential17 Order Service Order table Order total Customer table Credit limit Customer service Order Database Customer Database
- 18. To maintain consistency a service must atomically publish an event whenever a domain object changes. 2PC (aka. Distributed transactions) is not viable choice for most modern applications. NOSQL DBs do not support it! 6/13/2016 T-MobileConfidential18
- 19. How to maintain data consistency without 2PC 6/13/2016 T-MobileConfidential19 Order service Place order() Customer service updateCreditlimit() Invariant: Sum(open order.total)<=Customer.creditLimit Order Total Customer Credit limit Order management Customer management Belongs to Has orders
- 20. Use event-driven, eventually consistent order processing 6/13/2016 T-MobileConfidential20 Order Service Customer Service Place Order Order Created Credit Reserved OR Credit Check Failed
- 21. BEFORE: Update State + Publish Events Two actions must be atomic NOW: Persist (and publish) Events Single action that is atomic 6/13/2016 T-MobileConfidential21
- 22. Designing Domain Events 6/13/2016 T-Mobile Confidential22 • For each domain object (i.e.,DDD aggregate) • Identify(State changing) domain event • Define Event Classes • Event Enrichment • ID: TimeUUID • For example, • Shopping Cart:ItemAddedEvent, ItemRemovedEvent, OrderPlacedEvent • Order:OrderCreated,OrderCancelled,OrderApproved,OrderReje cted, OrderShipped
- 23. Persists events NOT Current state 6/13/2016 T-Mobile Confidential23 Order Status 101 Accepted Order table
- 24. Persists events NOT Current state 6/13/2016 T-Mobile Confidential24 Entity Id Entity Type Event ID Entity Type Event data 101 Order 901 Order Created ….. 101 Order 901 Order Approved ….. 101 Order 901 Order shipped ….
- 25. Replay events to recreate state 6/13/2016 T-Mobile Confidential25 Order State Events OrderCreated(..) orderAccepted(…) OrderShipped(….) Periodically snapshot to avoid loading all events…
- 26. Aggregates : Command Events 6/13/2016 T-Mobile Confidential26 Command Aggregate Event
- 27. Request handling in an event sourced application 6/13/2016 T-Mobile Confidential27 HTTP Handler Order Event Store PastEvents = findEvents(entityID) New() applyEvents(PastEvents) NewEvenets=processCmd(someCmd) applyEvents(NewEvents) saveEvents(NewEvents) (optimistic locking) Order Service
- 28. Event store publishes events consumed by other services 6/13/2016 T-Mobile Confidential28 Event Store Event Subscriber Customer Subscribe(EventTypes) Publish(event) Publish(event) Update() Customer Service
- 29. Event store = database + message broker 6/13/2016 T-Mobile Confidential29 Event Store Save Aggregate Events Get Aggregate Events Subscribe to Events • Hybrid database and message broker • Implementations: • Home Grown/DIY • getEventStore.com by greg Young • http://eventuate.io by Chris Richardson
- 30. Benefits of event Sourcing 6/13/2016 T-Mobile Confidential30 • Solves data consistency issues in a microservice/NoSQL based architecture • Reliable event publishes events needed by predictive analytics etc, user notifications • Eliminates O/R mapping problem(mostly) • Reifies state changes: • Built in, reliable audit log • Temporal queries • Preserved history More easily implement future requirements
- 31. Drawbacks of event Sourcing 6/13/2016 T-Mobile Confidential31 • Requires application rewrite • Weird and unfamiliar style of programming • Events = a historical record of your bad design decisions • Must detect and ignore duplicate events: • Idempotent event handlers • Track most recent event and ignore old ones • Querying the event store can be challenging • Some queries might be complex/inefficient e.g. accounts with balance> x • Event store might only support lookup of events by entity id • Must use command Query Responsibility Segeration(CQRS) to handle queries application must handle eventually consistent data
- 32. Anatomy of a microservice 6/13/2016 T-MobileConfidential32 Microservice Event Store Messages AdapterHTTP Adapter HTTP Request Messages Requests Aggregate Aggregate Event AdapterEvents Events Cmd Cmd
- 33. Implementing queries in an event sourced application 4/6/2016 T-Mobile Confidential33
- 34. Find recent, Valuable Customers 6/13/2016 T-Mobile Confidential34 SELECT * FROM CUSTOMER c, ORDER o WHERE c.id = o.ID AND o.ORDER_TOTAL > 100000 AND o.STATE = 'SHIPPED‘ AND c.CREATION_DATE > ? Customer Service Order Service What if sourcing is used?
- 35. Command query Responsibility Segregation(CQRS) 6/13/2016 T-Mobile Confidential35 Application Logic Commands Queries
- 36. Command query Responsibility Segregation(CQRS) 6/13/2016 T-Mobile Confidential36 Aggregate Command Side Materialized View Query Side Event Store Commands Queries Events Events
- 37. Command query Responsibility Segregation(CQRS) 6/13/2016 T-Mobile Confidential37
- 38. Benefits and Drawbacks of CQRS 6/13/2016 T-Mobile Confidential38 Benefits Drawbacks • Necessary in an event sourced architecture • Separation of concerns = simpler command and query models • Support multiple denormalized views • Improve Scalability and performance • Complexity • Potential code duplication • Replication lag/eventually consistent view
- 39. Summary 6/13/2016 T-Mobile Confidential39 • Use microservices to accelerate development • Use an event-driven architecture to maintain data consistency • Implement an event-driven architecture using event sourcing • Use CQRS to implement materialized views for queries
Editor's Notes
- #2: Encourage interactive session Informal discussion Eat lunch My Goal is to keep us all on the same page at a conceptual level. Please stop me and ask questions
- #3: Event sourcing and Command Query Responsibility Segregations Great ways to build microservices
- #5: Imagine our Apps: Monolith We have backend services Deploy: pkg it in tomcat = war file Backend : RDMS ACID Simple way of building apps (dev,test, deploy at scale) Any changes applied atomically Scale out by running multiple copied behind load balancer Adoption new technologies
- #6: X Y Z scalaing Z = sharding at db level Y: functional decomp X; Traditional scale out
- #7: Distinct functional area is now a standalone service with its own database Helps with scalability We can use no-sql db
- #8: We end up with a polyglot persistence architecture.
- #9: How do we bridge the gap between the current realities and our new MISSION? Make them granular
- #10: Distributed data management is the issue to solve.
- #11: Elastic search in our app Update rdbms and how to keep the search in sync?
- #12: Update two document how do I do that w/o real transactions
- #13: How to keep the index tables of Cassandra in sync with the main table. Without a transactionally consistent manner. How to keep it consistent w/o 2PC
- #14: Talk about eventual consistency This concept is also known as BASE (as opposed to ACID) – Basically Available, Soft State, Eventually Consistent ACID (Atomicity, Consistency, Isolation, Durability)
- #15: Distributed data management is the issue to solve.
- #16: Subscribers = Other interested parties = they update their data to reflect the change. Moving from 2 PC to event driven approach.
- #17: Atomicity, Consistency, Isolation, Durability BASE: Basically available soft state eventually consistent
- #18: When looking to split a large application into parts, often management focuses on the technology layer, leading to UI teams, server-side logic teams, and database teams. We can relate this to: Applications having dev and Op responsibilities under different leaderships Deloitte team doing the dev all by themselves
- #19: How do we atomically publish and update a datastore? Message broker does not support 2PC
- #22: How do we atomically publish and update a datastore? Message broker does not support 2PC
- #23: Has a huge impact on architecture. Monolith has this huge single database containing all the data for all business functions. MS should be responsible for its own data and its peristence. Removes integrated at db level Use data store that makes sense for the problem You may never talk to another service’s datastore. You may only talk to another service via its API Choice of db tech will be upto the service group Including languages. Event oriented architecture. What if Service-A needs the data of Service-B How a service landscape is governed and also in terms of data management
- #24: Don’t persist state Persist events
- #25: One table to rule all domain objects.!
- #34: Again a set of statements.
- #35: Divide into two modules: Commands (updates) and Queries ()
- #36: Has a huge impact on architecture. Monolith has this huge single database containing all the data for all business functions. MS should be responsible for its own data and its peristence. Removes integrated at db level Use data store that makes sense for the problem You may never talk to another service’s datastore. You may only talk to another service via its API Choice of db tech will be upto the service group Including languages. Event oriented architecture. What if Service-A needs the data of Service-B How a service landscape is governed and also in terms of data management
- #37: View Store = MaterializedView Mongo Neo4J Cassandra or Mysql Text Queries = Cloud Search
- #38: View Store = MaterializedView Mongo Neo4J Cassandra or Mysql Text Queries = Cloud Search
- #39: How do we bridge the gap between the current realities and our new MISSION? Make them granular
- #40: How do we bridge the gap between the current realities and our new MISSION? Make them granular