CQRS and Event Sourcing: A DevOps perspective
1 like317 views
This document discusses challenges of deploying, monitoring, and debugging systems using CQRS and event sourcing from a DevOps perspective. It describes using a blue/green deployment approach, implementing consistent and usable logging, monitoring key metrics and data streams, and employing distributed tracing to identify the origin of requests in order to quickly debug problems. The overall goal is to build scalable, resilient, and automated systems while facilitating operational tasks through iterative improvements to tools and processes.
CQRS and Event Sourcing: A DevOps perspective
- 1. D o u b l e - c l i c k t o e d i t CQRS & EVENT SOURCING A DEVOPS PERSPECTIVE Stacey Watro @_w_stacey Maria Gomez @mariascandella
- 3. 3
- 4. WHAT IS THIS TALK ABOUT?
- 5. WHAT IS THIS TALK ABOUT? Dev-Ops Culture Challenges with CQRS and Event Sourcing Systems Deployment Monitoring DebuggingLogging
- 6. WHAT IS THIS TALK NOT ABOUT?
- 7. WHAT IS THIS TALK NOT ABOUT? Pros & Cons of CQRS and Event Sourcing When to use CQRS and Event Sourcing?
- 8. LET ME TELL YOU ABOUT THIS PROJECT
- 9. LET ME TELL YOU ABOUT THIS PROJECT 20M+ accounts Bunch of monoliths Unscalable architecture Snowflake servers The old world
- 10. LET ME TELL YOU ABOUT THIS PROJECT The new world
- 11. LET ME TELL YOU ABOUT THIS PROJECT The new world Scalable Resilient Extensible Automated Auditable
- 13. Client 1 …. Client N
- 14. Client 1 …. Client N Mainframe 1 Mainframe n ….
- 15. Client 1 …. Client N Third party 1 Third party N …. Mainframe 1 Mainframe n ….
- 16. Client 1 …. Client N Third party 1 Third party N …. Mainframe 1 Mainframe n …. Platform Domain1 … Domain1 Domain1 Domain1 Domain1 Domain2 Domain1 Domain1 DomainN
- 17. Client 1 …. Client N Third party 1 Third party N …. Mainframe 1 Mainframe n …. Platform Domain1 … Domain1 Domain1 Domain1 Domain1 Domain2 Domain1 Domain1 DomainN
- 18. DEFINITIONS
- 19. DEFINITIONS “Event Sourcing ensures that all changes to application state are stored as a sequence of events. Not just can we query these events, we can also use the event log to reconstruct past states, and as a foundation to automatically adjust the state to cope with retroactive changes.” Martin Fowler EVENT SOURCING Command and Query Responsibility Segregation (CQRS) is a pattern that segregates the operations that read data (queries) from the operations that update data (commands) CQRS
- 21. EVENT SOURCING $122.22 $50.04 $294.37 $33.12 $68.91 $100.02 = $668.68f(x) }- $300.37 $300.37
- 23. DEVOPS CULTURE Image Source: https://martinfowler.com/bliki/DevOpsCulture.html
- 25. MEET THE TEAM
- 26. HOW DO WE PUT THE PLATFORM INTO PRODUCTION? Client 1 …. Client N Third party 1 Third party N …. Mainframe 1 Mainframe n …. Platform Domain1 … Domain1 Domain1 Domain1 Domain1 Domain2 Domain1 Domain1 DomainN
- 27. REQUIREMENTS FOR DEPLOYMENTS Zero Downtime Safe & Repeatable Eliminate Event Lose
- 28. Blue
- 29. Blue
- 30. Blue Green
- 31. Blue Green
- 32. Green
- 35. Blue Green Event Bus Event Bus
- 36. Blue Green Event Bus Event Bus
- 37. Blue Green Event Bus Event Bus
- 38. Blue Green Event Bus Event Bus
- 39. Blue Green Event Bus Event Bus
- 40. Green Event Bus
- 41. Blue Green Event Bus Event Bus
- 42. Blue Green Event Bus Event Bus
- 43. CD TO THE RESCUE Build Unit Tests Deploy Integration Tests
- 44. CD TO THE RESCUE Build Unit Tests Deploy Integration Tests DB migrations from new code Deploy production code Smoke tests
- 45. CD TO THE RESCUE DB migrations from new Deploy production code Smoke tests
- 46. CD TO THE RESCUE DB migrations from new Deploy production code Smoke tests
- 48. WHAT HAPPENS WHEN SOMETHING GOES WRONG? Client 1 …. Client N Third party 1 Third party N …. Mainframe 1 Mainframe n …. Platform Domain1 … Domain1 Domain1 Domain1 Domain1 Domain2 Domain1 Domain1 DomainN
- 49. WHAT HAPPENS WHEN SOMETHING GOES WRONG? Client 1 …. Client N Third party 1 Third party N …. Mainframe 1 Mainframe n …. Platform Domain1 … Domain1 Domain1 Domain1 Domain1 Domain2 Domain1 Domain1 DomainN
- 52. WHAT IS DIFFERENT ABOUT LOGGING FOR CQRS & EVENT SOURCING? … … … Domain1 Domain1 Domain1 Domain1 Domain1 Domain2 Write Read Domain
- 53. WHAT SHOULD YOU LOG?
- 54. WHAT SHOULD YOU LOG?
- 55. WHY IS NAMING SO IMPORTANT? … … … }{name: Stacey Watro skill: awesome CustomerSkillAddedEvent
- 56. CONSISTENCY IS ALSO IMPORTANT }{name: Stacey Watro skill: awesome CustomerSkillAddedEvent }{line1: 660 56th st zipcode: 10019 AddressModifiedEvent
- 58. HOW DO WE KNOW WHEN PROBLEMS OCCUR IN THE SYSTEM?
- 59. REQUIREMENTS Information at a glance Dynamic Usability
- 61. What to monitor?
- 62. EXAMPLE
- 63. EXAMPLE
- 65. EXAMPLE Online marketplace 3M+ articles 10M+ users per month
- 66. EXAMPLE Service
- 70. EXAMPLE Data Stream A Data Stream B ServiceDB
- 71. EXAMPLE Service Data Stream A Data Stream B DB
- 72. EXAMPLE Service
- 73. EXAMPLE Service /health { “app” : “ok” }
- 74. EXAMPLE Service /health { “app” : “ok”, “database” : “ok” } DB
- 75. EXAMPLE Service /health { “app” : “ok”, “database” : “ok”, “dataStreamA” : “ok”, “dataStreamB” : ”ok” } DB Data Stream A Data Stream B
- 76. EXAMPLE Service /health { “app” : “ok”, “database” : “down”, “dataStreamA” : “ok”, “dataStreamB” : ”ok” } DB Data Stream A Data Stream B
- 79. METRICS
- 80. METRICS
- 81. METRICS
- 82. METRICS
- 84. SYNTHETIC MONITORING https://martinfowler.com/bliki/SyntheticMonitoring.html Staging
- 87. HOW DO WE DEBUG PROBLEMS?
- 88. REQUIREMENTS Find problems quickly Origin of the Request Usability
- 89. ABILITY TO FIND THE ORIGIN OF A REQUEST Client 1 …. Client N Third party 1 Third party N Mainframe 1 Mainframe N …. Platform Domain1 … Domain1 Domain1 Domain1 Domain1 Domain2 Domain1 Domain1 DomainN ….
- 90. ABILITY TO FIND THE ORIGIN OF A REQUEST Client 1 …. Client N Third party 1 Third party N Mainframe 1 Mainframe N …. Platform Domain1 … Domain1 Domain1 Domain1 Domain1 Domain2 Domain1 Domain1 DomainN ….
- 91. ABILITY TO FIND THE ORIGIN OF A REQUEST Client 1 …. Client N Third party 1 Third party N …. Mainframe 1 Mainframe N …. Platform Domain1 … Domain1 Domain1 Domain1 Domain1 Domain2 Domain1 Domain1 DomainN
- 92. ABILITY TO FIND THE ORIGIN OF A REQUEST Client 1 …. Client N Third party 1 Third party N …. Mainframe 1 Mainframe N …. Platform Domain1 … Domain1 Domain1 Domain1 Domain1 Domain2 Domain1 Domain1 DomainN TraceID TraceID TraceID TraceID TraceID
- 93. LOG AGGREGATOR
- 94. LOG AGGREGATOR
- 95. LOG AGGREGATOR
- 96. LOG AGGREGATOR
- 97. LOG AGGREGATOR
- 99. HAPPY TEAM
- 100. KEY LEARNINGS CQRS & Event Sourcing heavily influenced the deployment, logging, monitoring & debugging of this system Blue/Green was best given our constraints Log aggregation was not enough for us. Also needed monitoring and distributed tracing. Invest in making the system resilient and the tools accessible
- 101. For questions or suggestions: swatro@stridenyc.com mgomez@thoughtworks.com THANK YOU