"Database isolation: how we deal with hundreds of direct connections to the database", Mykhailo Kratiuk .pdf
0 likes179 views
What can go wrong if you allow each service to access the database directly? In a startup, this seems like a quick and easy solution, but as the system scales, problems appear that no one could have guessed. In my talk, I'll share Solidgate's experience in transforming its architecture: from the chaos of direct connections to a service-based data access model. I will talk about the transition stages, bottlenecks, and how isolation affected infrastructure support. I will honestly show what worked and what didn't. In short, we will analyze the controversy of this talk.
"Database isolation: how we deal with hundreds of direct connections to the database", Mykhailo Kratiuk .pdf
- 2. About me Worked with dinosaurs, but fell in love with Ukrainian fintech Worked with dinosaurs, but fell in love with Ukrainian fintech Worked with dinosaurs, but fell in love with Ukrainian fintech 4+ years of developing cloud and self-hosted products 4+ years of developing cloud and self-hosted products 4+ years of developing cloud and self-hosted products Platform Engineer in past Platform Engineer in past Platform Engineer in past Backend Software Engineer at Core Team, Solidgate Backend Software Engineer at Core Team, Solidgate Backend Software Engineer at Core Team, Solidgate
- 3. About Solidgate 7+ years online 7+ years online 7+ years online 80 engineers 80 engineers 80 engineers 4.5× growth in team size since 2020 4.5× growth in team size since 2020 4.5× growth in team size since 2020 Сard acquiring and local payment methods across the globe Сard acquiring and local payment methods across the globe Сard acquiring and local payment methods across the globe 60 SW engineers, 20 Infra + Data engineers + AQA 60 SW engineers, 20 Infra + Data engineers + AQA 60 SW engineers, 20 Infra + Data engineers + AQA
- 5. Engineering as it was 3 engineering teams 3 engineering teams 3 engineering teams 5-12 engineers perteam 5-12 engineers perteam 5-12 engineers perteam 2 services each team owned 2 services each team owned 2 services each team owned 5+ databases 5+ databases 5+ databases
- 6. System as it was Merchant Back-office Finance System Card Gate Alt Gate Database Database Database
- 8. Simplicity and speed of implementation Simplicity and speed of implementation Simplicity and speed of implementation Lower overhead – fewer network calls and intermediate layers Lower overhead – fewer network calls and intermediate layers Lower overhead – fewer network calls and intermediate layers Query flexibility without contract limitations Query flexibility without contract limitations Query flexibility without contract limitations Independence from other services Independence from other services Independence from other services Benefits of shared databases
- 10. E-commerce My name is Command Query Responsibility Segregation Pattern but friends call me CQRS
- 11. My name is Command Query Responsibility Segregation Pattern
- 12. Engineeringasitisnow 11 engineering teams 11 engineering teams 11 engineering teams 3-8 engineers perteam 3-8 engineers perteam 3-8 engineers perteam 7 services perteam 7 services perteam 7 services perteam 50+ databases 50+ databases 50+ databases
- 14. Once upon a time read/write
- 15. Once upon a time read/write read
- 17. Inconsistency of data structures across services Inconsistency of data structures across services Inconsistency of data structures across services Changes by one service can unexpectedly break others Changes by one service can unexpectedly break others Changes by one service can unexpectedly break others Difficulty in coordinating and rolling back schema modifications Difficulty in coordinating and rolling back schema modifications Difficulty in coordinating and rolling back schema modifications Increased risk of data corruption or loss during schema changes Increased risk of data corruption or loss during schema changes Increased risk of data corruption or loss during schema changes Database schema changes
- 18. Custom validation and data formatting logic for each service Custom validation and data formatting logic for each service Custom validation and data formatting logic for each service One service may update a field in a way that makes it unreadable by another service One service may update a field in a way that makes it unreadable by another service One service may update a field in a way that makes it unreadable by another service Data that once passed validation in the first service can no longer be considered valid if another service modifies them directly Data that once passed validation in the first service can no longer be considered valid if another service modifies them directly Data that once passed validation in the first service can no longer be considered valid if another service modifies them directly Validation and data transformation
- 19. Problems caused by the lack of database isolation Transaction and consistency issues; Transaction and consistency issues; Transaction and consistency issues; Violation of microservice independence Violation of microservice independence Violation of microservice independence Risk of implicit “agreements” at the database level Risk of implicit “agreements” at the database level Risk of implicit “agreements” at the database level Data security concerns Data security concerns Data security concerns Scalability challenges Scalability challenges Scalability challenges
- 20. Howto make it right?
- 21. ANALYSIS
- 22. Decompose by business context
- 23. Breakdown by entities Owner service DB Entity Client services Estimated cost Notes Jira issue merchant configuration cardgate config shop HUB (old) cardgate finance system ETA ??? ~3 weeks ~1 week Remove old HUB will do the job A lot of usages via ORM relations • use event stream to get shops • store shops in operations DB (as replicated data) Migrate from shops to channels Note: depends on channels isolation Note2: PHP code can remain unchanged due to further decommissioning.k merchant configuration cardgate config currency cardgate finance system reconciliation- kotlin currency- convertor (go) ~0.5 week ~0.5 week ETA ??? ~0.5 week Use merchant configuration API. Use merchant configuration API. We already have data replicated to Redshift, but its sync period is 1 hour which doesn’t cover some merchants' requirements. We have 2 solutions here: Force merchants to use these reports even with 1-hour delay and propose webhooks as a way of getting fresh updates. Tune the data pipelines to minimize delay to minutes (NordStream). Use merchant configuration API. cardgate cardgate operations PSP transaction search ~2 weeks Estimation includes moving to event streaming and API calls instead of direct DB queries. It doesn’t include the replicating data in the search’s own DB.
- 24. Patterns forDB decomposition Change Data Ownership Change Data Ownership Change Data Ownership Database Wrapping Service Database Wrapping Service Database Wrapping Service Monolith as data access layer Monolith as data access layer Monolith as data access layer Database View Database View Database View Synchronize Data in Application Synchronize Data in Application Synchronize Data in Application
- 25. DatabaseView
- 29. Impl. API Development on the Service- owner side: API / Event-streaming with all the necessary data. Impl. API Development on the Service- owner side: API / Event-streaming with all the necessary data. 1 Impl. API Development on the Service- owner side: API / Event-streaming with all the necessary data. Acquire The client service implements a component that uses the API provided by the service owner, while continuing to make direct database queries in parallel. Acquire The client service implements a component that uses the API provided by the service owner, while continuing to make direct database queries in parallel. 2 Acquire The client service implements a component that uses the API provided by the service owner, while continuing to make direct database queries in parallel. Dark read The client service team tests functionality, absence of bugs, and changes in performance. They simultaneously compare results obtained via the API and direct database queries. Dark read The client service team tests functionality, absence of bugs, and changes in performance. They simultaneously compare results obtained via the API and direct database queries. 3 Dark read The client service team tests functionality, absence of bugs, and changes in performance. They simultaneously compare results obtained via the API and direct database queries. Switch Development of a switching mechanism: all traffic at once, or percentage-based, or by specific clients. Switch Development of a switching mechanism: all traffic at once, or percentage-based, or by specific clients. 4 Switch Development of a switching mechanism: all traffic at once, or percentage-based, or by specific clients. Clean-up The client service completely removes all direct queries to the external database. Clean-up The client service completely removes all direct queries to the external database. 5 Clean-up The client service completely removes all direct queries to the external database. Client service switchworkflow
- 30. Synchronize Data in Application
- 31. Copy original DB Copy original DB Copy original DB Write to both, read from original Write to both, read from original Write to both, read from original Write to both, read from new Write to both, read from new Write to both, read from new Clean up Clean up Clean up Synchronize Data in Application
- 32. Copy original DB Copy original DB Copy original DB Write to both, read from original Write to both, read from original Write to both, read from original Write to both, read from new Write to both, read from new Write to both, read from new Clean up Clean up Clean up Synchronize Data in Application
- 33. Copy original DB Copy original DB Copy original DB Write to both, read from original Write to both, read from original Write to both, read from original Write to both, read from new Write to both, read from new Write to both, read from new Clean up Clean up Clean up Synchronize Data in Application
- 34. Copy original DB Copy original DB Copy original DB Write to both, read from original Write to both, read from original Write to both, read from original Write to both, read from new Write to both, read from new Write to both, read from new Clean up Clean up Clean up Synchronize Data in Application
- 35. Use new API Keep querying DB Client service dev team during the transition
- 36. Chargebacks Isolation Consistency Req.: High RPS: 0.03 Latency: 2s Consistency Req.: High RPS: 0.03 Latency: 2s 01 Occasional READ Consistency Req.: High RPS: 0.03 Latency: 2s Consistency Req.: Eventual RPS: 5.6 Latency: 60ms Consistency Req.: Eventual RPS: 5.6 Latency: 60ms 02 Intensive READ Consistency Req.: Eventual RPS: 5.6 Latency: 60ms
- 37. PACELC design principle … in case of network partitioning (P) in a distributed computer system, one has to choose between availability (A) and consistency (C) (as per the CAP theorem), but else (E), even when the system is running normally in the absence of partitions, one has to choose between latency (L) and loss of consistency (C). Wikipedia, 2025 Tradeoff: Latency vs. availabity Tradeoff: Latency vs. loss of Consistency PA EL PC EC Partitioned? (E) No (availability suffers) (P) Yes (latency suffers) A L C C
- 38. 01 - Occasional Read - PC/EL Tradeoff: Latency vs. availabity Tradeoff: Latency vs. loss of Consistency EL PC Partitioned? (E) No (availability suffers) (P) Yes (latency suffers) We call the API each time to fetch the latest data, thereby increasing Latency. We sacrifice Availability to remain Consistent and avoid making decisions based on stale data. L C
- 39. 02-IntensiveRead-PA/EC Tradeoff: Latencyvs. availabity Tradeoff: Latencyvs. loss of Consistency EC PA Partitioned? (E) No (availability suffers) (P) Yes (latency suffers) We use a local snapshot of the data, thereby choosing loss of Consistency over Latency. We continue processing requests as usual, remaining Available. C A
- 40. Consequences ofisolation for the infrastructure
- 41. Distributed load across RDS instances Distributed load across RDS instances Distributed load across RDS instances Increased scalable Event-Driven interactions Increased scalable Event-Driven interactions Increased scalable Event-Driven interactions Improved deployment stability Improved deployment stability Improved deployment stability Enhanced data security Enhanced data security Enhanced data security Consequently, higher infrastructure maintenance costs Consequently, higher infrastructure maintenance costs Consequently, higher infrastructure maintenance costs Significant increase in the number of transport components in the architecture Significant increase in the number of transport components in the architecture Significant increase in the number of transport components in the architecture Increased latency across the entire system Increased latency across the entire system Increased latency across the entire system Reduced DB maintenance overhead Reduced DB maintenance overhead Reduced DB maintenance overhead Howhas the infrastructure changed?
- 42. Lesson learned
- 43. The data returned via API/Streaming is always limited, requiring repeated requests for additional information The data returned via API/Streaming is always limited, requiring repeated requests for additional information The data returned via API/Streaming is always limited, requiring repeated requests for additional information The database schema often prevents isolation from being implemented The database schema often prevents isolation from being implemented The database schema often prevents isolation from being implemented Every sprint now includes one or more tasks related to API expansion Every sprint now includes one or more tasks related to API expansion Every sprint now includes one or more tasks related to API expansion Idempotency is now required everywhere Idempotency is now required everywhere Idempotency is now required everywhere Lesson learned
- 44. is it really worth it?
- 45. Hell yeah
- 46. Efficient and sophisticated systems aren’t built overnight — they are the product of repeated failures, incremental optimizations, and relentless learning from mistakes.
- 47. Дякую за увагу! telegram: @Swoqe