SlideShare a Scribd company logo
qaware.de
Fully-managed Cloud-native Databases: The
path to indefinite scale
Dirk Kröhan
dirk.kroehan@qaware.de
2
Dirk Kröhan
Software-Architect
#qaware #mainz
#cloudnative #spring
#cleanarchitecture
QAware
3
The history of rail gauges: https://parovoz.com/spravka/gauges-standard.html
What does a horse have to do with a space shuttle?
QAware
4
Where do we actually store our application data?
QAware
MagentaTV Voice @ Deutsche Telekom
5
QAware
High-level architecture
6
CVI Event-History BI Analytics
Invoke API History API
WRITE
READ
READ
QAware
7
QAware
8
We want a scalable NoSQL database and
preferably fully managed!
QAware
9
Runs
on
Isn't there something provided by Azure?
QAware
Azure Cosmos DB
Azure Cosmos DB
11
Azure Cosmos DB is a globally distributed, multi-model database service offered by Microsoft Azure. It is
designed to provide high availability, scalability, and low-latency access to data for mission-critical
applications.
Cosmos DB is a NoSQL database, which means it can handle unstructured and semi-structured data types.
Cosmos DB supports multiple data models, including document, key-value, column-family, and graph, making
it suitable for a wide range of use cases.
QAware
Globally distributed
12
Quelle: Azure Portal
QAware
Multi-Model
13
■ API for NoSQL (native)
■ API for MongoDB
■ API for PostgreSQL (new)
■ API for Apache Cassandra
■ API for Apache Gremlin
■ API for Table
👍
QAware
Consistency Levels
14
https://learn.microsoft.com/en-us/azure/cosmos-db/consistency-levels#consistency-levels-and-latency
QAware
High Availability, Scalability, Low Latency
15
■ SLA with 99,999 % read availability
■ Scalability via guaranteed throughput
– Manual scaling
– Automatic instant scaling based usage
■ Low Read-/Write-Latency
– Guaranteed < 10ms at 99th percentile
– Average Read-Latency ~4 ms (read) / 5 ms (write) at 50th percentile
https://uptime.is/99.999
QAware
Request Unit as scaling and throughput unit
16
A point read for a 1-KB Item costs exactly 1
Request Unit.
point read: Reading a single Item per ID and
Partition-Key value
QAware
Scaling based on provisioned throughput
17
■ Throughput on database level
– Shared throughput across all containers
– No throughput guarantees for a specific container
■ Throughput on container level
– Exclusive throughput for a single container
If a database operation requires more RUs than the currently available throughput, rate limiting applies, i.e. the request is
rejected with a 429 response. The response contains hints as to when the request should be repeated.
The required RU throughput is always linked to the underlying physical partition.
–> Hot Partition
QAware
Indefinite scale?
19
SCALE
LIKE A BOSS
QAware
20
Quelle: https://tenor.com/view/yaburnt-joker-burn-burnward-calltheburnunit-gif-5274995
QAware
High scalability in the real world
Indefinite scaling!?
22
¹ You can increase Maximum RUs per container or database by filing an Azure support ticket.
QAware
Yes, but you have to know what you're doing!
23
Quelle:https://cdn.prod.www.spiegel.de/images/6c9a2605-8b97-4545-a815-926da609c698_w1600_r1.5_fpx58_fpy55.01.jpg
vs.
Quelle: https://de.wikipedia.org/wiki/Kombinationskraftwagen
QAware
Data modeling and partitioning is the key!
24
■ Scalability relies on the partitioning of data
■ Choosing the right partition keys is essential
■ A partition key can not be changed
■ The partition key defines a logical partition
■ Goal: All database operations should hit one
logical partition
QAware
Examples from our project
High level architecture
26
Azure
CosmosDB
Container: event
Touchpoints
APIs
ETL APIs
PDM APIs
Customer-
Care APIs
VLP APIs
QAware
Get cards
Get conversations
Cons
27
■ Event container stores two different types of items
■ Touchpoint APIs require only a small portion of the whole JSON data
– Get single card by ID
– Get cards paginated by size
– Get conversations paginated by time + various filter criteria
– Vote a single conversation
– Delete bulk
– Delete all + by filter criteria
■ Vote requires an update of an existing conversation item
■ Delete requires an update of an existing conversation to set a delete flag and to set sensitive data to NULL
QAware
Cons TL;DR
28
■ Database queries are expensive in terms of required database throughput
■ Database must be scaled higher to provide sufficient throughput
■ Higher code complexity
QAware
Goal: Separation of concerns
29
■ Store data that is needed from the customer's perspective separately from the data stored for analysis and debugging purposes
■ Reduce complexity
– Simplify the deletion process: Delete data that the customer sees immediately, delete remaining data asynchronously
QAware
Azure
CosmosDB
Container: event
Container: user_conversation
Consumer APIs of the Event-History
30
Touchpoints
APIs
ETL APIs
PDM APIs
Customer-
Care APIs
VLP APIs
QAware
RU Throughput before
31
QAware
RU Throughput after
32
~90% RU throughput savings!!
QAware
Some of the features
TTL-based Deletion
34
■ Globally on container level
■ Individually per Item via ttl property
– Global TTL still has to be activated,
at least with “-1” (Items do not expire)
QAware
ChangeFeed
35
QAware
A look outside the box:
AWS DynamoDB & Google Firestore
AWS DynamoDB
37
■ “Fully managed NoSQL database service that provides fast and predictable performance with seamless scalability”
■ Provides a single own API
■ Scaling is also based on provisioned throughput
– Autoscaling is possible
■ 2 Consistency Levels (eventually consistent, strongly consistent)
– Can be selected for each individual read access
■ Data is automatically replicated to multiple availability zones in an AWS region
– Global tables enable multi-region replication
QAware
AWS DynamoDB
38
■ Maximum item size: 400KB (CosmosDB 2MB)
■ Throughput is measured in read capacity unit (RCU) und write capacity unit (WCU)
– RCU: 1 strongly consistent / 2 eventually consistent read access per second for items up to 4KB
– WCU: 1 write access per second for items up to 1 KB
■ Throughput limits per partition: 3000 RCUs and up to 1000 WCUs
■ DynamoDB Streams is similar to CosmosDB ChangeFeed, but only for the last 24h. Contains all changes to an item including deletions!
■ Supports ACID transactions across multiple items, also across multiple tables
QAware
Google Firestore
39
■ “Serverless, maintenance-free document database that can be easily scaled to meet any need”
■ Automatic replication across multiple regions. Guaranteed availability of 99.999 %
■ Strong focus on App and Mobile development
– Live-synchronisation of changes
– Offline-Mode
■ Maximum item size: 1 MB
■ ACID transaction support
■ Automatic scaling
– Up to 1 million concurrent connections and 10000 writes/second
■ Pricing is mainly based on database operations (read, write, delete) plus some extra costs for bandwidth and storage
QAware
1. + 2. Juli 2024
https://www.kcdmunich.de/
qaware.de
QAware GmbH Mainz
Rheinstraße 4 C
55116 Mainz
Tel. +49 6131 21569-0
info@qaware.de
twitter.com/qaware
linkedin.com/company/qaware-gmbh
xing.com/companies/qawaregmbh
slideshare.net/qaware
github.com/qaware

More Related Content

PDF
GECon2017_High-volume data streaming in azure_ Aliaksandr Laisha
PDF
Arquitetura Hibrida - Integrando seu Data Center com a Nuvem da AWS
PDF
The Crown Jewels: Is Enterprise Data Ready for the Cloud?
PDF
Revolutionary Storage for Modern Databases, Applications and Infrastrcture
PPTX
Sql Start! 2020 - SQL Server Lift & Shift su Azure
PDF
Azure Cosmos DB - NoSQL Strikes Back (An introduction to the dark side of you...
PDF
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
PDF
Accelerated SDN in Azure
GECon2017_High-volume data streaming in azure_ Aliaksandr Laisha
Arquitetura Hibrida - Integrando seu Data Center com a Nuvem da AWS
The Crown Jewels: Is Enterprise Data Ready for the Cloud?
Revolutionary Storage for Modern Databases, Applications and Infrastrcture
Sql Start! 2020 - SQL Server Lift & Shift su Azure
Azure Cosmos DB - NoSQL Strikes Back (An introduction to the dark side of you...
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
Accelerated SDN in Azure

Similar to Fully-managed Cloud-native Databases: The path to indefinite scale @ CNN Mainz (20)

PDF
Capital One Delivers Risk Insights in Real Time with Stream Processing
PDF
Running Production CDC Ingestion Pipelines With Balaji Varadarajan and Pritam...
PDF
Surviving KubeDoom - Even demons can't kill an intrusion of CockroachDB!
PPTX
Ingestion and Dimensions Compute and Enrich using Apache Apex
PDF
Eliminating Volatile Latencies Inside Rakuten’s NoSQL Migration
PDF
Better, faster, cheaper infrastructure with apache cloud stack and riak cs redux
PDF
A Technical Deep Dive on Protecting Acropolis Workloads with Rubrik
PDF
AWS Database Services-Philadelphia AWS User Group-4-17-2018
PDF
To Serverless and Beyond
PDF
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)
PPTX
QCT Ceph Solution - Design Consideration and Reference Architecture
PPTX
QCT Ceph Solution - Design Consideration and Reference Architecture
PPTX
Optimizing Apache HBase for Cloud Storage in Microsoft Azure HDInsight
PDF
Aerospike Hybrid Memory Architecture
PPTX
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
PDF
Confluent Tech Talk Korea
PDF
Azure Data services
PDF
TechTalkThai-CiscoHyperFlex
PPTX
Westpac Bank Tech Talk 1: Dive into Apache Kafka
PDF
AWS re:Invent 2016 Recap: What Happened, What It Means
Capital One Delivers Risk Insights in Real Time with Stream Processing
Running Production CDC Ingestion Pipelines With Balaji Varadarajan and Pritam...
Surviving KubeDoom - Even demons can't kill an intrusion of CockroachDB!
Ingestion and Dimensions Compute and Enrich using Apache Apex
Eliminating Volatile Latencies Inside Rakuten’s NoSQL Migration
Better, faster, cheaper infrastructure with apache cloud stack and riak cs redux
A Technical Deep Dive on Protecting Acropolis Workloads with Rubrik
AWS Database Services-Philadelphia AWS User Group-4-17-2018
To Serverless and Beyond
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)
QCT Ceph Solution - Design Consideration and Reference Architecture
QCT Ceph Solution - Design Consideration and Reference Architecture
Optimizing Apache HBase for Cloud Storage in Microsoft Azure HDInsight
Aerospike Hybrid Memory Architecture
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Confluent Tech Talk Korea
Azure Data services
TechTalkThai-CiscoHyperFlex
Westpac Bank Tech Talk 1: Dive into Apache Kafka
AWS re:Invent 2016 Recap: What Happened, What It Means
Ad

More from QAware GmbH (20)

PDF
QAware_Mario-Leander_Reimer_Architecting and Building a K8s-based AI Platform...
PDF
Frontends mit Hilfe von KI entwickeln.pdf
PDF
Mit ChatGPT Dinosaurier besiegen - Möglichkeiten und Grenzen von LLM für die ...
PDF
50 Shades of K8s Autoscaling #JavaLand24.pdf
PDF
Make Agile Great - PM-Erfahrungen aus zwei virtuellen internationalen SAFe-Pr...
PDF
Down the Ivory Tower towards Agile Architecture
PDF
"Mixed" Scrum-Teams – Die richtige Mischung macht's!
PDF
Make Developers Fly: Principles for Platform Engineering
PDF
Der Tod der Testpyramide? – Frontend-Testing mit Playwright
PDF
Was kommt nach den SPAs
PDF
Cloud Migration mit KI: der Turbo
PDF
Migration von stark regulierten Anwendungen in die Cloud: Dem Teufel die See...
PDF
Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster
PDF
Endlich gute API Tests. Boldly Testing APIs Where No One Has Tested Before.
PDF
Kubernetes with Cilium in AWS - Experience Report!
PDF
50 Shades of K8s Autoscaling
PDF
Kontinuierliche Sicherheitstests für APIs mit Testkube und OWASP ZAP
PDF
Service Mesh Pain & Gain. Experiences from a client project.
PDF
50 Shades of K8s Autoscaling
PDF
Blue turns green! Approaches and technologies for sustainable K8s clusters.
QAware_Mario-Leander_Reimer_Architecting and Building a K8s-based AI Platform...
Frontends mit Hilfe von KI entwickeln.pdf
Mit ChatGPT Dinosaurier besiegen - Möglichkeiten und Grenzen von LLM für die ...
50 Shades of K8s Autoscaling #JavaLand24.pdf
Make Agile Great - PM-Erfahrungen aus zwei virtuellen internationalen SAFe-Pr...
Down the Ivory Tower towards Agile Architecture
"Mixed" Scrum-Teams – Die richtige Mischung macht's!
Make Developers Fly: Principles for Platform Engineering
Der Tod der Testpyramide? – Frontend-Testing mit Playwright
Was kommt nach den SPAs
Cloud Migration mit KI: der Turbo
Migration von stark regulierten Anwendungen in die Cloud: Dem Teufel die See...
Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster
Endlich gute API Tests. Boldly Testing APIs Where No One Has Tested Before.
Kubernetes with Cilium in AWS - Experience Report!
50 Shades of K8s Autoscaling
Kontinuierliche Sicherheitstests für APIs mit Testkube und OWASP ZAP
Service Mesh Pain & Gain. Experiences from a client project.
50 Shades of K8s Autoscaling
Blue turns green! Approaches and technologies for sustainable K8s clusters.
Ad

Recently uploaded (20)

PDF
Softaken Excel to vCard Converter Software.pdf
PDF
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
PPTX
ISO 45001 Occupational Health and Safety Management System
PPTX
Agentic AI : A Practical Guide. Undersating, Implementing and Scaling Autono...
PPTX
history of c programming in notes for students .pptx
PDF
Nekopoi APK 2025 free lastest update
PDF
top salesforce developer skills in 2025.pdf
PDF
How to Choose the Right IT Partner for Your Business in Malaysia
PDF
Why TechBuilder is the Future of Pickup and Delivery App Development (1).pdf
PDF
How to Migrate SBCGlobal Email to Yahoo Easily
PPTX
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
PPTX
ManageIQ - Sprint 268 Review - Slide Deck
PDF
medical staffing services at VALiNTRY
PDF
SAP S4 Hana Brochure 3 (PTS SYSTEMS AND SOLUTIONS)
PPTX
Odoo POS Development Services by CandidRoot Solutions
PPTX
VVF-Customer-Presentation2025-Ver1.9.pptx
PPTX
Operating system designcfffgfgggggggvggggggggg
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
PDF
Which alternative to Crystal Reports is best for small or large businesses.pdf
PPTX
CHAPTER 2 - PM Management and IT Context
Softaken Excel to vCard Converter Software.pdf
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
ISO 45001 Occupational Health and Safety Management System
Agentic AI : A Practical Guide. Undersating, Implementing and Scaling Autono...
history of c programming in notes for students .pptx
Nekopoi APK 2025 free lastest update
top salesforce developer skills in 2025.pdf
How to Choose the Right IT Partner for Your Business in Malaysia
Why TechBuilder is the Future of Pickup and Delivery App Development (1).pdf
How to Migrate SBCGlobal Email to Yahoo Easily
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
ManageIQ - Sprint 268 Review - Slide Deck
medical staffing services at VALiNTRY
SAP S4 Hana Brochure 3 (PTS SYSTEMS AND SOLUTIONS)
Odoo POS Development Services by CandidRoot Solutions
VVF-Customer-Presentation2025-Ver1.9.pptx
Operating system designcfffgfgggggggvggggggggg
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
Which alternative to Crystal Reports is best for small or large businesses.pdf
CHAPTER 2 - PM Management and IT Context

Fully-managed Cloud-native Databases: The path to indefinite scale @ CNN Mainz

  • 1. qaware.de Fully-managed Cloud-native Databases: The path to indefinite scale Dirk Kröhan dirk.kroehan@qaware.de
  • 3. 3 The history of rail gauges: https://parovoz.com/spravka/gauges-standard.html What does a horse have to do with a space shuttle? QAware
  • 4. 4 Where do we actually store our application data? QAware
  • 5. MagentaTV Voice @ Deutsche Telekom 5 QAware
  • 6. High-level architecture 6 CVI Event-History BI Analytics Invoke API History API WRITE READ READ QAware
  • 8. 8 We want a scalable NoSQL database and preferably fully managed! QAware
  • 9. 9 Runs on Isn't there something provided by Azure? QAware
  • 11. Azure Cosmos DB 11 Azure Cosmos DB is a globally distributed, multi-model database service offered by Microsoft Azure. It is designed to provide high availability, scalability, and low-latency access to data for mission-critical applications. Cosmos DB is a NoSQL database, which means it can handle unstructured and semi-structured data types. Cosmos DB supports multiple data models, including document, key-value, column-family, and graph, making it suitable for a wide range of use cases. QAware
  • 13. Multi-Model 13 ■ API for NoSQL (native) ■ API for MongoDB ■ API for PostgreSQL (new) ■ API for Apache Cassandra ■ API for Apache Gremlin ■ API for Table 👍 QAware
  • 15. High Availability, Scalability, Low Latency 15 ■ SLA with 99,999 % read availability ■ Scalability via guaranteed throughput – Manual scaling – Automatic instant scaling based usage ■ Low Read-/Write-Latency – Guaranteed < 10ms at 99th percentile – Average Read-Latency ~4 ms (read) / 5 ms (write) at 50th percentile https://uptime.is/99.999 QAware
  • 16. Request Unit as scaling and throughput unit 16 A point read for a 1-KB Item costs exactly 1 Request Unit. point read: Reading a single Item per ID and Partition-Key value QAware
  • 17. Scaling based on provisioned throughput 17 ■ Throughput on database level – Shared throughput across all containers – No throughput guarantees for a specific container ■ Throughput on container level – Exclusive throughput for a single container If a database operation requires more RUs than the currently available throughput, rate limiting applies, i.e. the request is rejected with a 429 response. The response contains hints as to when the request should be repeated. The required RU throughput is always linked to the underlying physical partition. –> Hot Partition QAware
  • 21. High scalability in the real world
  • 22. Indefinite scaling!? 22 ¹ You can increase Maximum RUs per container or database by filing an Azure support ticket. QAware
  • 23. Yes, but you have to know what you're doing! 23 Quelle:https://cdn.prod.www.spiegel.de/images/6c9a2605-8b97-4545-a815-926da609c698_w1600_r1.5_fpx58_fpy55.01.jpg vs. Quelle: https://de.wikipedia.org/wiki/Kombinationskraftwagen QAware
  • 24. Data modeling and partitioning is the key! 24 ■ Scalability relies on the partitioning of data ■ Choosing the right partition keys is essential ■ A partition key can not be changed ■ The partition key defines a logical partition ■ Goal: All database operations should hit one logical partition QAware
  • 25. Examples from our project
  • 26. High level architecture 26 Azure CosmosDB Container: event Touchpoints APIs ETL APIs PDM APIs Customer- Care APIs VLP APIs QAware Get cards Get conversations
  • 27. Cons 27 ■ Event container stores two different types of items ■ Touchpoint APIs require only a small portion of the whole JSON data – Get single card by ID – Get cards paginated by size – Get conversations paginated by time + various filter criteria – Vote a single conversation – Delete bulk – Delete all + by filter criteria ■ Vote requires an update of an existing conversation item ■ Delete requires an update of an existing conversation to set a delete flag and to set sensitive data to NULL QAware
  • 28. Cons TL;DR 28 ■ Database queries are expensive in terms of required database throughput ■ Database must be scaled higher to provide sufficient throughput ■ Higher code complexity QAware
  • 29. Goal: Separation of concerns 29 ■ Store data that is needed from the customer's perspective separately from the data stored for analysis and debugging purposes ■ Reduce complexity – Simplify the deletion process: Delete data that the customer sees immediately, delete remaining data asynchronously QAware
  • 30. Azure CosmosDB Container: event Container: user_conversation Consumer APIs of the Event-History 30 Touchpoints APIs ETL APIs PDM APIs Customer- Care APIs VLP APIs QAware
  • 32. RU Throughput after 32 ~90% RU throughput savings!! QAware
  • 33. Some of the features
  • 34. TTL-based Deletion 34 ■ Globally on container level ■ Individually per Item via ttl property – Global TTL still has to be activated, at least with “-1” (Items do not expire) QAware
  • 36. A look outside the box: AWS DynamoDB & Google Firestore
  • 37. AWS DynamoDB 37 ■ “Fully managed NoSQL database service that provides fast and predictable performance with seamless scalability” ■ Provides a single own API ■ Scaling is also based on provisioned throughput – Autoscaling is possible ■ 2 Consistency Levels (eventually consistent, strongly consistent) – Can be selected for each individual read access ■ Data is automatically replicated to multiple availability zones in an AWS region – Global tables enable multi-region replication QAware
  • 38. AWS DynamoDB 38 ■ Maximum item size: 400KB (CosmosDB 2MB) ■ Throughput is measured in read capacity unit (RCU) und write capacity unit (WCU) – RCU: 1 strongly consistent / 2 eventually consistent read access per second for items up to 4KB – WCU: 1 write access per second for items up to 1 KB ■ Throughput limits per partition: 3000 RCUs and up to 1000 WCUs ■ DynamoDB Streams is similar to CosmosDB ChangeFeed, but only for the last 24h. Contains all changes to an item including deletions! ■ Supports ACID transactions across multiple items, also across multiple tables QAware
  • 39. Google Firestore 39 ■ “Serverless, maintenance-free document database that can be easily scaled to meet any need” ■ Automatic replication across multiple regions. Guaranteed availability of 99.999 % ■ Strong focus on App and Mobile development – Live-synchronisation of changes – Offline-Mode ■ Maximum item size: 1 MB ■ ACID transaction support ■ Automatic scaling – Up to 1 million concurrent connections and 10000 writes/second ■ Pricing is mainly based on database operations (read, write, delete) plus some extra costs for bandwidth and storage QAware
  • 40. 1. + 2. Juli 2024 https://www.kcdmunich.de/
  • 41. qaware.de QAware GmbH Mainz Rheinstraße 4 C 55116 Mainz Tel. +49 6131 21569-0 info@qaware.de twitter.com/qaware linkedin.com/company/qaware-gmbh xing.com/companies/qawaregmbh slideshare.net/qaware github.com/qaware