SREDAY London 2024 | Cloud Native Technologies: The Building Blocks of Modern Database Software
0 likes103 views
The document outlines the evolution of database deployment from early mainframe systems to modern cloud-native technologies, emphasizing how Kubernetes facilitates management of stateful workloads like databases. It discusses the rise of Database-as-a-Service (DBaaS) and the challenges associated with stateful workloads, while also detailing the architecture and components necessary for building DBaaS on Kubernetes. Additionally, it showcases examples of organizations that have successfully implemented DBaaS solutions using Kubernetes.
SREDAY London 2024 | Cloud Native Technologies: The Building Blocks of Modern Database Software
- 1. 1 Cloud Native Technologies : The Building Blocks of Modern Database Software Divine Odazie 2024 London 20th September 2024
- 2. 2 2 Divine Odazie ● Technology Evangelist at Severalnines ● Electrical/Electronics Engineer ● Certified Kubernetes Application Developer (CKAD) ● Data on Kubernetes Ambassador ● Certified AWS Solutions Architect ● I’m excited to speak at SREDAY London 2024! Database ops, your way
- 3. 3 1. A brief history of database deployments 2. Cloud Native Technologies 3. Introduction to Stateful Workloads on Kubernetes 4. Building a DBaaS on Kubernetes 5. Building a DBaaS with Kubernetes 6. Conclusion
- 5. 5 5 Early Days ● Data was mainly stored in mainframe computers. ● At this time, limitations included high cost and rigid data models. ● Oxford-educated mathematician and computer scientist, Edgar Codd's 1970 paper introduced the relational database, revolutionizing the field. Database ops, your way
- 6. 6 6 Pre-Cloud ● Organizations had to purchase and manage their own database hardware and software. ● Cloud computing emerged as a potential solution, promising easier and more affordable database options. ● These on-prem deployments are still important today, especially for data sovereignty, compliance, and low-latency requirements. Database ops, your way
- 7. 7 7 The Cloud Era ● Cloud-managed databases simplified provisioning and management of database infrastructure. ● Hardware tasks were offloaded to the cloud provider, allowing organizations to focus on their applications. ● Database-as-a-Service (DBaaS) further abstracted complexity and offered pay-as-you-go pricing. Database ops, your way
- 8. 8 8 Container Revolution ● Container technologies, led by Docker, have revolutionized application and database deployment. ● It enabled easier and faster deployment across various platforms. ● Kubernetes emerged as the defacto orchestration standard for containerized applications. Database ops, your way
- 9. 9 Road to databases on Kubernetes Database ops, your way
- 10. 10 Cloud Native Technologies Database ops, your way
- 11. 11 What is Cloud Native? Cloud native practices empower organizations to develop, build, and deploy workloads in computing environments (public, private, hybrid cloud) to meet their organizational needs at scale in a programmatic and repeatable manner. 11 Cloud native technologies Cloud native technologies and architectures typically consist of some combination of containers, service meshes, multi-tenancy, microservices, immutable infrastructure, serverless, and declarative APIs — this list is non-exhaustive. Source: Cloud Native Computing Foundation (CNCF) Definitions Database ops, your way
- 12. 12 12 The Cloud Native Computing Foundation (CNCF) The CNCF seeks to drive adoption of the Cloud Native paradigm by fostering and sustaining an ecosystem of open source, vendor-neutral projects. It democratize state-of-the-art patterns to make these innovations accessible for everyone.
- 13. 13 Introduction to Stateful Workloads on Kubernetes Database ops, your way
- 14. 14 What is a Stateful workload? A workload that requires persistent data and must maintain its state across restarts or failures. Examples include: ● Databases: Store transactional data, user information, etc. ● Game servers: Persist player data, game progress, etc. ● AI/ML applications: Store training data, model checkpoints, etc. Stateful workloads require meticulous orchestration! Database ops, your way
- 15. 15 Challenges Stateful workloads face ● Maintaining a consistent identity ○ Often for connection to other services ● High & Consistent Availability ○ Upgrades must be handled carefully to avoid disruption ○ Stateful workloads often have complex start and shutdown processes Source: Stateful Workloads in Kubernetes: A Deep Dive Database ops, your way
- 16. 16 Kubernetes was originally designed for stateless workloads. Database ops, your way
- 17. 17 How is Kubernetes addressing these challenges? Deployment ● StatefulSets ○ A StatefulSet ensures each of its pods retains a sticky, stable, unique identity. ● Jobs ○ Training datasets, models in ML ○ Backups + Other components Lifecycle and Day 2 Management ● Custom Resource Definitions (CRD) ● Operators (How Kubernetes runs CRDs) ○ PostgreSQL, ○ Kafka, ○ MySQL, ○ Redis, etc. Database ops, your way
- 18. 18 How does Kubernetes handle storage? Database ops, your way
- 19. 19 19 Persisting the Data Persistent Volumes (PVs): Kubernetes objects that represent storage resources in your cluster. Persistent Volume Claims (PVCs): a request for storage by a user. PVCs consume PV resources. Container Storage Interface (CSI): is a standard for exposing arbitrary block and file storage systems to containerized workloads Getting features like: ● Dynamic provisioning, resizing ● Snapshots, cloning etc. Source: DoK Whitepaper Database ops, your way
- 20. 20 20 Source: DoK Whitepaper Putting it all together ● The user submits a configuration… ● The Operator processes the request… ● The StatefulSet creates pods in order… ● The PVCs are bound to Persistent Volumes… ● The database application within the pods… ● The Operator continuously monitors the cluster… Database ops, your way
- 21. 21 Building a DBaaS on Kubernetes Database Ops, Your Way
- 22. 22 What does a DBaaS on Kubernetes need? ● Requires a system to manage users and databases ● Needs a method to apply schemas ● Must generate secrets with authentication data and distribute them to clusters and applications using those databases ● More importantly the ability to automate day 2 ops and standardize daily lifecycle ops ● Requires a user-friendly interface Database ops, your way
- 23. 23 Being Independent ● Involves procuring your own infrastructure, building custom software to handle each job within the DBaaS framework. ● Requires expertise and significant investment in time and resources. 23 You choose Being Interdependent ● A middle ground between buying a solution and building everything from scratch. ● You would integrate your chosen infrastructure, code, and tools with off-the-shelf software to function as the control plane. https://youtu.be/G8MnpkbhClc?si=- he2TdIiJI7dKUv4 Database ops, your way
- 24. 24 Source: How to Build Modern DBaaS Using CNCF Ecosystem Technologies Interdependency with Cloud Native Technologies Database ops, your way
- 25. 25 Do-It-Yourself DBaaS Database ops, your way
- 26. 26 Zalando ● PostgreSQL-as-a-Service on Kubernetes since as far back as 2018. Learn more in this talk. ● Deploying and managing 500+ PostgreSQL HA clusters. ● Creators of the Postgres operator and Patroni. 26 Who built a DBaaS on Kubernetes? Shopify ● KateSQL – Shopify’s custom-built Database-as-a-Service platform on GKE. ● Supporting hundreds of production MySQL instances. ● Kubernetes for efficient resource utilization and scaling. https://youtu.be/G8MnpkbhClc?si=- he2TdIiJI7dKUv4 Database ops, your way
- 27. 27 Not everyone wants to put their databases on Kubernetes… YET! Database ops, your way
- 28. 28 Building a DBaaS with Kubernetes Database ops, your way
- 29. 29 29 Introducing CCX CCX is a drop-in, white-label DBaaS that allows CSPs to offer managed open-source databases to their end users, using their own infrastructure. Details: ● Kubernetes-based ● Built on the industry-leading ClusterControl (CC) platform ● Prioritizes data sovereignty through a loosely coupled architecture and regulatory compliance ● Quick to set up and integrate ● Provides 24/7 support and high SLA ● Offers an easy-to-use ops and end-user interface. Database ops, your way
- 30. 30 Architecture & Overview CCX Cloud Service Provider Architecture Database ops, your way
- 31. 31 Deploy CCX Dependencies ● CCX core DB operators ● NATS ● ExternalDNS ● Ingress controller 1 CCX Installation in 3 Steps 2 Deploy CCX Core Services ● CCXUI ● Authentication service, etc. ● DB Orchestration service 3 Deploy CCX Observability Stack ● Grafana ● Alert manager ● Prometheus node exporter CCX Documentation | Public On Github | ArtifactHUB Database ops, your way
- 32. 32 32 Elastx: A Swedish Cloud Provider || Lintasarta Cloudeka: An Indonesian Cloud Provider ● They implemented a full-fledged, premium service quickly instead of committing a ton of R&D resources and time to build one from scratch. ● They are now offering DBaaS to thousands of end users. ● They fully customized the platform to meet their specific needs. Who built a DBaaS with CCX? Database ops, your way
- 33. 33 33 Wrapping up The journey of database deployment is far from over. Cloud-native technologies, particularly Kubernetes, offer powerful tools for managing Stateful workloads like databases. If you are interested in a drop in DBaaS solution, scan the QR code. Database ops, your way
- 34. 34 Tech Evangelist Divine Odazie Check out our podcast: Sovereign DBaaS Decoded Sovereign DBaaS Decoded @Severalnines @_Odazie