The Operator Pattern - Managing Stateful Services in Kubernetes
1 like627 views
The document discusses the operator pattern for managing stateful services in Kubernetes, emphasizing the need for automated day-2 operations such as backups, scaling, and healing. It outlines how operators facilitate custom resource definitions (CRDs) that define new APIs and integrate seamlessly with existing Kubernetes APIs. Additional topics include the management of Kafka clusters, advanced scheduling features, and considerations for using operators vs. Helm vs. controllers.
The Operator Pattern - Managing Stateful Services in Kubernetes
- 1. The Operator Pattern 1 Managing Stateful Services in Kubernetes Jakob Karalus, @krallistic
- 2. 2 • Data Science + DevOps • Codecentric • CKA • Twiiter: @krallistic • Github: github.com/krallistic $whoami
- 3. 3 • Write some Deployment, Services, Configmaps etc • Deploy them to K8s • Maybe create a Helm Chart Normal Kubernetes Deployment
- 4. 4 Success?!
- 5. 5 • Backups? • Upscaling? Reshuffle Data? • Downscaling? Without Dataloss? • Healing? Restore Backups? • Configuration? Tedious Templating? But Day 2 Operations?
- 6. If only we could automate this! 6 In a Kubernetes native way!
- 8. 8 • Human Operational Software • Custom Software • Kubernetes Native: • CustomResourceDefinition • So lets write one: Operators
- 9. • Defines a new API • Seamless integration with existing API • Kubectl support 9 CustomResourceDefinition apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: crontabs.stable.example.com spec: group: stable.example.com version: v1 scope: Namespaced names: plural: crontabs singular: crontab kind: CronTab shortNames: - ct validation: # openAPIV3Schema is the schema for validating custom objects.
- 10. • Actual Object in new API • No functionality. 10 CustomResourceDefinition apiVersion: "stable.example.com/v1" kind: CronTab metadata: name: my-new-cron-object spec: cronSpec: "* * * * */5" image: my-awesome-cron-image replicas: 5
- 11. 11 • Operator create WATCH on CR Objects • Analyze difference Actual vs Desired State • Act on changes Control Loop Observe Analyze Act
- 12. Kafka Basics 12 Broker 0 Topic: “test” Partitions: 4 Replicas: 2 Partition 0 – Replica 0 Partition 2 – Replica 1 Partition 3 – Replica 1 Broker 1 Broker 2 Partition 0 – Replica 1 Partition 1 – Replica 1 Partition 2 – Replica 0 Partition 1 – Replica 0 Partition 3 – Replica 0 Topic: “hello” Partitions: 8 Replicas: 3
- 13. 13 Create Cluster Kind: KafkaCluster Name: analytics Replicas: 3 ADD Event Kind: ConfigMap Name: kafka Data: | Kind: Statefulset Name: analytics Replicas: 3 Image: Kafka POST Create Pods
- 14. 14 Downsize Cluster Kind: KafkaCluster Name: analytics Replicas: 2 Update Event Desired: Kind: Statefulset Name: analytics Replicas: 2 Image: Kafka UPDATE Current: Kind: Statefulset Name: analytics Replicas: 3 Image: Kafka Cluster Rebalanced Rebalance Topic Delete Pod
- 15. 15 • Upsizing • Live Rebalancing • Updates • Backups & Restores • Failovers Additional Usecases
- 16. 16 • Extra Scheduling needs? -> Scheduler Extenders (Webhooks) • More Advanced Scheduling Features -> Custom Schedulers • Complex APIs beyond CRD -> Custom API Server with API Aggregation • Control over what object can be created -> Admission Webhooks • Control how objects should be initialized? -> Initializer • Different Runtime needs? -> CRI, CNI • Special Storage -> Flex Volumes, (CSI) Not enough?
- 17. Elasticsearch Prometheus Postgres 17 Other Operators
- 18. 18 • Helm itself a Operator (somewhat, working on it https://github.com/kubernetes/helm/issues/3089) ) • Controllers • Operator = Controller + CRD • Operator = External Software • Controller = Internal • Only do operators if you cant solve it with Helm. Operators vs Helm vs Controller
- 19. 19 • WATCH for updates • Generator Controller • Informer • Create API Spec • Generate some Objects needed by Informer etc (Since 1.8) • See: https://blog.openshift.com/kubernetes-deep-dive-code-generation- customresources/ (Excellent, by sttts) • Main Code!
- 20. 20 • Microservices, single Deployment • Stateless, use CRD for States • Operations should be Idempotent • Leverage K8S Objects as most as possible • CRD should be versioned, backwards compatible Best Practice Operators
- 22. 22 • Are we reinventing the Wheel? • Mesos Frameworks? • Nomad Custom Scheduler? • Docker Swarm Plugins? Take a step back
- 23. 23 Custom Logic in Mesos
- 24. 24 Task Mesos Kuberentes Custom Resource Placement Write a framework Write a custom scheduler Special resource init Write a framework Initializer API access Every Framework has its own API Unified API Special lifecyle Write a framework Kubernetes Operator Custom execution Write a framework + executioner CRI Interface + Scheduler Comparison