Es-operator: Building an Elasticsearch Operator from the bottom up - kube-con eu 2019
- 1. Es-operator Building an Operator From the Bottom Up MIKKEL LARSEN @mikkeloscar 2019-05-21
- 2. 2 $ whoami Mikkel Larsen Software Engineer Cloud Infrastructure (Kubernetes/AWS) @ Zalando SE @mikkeloscar @mikkeloscar
- 3. 3 “EUROPE’S LEADING ONLINE FASHION PLATFORM”
- 4. 4 17 markets WE BRING FASHION TO PEOPLE IN 17 COUNTRIES 7 fulfillment centers 26 million active customers 5.4 billion € revenue 2018 250 million visits per month 15,000 employees in Europe
- 5. 5 KUBERNETES @ ZALANDO ~125 clusters 1400~ nodes Since Oct 2016 Node Autoscaling From v1.4 to v1.13 Default Deployment Target
- 6. 6 300k+ Products per country ~2000 Brands ~700 Categories 45% Mobile Traffic 12K QPS 8K Updates/s SEARCH @ ZALANDO
- 9. 9 RUNNING ELASTICSEARCH IN KUBERNETES 1. Safe automatic updates (Including Kubernetes cluster updates) 2. Advanced auto-scaling for cost efficiency
- 10. 10 Node UPDATING ELASTICSEARCH (STATEFULSET) Node ES Pod ready ES Pod terminating ES Pod ready Node ES Pod ready ES Pod draining Node ES Pod 1) PreStop Hook (bash script) ● Exclude node in ES ● Wait for node to drain (up to 1h) ● Data is moved to existing nodes ready 2) PostStart Hook (bash script) ● Remove all excludes ● Let ES rebalance from existing nodes
- 12. 12 v0: MANAGE STATEFULSET apiVersion: apps/v1 kind: StatefulSet metadata: name: test-cluster annotations: es-operator/desired-replicas: ”3” spec: updateStrategy: type: OnDelete replicas: 2 template: # PodTemplate {...} ● Complicated to update without changing replicas. ● State must be stored in annotations
- 13. 13 v1: ELASTICSEARCH DATA SETS apiVersion: zalando.org/v1 kind: ElasticsearchDataSet metadata: name: test-cluster spec: scaling: {...} replicas: 3 template: # PodTemplate {...} volumeClaimTemplates: {...} github.com/zalando-incubator/es-operator
- 15. 15 Node UPDATING ELASTICSEARCH (OPERATOR) github.com/zalando-incubator/es-operator Node ES PodES Pod ready draining ES Pod ready Node ES Pod ready ES Operator draining Node ES Service 2) Drain node 3) Delete Pod ES PodES Pod ready 1) Scale out by 1
- 16. 16 SCALING UP ELASTICSEARCH (1) METRICS Thresholds ● CPU ● Duration ● Cooldown Boundaries ● Max # Pod replicas ● Min # Shards per node Node ES Pod6 shards ready Node ES Pod3 shards ready Node ES Pod3 shards ready Node ES Pod2 shards ready Node ES Pod2 shards ready Node ES Pod2 shards ready Increase pod replicas Node ES Pod2 shards ready Node ES Pod2 shards ready Node ES Pod2 shards ready Node ES Pod2 shards ready Node ES Pod2 shards ready Node ES Pod1 shard ready
- 17. 17 SCALING DOWN ELASTICSEARCH METRICS Thresholds ● CPU ● Duration ● Cooldown Boundaries ● Min # Replica ● Max # Shards per node ● Max disk usage (%) Node ES Pod6 shards ready Node ES Pod3 shards ready Node ES Pod3 shards readyNode ES Pod2 shards ready Node ES Pod2 shards ready Node ES Pod2 shards ready Node ES Pod2 shards ready Node ES Pod2 shards ready Node ES Pod2 shards ready Node ES Pod2 shards ready Node ES Pod2 shards ready Node ES Pod1 shard ready Decrease Pod replicas DON’T OPERATE WHEN CLUSTER IS NOT GREEN!
- 18. 18 SCALING UP ELASTICSEARCH (2) METRICS Thresholds ● CPU ● Duration ● Cooldown Boundaries ● Min # Shards per node ● Max # Pod replicas Node ES Pod1 shard ready Node ES Pod3 shards ready Node ES Pod1 shard ready Node ES Pod2 shards ready Node ES Pod2 shards ready Node ES Pod1 shard ready Node ES Pod2 shards ready Node ES Pod2 shards ready Node ES Pod2 shards ready Node ES Pod1 shard ready Increase index replicas
- 19. 19 SCALING IN PRODUCTION (7d)
- 20. 20 SCALING IN PRODUCTION (24h)
- 21. 21 LESSONS LEARNED / TAKEAWAYS ● Turn those bash scripts into an operator! ● Assume Operator can die at any point. ● Start simple, add abstractions only when needed.
- 22. 22 OPEN SOURCE Elasticsearch Operator github.com/zalando-incubator/es-operator Kubernetes on AWS github.com/zalando-incubator/kubernetes-on-aws Postgres Operator github.com/zalando/postgres-operator Kubernetes Operator Pythonic Framework (Kopf) github.com/zalando-incubator/kopf