Cassandra Operator with Yelp PaaSTA
0 likes342 views
The document discusses Yelp's deployment of Cassandra using Kubernetes, highlighting the migration strategy, cluster management, and operational challenges. It outlines the architecture of Cassandra clusters, including state reconciliation, storage management, and IAM role configurations. The document also addresses pain points related to stateful sets and outlines future goals like improved autoscaling and expanded workloads on Kubernetes.
Cassandra Operator with Yelp PaaSTA
- 1. Cassandra Operator with Yelp PaaSTA Raghavendra Prabhu & Matthew Mead-Briggs JetStack Connect Nov 2019
- 2. Yelp’s Mission Connecting people with great local businesses.
- 3. Powered by Cassandra Your Reviews 1 2 1 Waitlist for diners2
- 4. Overview ● Why Kubernetes ● Status quo with Cassandra ● Cassandra Operator ● Pain Points ● Future
- 5. What is PaaSTA github.com/yelp/paasta github.com/yelp/clusterman
- 6. Why K8s ● Why Kubernetes ○ Why not persist with plain EC2 ● Why DIY the operator
- 7. Status Quo of Cassandra @ Yelp ● What is Cassandra ● Roughly a hundred clusters on AWS ASG ● New cluster launches with k8s ○ Batteries included: Good defaults, TLS ● Migration strategy in place ○ Backward compatible discovery mechanism ● K8s clusters deployed on spot fleet as well. ● Local k8s development cluster with https://github.com/kubernetes-sigs/kind
- 9. us-west-2 us-west-1 us-east-1 Yelp Cassandra @ 100000 ft Multi-region Cluster
- 10. us-west-2 us-west-1 us-east-1 Yelp Cassandra @ 100000 ft + Operator Multi-region Cluster
- 11. State of Cassandra Operator at Yelp ● Cassandra cluster specification ● What is in a Cassandra Pod ● Storage aka State ● Reconciliation ○ StatefulSet ○ Core event loop ● Deployment
- 12. The Recipe aka the Cluster Spec
- 15. Cassandra Pod ● Cassandra container ● Sidecars ○ Hacheck for Nerve (Smartstack) ○ Cron Jobs ○ Sensu alerting ● Node: metrics collection ○ Puppet
- 16. To sidecar or not to sidecar ● Emit data to host/external service ● Collect data from process in hosts namespace ● Sidecar collects data
- 17. Storage aka State ● Dynamic Provisioning ○ StorageClass per cluster ○ “Compute follows Data” ■ Immediate Volume Binding Mode ■ Stripe cluster across AZs ● EBS for Cassandra ○ Clear separation of stateful and stateless ○ Makes it easy to delete statefulsets ○ Bouncing the cluster is also quite fast
- 23. Hash-based reconciliation ● Compute hash of Pod Template ● Attach as label to the StatefulSet ● Compare label on existing StatefulSet to newly computed
- 24. Cluster Readiness ● Cluster ready = AND(pod readiness) over all ○ Service Readiness ● Readiness per pod: UN in Cassandra ● Liveness check: U for Cassandra ● Hooks ○ Draining
- 25. Locking ● Clusters are multi-region ● Operators are per-region ● Non-federated setup ● Coordination with etcd leases ● LeaseID stored in Custom Resource Status
- 26. IAM roles ● For cassandra we need access to S3 and dynamoDB for backups ● https://github.com/uswitch/kiam ● Proxies the EC2 metadata service for Pods ● Allows us to lift and shift IAM profiles from EC2
- 27. PaaSTA Secrets Support ● PaaSTA on Mesos already supports secrets ● User friendly cli to “create” secrets ● Use Vault’s transit endpoint to encrypt ● Sync these secrets into kubernetes Secrets ● Cassandra is using these for TLS secrets $ echo "SOMETHINGSECRET" | paasta secret add -s cassandra_k8s -n secret-name-here -c norcal-devc
- 28. Deployment / PaaSTA integration
- 31. Migration ● Launching new clusters in k8s is easy ● Migration of existing clusters without downtime is hard ● Unified discovery with smartstack ● How: Add k8s nodes to existing Cassandra cluster ● We have migrated a few already!
- 32. Pain Points
- 33. Pain points ● Client-side Validation ● Statefulset inflexibility with changes ○ Manual intervention for stuck statefulset deployments ○ Resizing the Persistent Volume ○ Orphaned EBS volumes ● Unready/Dead Nodes, Spot fleet and garbage collection
- 35. Heading towards > ● Load-based autoscaling for Cassandra pods ● EBS snapshotting automation ● Fleet autoscaling with Clusterman ● Better integration tests for the operator and for the clusters! ● Production clusters on AWS spot fleet ● More workloads on kubernetes (just started our Kafka operator)
- 36. Conclusion
- 39. Questions?
- 40. Credits ● Apache cassandra logo ● https://kubernetes.io/ ● https://etcd.io/ ● https://aws.amazon.com/architecture/icons/ ● https://www.yelp.com/brand ● https://thenounproject.com/ ● https://commons.wikimedia.org/wiki/File:Back-to-the-future-logo.svg ● https://www.writeups.org/star-trek-brent-spiner-data/