Demystifying Kubernetes Statefulsets
Download as PPTX, PDF1 like2,192 views
This document discusses using StatefulSets in Kubernetes to deploy the distributed database YugaByte DB. It provides examples of deploying YugaByte DB masters and tservers as StatefulSets, ensuring high performance through local storage and data resilience through pod anti-affinity. It also demonstrates a real-world e-commerce application deployed on this stack and discusses extending StatefulSets with operators.
![13© 2018 All rights reserved.
Configuring Data Resilience
POD ANTI-AFFINITY MULTI-ZONE/REGIONAL/MULTI-REGION
POD SCHEDULING
Pods of the same type should not be
scheduled on the same node
Keeps impact of node failures to
absolute minimum
(Beta in latest v1.10)
Multi-Zone - Tolerate zone failures for
k8s slave nodes
Regional – Tolerate zone failures for
both k8s slave and master nodes [GKE
only]
Multi-Region – Requires federation of
k8s clusters](https://image.slidesharecdn.com/2018-03-29-demystifyingstatefulsets-180330173730/85/Demystifying-Kubernetes-Statefulsets-13-320.jpg)
Demystifying Kubernetes Statefulsets
- 1. 1© 2018 All rights reserved. Demystifying Kubernetes StatefulSets Karthik Ranganathan Co-Founder & CTO, YugaByte March 2018
- 2. 2© 2018 All rights reserved. About YugaByte Kannan Muthukkaruppan, CEO Nutanix ♦ Facebook ♦ Oracle Karthik Ranganathan, CTO Nutanix ♦ Facebook ♦ Microsoft Mikhail Bautin, Software Architect Clear Story Data ♦ Facebook ♦ D.E.Shaw Founded Feb 2016 25 employees Ex-Facebook, Oracle, Nutanix, Google & LinkedIn engineers Apache HBase committers Built NoSQL platform at Facebook for: message inbox message search time series spam detection
- 3. 3© 2018 All rights reserved. TRANSACTIONAL PLANET-SCALEHIGH PERFORMANCE Distributed ACID Transactions Document-Based, Strongly Consistent Low Latency, Tunable Reads High Throughput CLOUD-NATIVE OPEN SOURCE Apache 2.0 Popular APIs Extended Apache Cassandra, Redis and PostgreSQL (coming soon) Built For The Container Era Self-Healing, Fault-Tolerant Auto Sharding & Rebalancing Global Data Distribution YugaByte DB
- 4. 4© 2018 All rights reserved. Inside The Hood - 3 Node Cluster DocDB Storage Engine Purpose-built for ever-growing data, extended from RocksDB yb-master1 yb-master3 yb-master2 YB-Master Manage shard metadata & coordinate cluster-wide ops node1 node3 node2 Global Transaction Manager Tracks ACID txns across multi-row ops, incl. clock skew mgmt. Raft Consensus Replication Highly resilient, used for both data replication & leader election tablet 1’ tablet 1’ yb-tserver1 yb-tserver2 yb-tserver3 tablet 1’ tablet2-leader tablet3-leader tablet1-leaderYB-TServer Stores/serves data in/from tablets (shards) tablet1-follower tablet1-follower tablet3-follower tablet2-follower tablet3-follower tablet2-follower … … …
- 5. 5© 2018 All rights reserved. Starting a Cluster https://docs.yugabyte.com/deploy/multi-node-cluster/ Stable network IDs Persistent volumesOrdered operations STATEFUL 1. Start all yb-masters first (usually 3) 2. Start as many yb-tservers as needed
- 6. 6© 2018 All rights reserved. Mapping to Kubernetes Controller APIs DEPLOYMENT Handles updates (rolling/recreation) on ReplicaSets Most commonly used REPLICASET Run N identical Pods Can only use ephemeral storage (pod loss = data loss) DAEMONSET Max 1 Pod per node For node-level functions (e.g. monitoring) JOB Ensure N successful completions For batch jobs such as crons NOT FOR STATEFUL APPS
- 7. 7© 2018 All rights reserved. StatefulSets API 1. Ordered operations with ordinal index – Startup, scale-up, scale-down, rolling upgrades, termination 2. Stable, unique network ID/name across restarts – Re-spawning a pod will not make the cluster treat it as a new member 3. Stable, persistent storage (linked to ordinal index/name) – Attach same persistent disk to a pod even if it gets rescheduled to new node 4. Mandatory headless service (no single IP) for integrations – No load balancer, smart clients aware of all pods and connect to any Alpha v1.3 Jul 2016 Beta v1.5 Dec 2016 Stable v1.9 Dec 2017 Purpose-built for stateful apps
- 8. 8© 2018 All rights reserved. YugaByte DB Deployed as StatefulSets node2node1 node4node3 yb-master StatefulSet yugabytedb yb-master-1 pod yugabytedb yb-master-0 pod yugabytedb yb-master-2 pod yb-tserver StatefulSet tablet 1’ yugabytedb yb-tserver-1 podtablet 1’ yugabytedb yb-tserver-0 pod tablet 1’ yugabytedb yb-tserver-3 podtablet 1’ yugabytedb yb-tserver-2 pod … Local/Remote Persistent Volume Local/Remote Persistent Volume Local/Remote Persistent Volume Local/Remote Persistent Volume yb-masters Headless Service yb-tservers Headless Service App ClientsAdmin Clients
- 9. 9© 2018 All rights reserved. YB-Master yb-masters Headless Service yb-master StatefulSet Access the yb-masters service as $name.$namespace.svc.cluster.local 3 pods https://github.com/YugaByte/yugabyte-db/blob/master/cloud/kubernetes/yugabyte-statefulset-local-ssd-gke.yaml ui & rpc ports volume mountpath matches fs_data_dirs Headless
- 10. 10© 2018 All rights reserved. YB-TServer yb-tservers Headless Service yb-tserver StatefulSet Access the yb-masters service as $name.$namespace.svc.cluster.local 3 pods to begin with, scale as needed https://github.com/YugaByte/yugabyte-db/blob/master/cloud/kubernetes/yugabyte-statefulset-local-ssd-gke.yaml ui, rpc, cassandra & redis ports volume mountpath matches fs_data_dirs Headless
- 11. 11© 2018 All rights reserved. Other Examples https://www.slideshare.net/ssuser6bb12d/kubernetes-introduction-71846110 http://blog.kubernetes.io/2017/02/postgresql- clusters-kubernetes-statefulsets.html
- 12. 12© 2018 All rights reserved. Ensuring High Performance LOCAL STORAGE (Beta in latest v1.10) (Stable) REMOTE STORAGE Lower latency, Higher throughput Recommended for workloads that do their own replication Pre-provision outside of K8s Use SSDs for latency-sensitive apps Higher latency, Lower throughput Recommended for workloads do not perform any replication on their own Provision dynamically in K8s Use alongside local storage for cost-efficient tiering
- 13. 13© 2018 All rights reserved. Configuring Data Resilience POD ANTI-AFFINITY MULTI-ZONE/REGIONAL/MULTI-REGION POD SCHEDULING Pods of the same type should not be scheduled on the same node Keeps impact of node failures to absolute minimum (Beta in latest v1.10) Multi-Zone - Tolerate zone failures for k8s slave nodes Regional – Tolerate zone failures for both k8s slave and master nodes [GKE only] Multi-Region – Requires federation of k8s clusters
- 14. 14© 2018 All rights reserved. A Real-World Example Yugastore – E-Commerce app on the YERN stack Deployed on github.com/YugaByte/yugastore
- 15. 15© 2018 All rights reserved. Kubernetes Deployment Architecture yb-master StatefulSet yugabytedb yb-master-1 pod yugabytedb yb-master-0 pod yugabytedb yb-master-2 pod yb-tserver StatefulSet tablet 1’ yugabytedb yb-tserver-1 podtablet 1’ yugabytedb yb-tserver-0 pod tablet 1’ yugabytedb yb-tserver-2 pod yb-masters Headless Service yb-master-ui LoadBalancer Service yb-tservers Headless Service yugastore Deployment tablet 1’ yugastore yugastore-1 podtablet 1’ yugastore yugastore-0 pod yugastore LoadBalancer Service End User Admin User
- 16. 16© 2018 All rights reserved. DEMO
- 17. 17© 2018 All rights reserved. Extending StatefulSets with Operators https://kubernetes.io/docs/concepts/api-extension/custom-resources/#custom-controllers Based on Custom Controllers that have direct access to lower level K8S API Excellent fit for stateful apps requiring human operational knowledge to correctly scale, reconfigure and upgrade while simultaneously ensuring high performance and data resilience Complementary to Helm for packaging CPU usage in the yb-tserver StatefulSet Scale yb-tserver by 1 pod CPU > 80% for 1min and max_threshold not exceeded
- 18. 18© 2018 All rights reserved. Summary 1. StatefulSets are getting increasingly more powerful 2. Ensuring high performance and data resilience requires careful planning 3. Operators make day-to-day operations a breeze 4. Community ready to help if you want to get started
- 19. 19© 2018 All rights reserved. YugaByte DB is Apache 2.0 Watch/Star us on GitHub! github.com/YugaByte/yugabyte-db
- 20. 20© 2018 All rights reserved. Try on your laptop docs.yugabyte.com/quick-start
- 21. 21© 2018 All rights reserved. gitter.im/YugaByte @YugaByte Thank you