Intro into Rook and Ceph on Kubernetes
0 likes303 views
The document presents an introduction to using Rook/Ceph on Kubernetes, emphasizing its utility in hybrid, distributed, and complex environments. It discusses the advantages of Kubernetes as a container management solution, addresses challenges related to heterogeneous infrastructure, and demonstrates the setup of a stateful application using Kublr, Submariner, and Ceph/Rook for storage management. The presentation also covers operational aspects, architecture specifications, and examples of backup, recovery, and resource management strategies.
Intro into Rook and Ceph on Kubernetes
- 1. Rook/Ceph on K8S Introduction Oleg Chunikhin | CTO, Kublr
- 2. Oleg Chunikhin CTO, Kublr âą 25 years in software architecture & development âą Working w/ Kubernetes since its release in 2015 âą Software architect behind Kublrâan enterprise ready container management platform âą @olgch Introductions
- 3. Automation Ingress Custom Clusters Infrastructure Logging Monitoring Observability API Usage Reporting RBAC IAM Air Gap TLS Certificate Rotation Audit Storage Networking Container Registry CI / CD App Mgmt Infrastructure Container Runtime Kubernetes OPERATIONS SECURITY & GOVERNANCE
- 4. Application DevOps Automation Developers SRE/Ops/DevOps/ SecOps âą Self-service âą Compatible âą Conformant âą Configurable âą Open & flexible âą Governance âą Org multi-tenancy âą Single pane of glass âą Operations âą Monitoring âą Log collection âą Image management âą Identity management âą Security âą Reliability âą Performance âą Portability
- 5. Hybrid Architecture â Hybrid ~ Distributed ~ Complex â Services â Connectivity â Discovery â Data â Sharding â Mirroring / Replication â BCDR â Failure and recovery scenarios â Architecture â Application â Middleware â Infrastructure â PoC â Hardening and Optimization â Operations
- 6. Kubernetes as Container Management K8s is a solution for: âą Uniform application management âą Uniform resource management âą In-cluster failover, load balancing, traffic management (service mesh) Challenges: âą Heterogeneous middleware â distributed data is difficult âą Heterogeneous infrastructure â distributed load balancing and ingress is difficult Infrastructure K8S Applications Infrastructure K8S Applications Managed Middleware (e.g. RDS, EFS, ...) Managed Middleware (e.g. Azure SQL, FS, ...) ?
- 7. Kubernetes as Infrastructure and Platform Adds: â Homogenous middleware â Ceph/Rook, Portworx, Vitess, Strimzi/Kafka â Open cross-vendor inter-cluster connectivity â Submariner â Uniform BCDR â Velero â Uniform (operator-based) and portable middleware management â Flexible and portable infrastructure mapping for middleware Infrastructure K8S Applications VPN / WAN Self-hosted Middleware (e.g. Ceph/Rook, Vitess, ...) Infrastructure K8S Applications Managed Middleware (e.g. RDS, EFS, ...) Managed Middleware (e.g. Azure SQL, FS, ...) VPN / WAN Self-hosted Middleware (e.g. Ceph/Rook, Vitess, ...) BCDR (e.g. Velero) BCDR (e.g. Velero) IPSec, Wireguard, ... Mirroring, Sharding, ...
- 8. Demo: Stateful App in Hybrid Environment â AWS and Azure â Kublr for Infrastructure and K8S provisioning â Submariner as reliable VPN â Ceph / Rook as an example of portable storage middleware â HA PoC: multi-zone, HA storage â BCDR PoC: mirroring, failover â Cost control PoC: spot instances Infrastructure K8S Applications Submariner Ceph / Rook storage Infrastructure K8S Applications AZ, EBS Zones, Azure Disks Submariner Ceph / Rook storage IPSec Mirroring, Snapshots
- 9. Kubernetes Operators â Operator Pattern â CRD â Spec: component definition â Status: component status â Operator â Links the component and CRD â Operator in this demo â Submariner â Rook â ~ Kublr
- 10. VPN: Submariner Worker Node Worker Node Passive Gateway Node Active Gateway Node Gateway Label Gateway Label Public Network Passive Gateway Node Active Gateway Node Gateway Label Gateway Label Worker Node Worker Node Cluster Node Route Agent Gateway Engine VxLAN Traffic IPSEC Traffic
- 11. Storage: Rook / Ceph Data pool mon mon mon config data raw data osd raw data osd raw data mds osd Data pool Image Image Ceph Filesystem Components Abstractions Ceph rgw S3/Swift Object Store mgr Rook Operator CSI plugins osd osd ganesha NFS CephCluster Block Pool Object Store Filesystem NFS Object Store User Provisioners rbd-mirror
- 12. Stack DeïŹnition SOURCE TYPE DESCRIPTION Infrastructure Specification Virtual Machines, Networks, Disks, etc Cloud Formation, ARM Templates, Terraform, Kublr Kubernetes Specification Cluster topology, masters and workers number, groups, K8S components configuration, versions System/support Software Specification Kubernetes system components: e.g. overlay network, DNS, etc (Self-)managed application services: Cloud Native Storage (Ceph/Rook), DB (Vitess), Messaging (Strimzi/Kafka, Nats), API Management (Ambassador, Kong), etc DevOps tools: CI (Jenkins), CD (Spinnaker), Repositories (Nexus, Artifactory) etc Provisioning scripts Provisioning procedures and processes: shell, Makefile, Jenkinsfile, CircleCI etc
- 13. kind: Cluster metadata: name: demo-hybrid-1-aws spec: ... network: apiServerSecurePort: 6443 locations: - name: aws1 aws: ... master: minNodes: 1 ... locations: - aws: ... nodes: - name: group1 minNodes: 3 ... locations: - aws: ... features: monitoring: { ... } packages: my-package: { ... } Cluster SpeciïŹcation Kublr metadata for the cluster - name, space, labels Cluster-wide non provider specific configuration - network, cluster-wide settings, k8s version, etc Infrastructure provider specific cluster-wide configuration - account, access creds, AZs etc Infrastructure provider specific group configuration - AZs, image, group type, zone locking, etc Group-specific non provider specific configuration - k8s options, autoscaling, etc Kublr-specific built-in K8S components Additional custom helm packages
- 14. kind: Cluster metadata: name: demo-hybrid-1-aws spec: ... network: clusterCIDR: 100.64.0.0/10 dnsDomain: cluster1.local stubDomains: - dns: cluster2.local servers: - 100.128.0.10 locations: - name: aws1 aws: resourcesCloudFormationExtras: SgDefaultSubmariner500: Type: AWS::EC2::SecurityGroupIngress ... ... master: minNodes: 1 ... locations: - aws: groupType: asg-mip mixedInstancesPolicyCloudFormationExtras: ... nodes: - name: group1 minNodes: 3 ... locations: - aws: groupType: asg-mip mixedInstancesPolicyCloudFormationExtras: ... pinToZone: pin availabilityZones: - us-east-1a - us-east-1b - us-east-1c Infrastructure Additional ports for VPN kind: Cluster metadata: name: demo-hybrid-2-azure spec: ... network: clusterCIDR: 100.128.0.0/10 dnsDomain: cluster2.local stubDomains: - dns: cluster1.local servers: - 100.64.0.10 locations: - name: azure1 azure: virtualNetworkSubnetCidrBlock: 172.18.0.0/16 armTemplateExtras: securityGroup: ... ... master: minNodes: 1 ... locations: - azure: armTemplateExtras: ... priority: Spot nodes: - name: group1 minNodes: 3 ... locations: - azure: armTemplateExtras: ... priority: Spot zones: - '1' - '2' - '3' pinToZone: pin Non-intersecting CIDR Mutual discoverability Mixed instance policy and spot instances Multi-zone
- 15. kind: Cluster metadata: name: demo-hybrid-1-aws spec: ... packages: submariner-broker: { ... } rook-ceph: { ... } rook-ceph-additional-configuration: { ... } rook-ceph-cluster: { ... } Middleware Prepare namespace for Submariner broker Ceph cluster definition Rook operator Auxiliary preconfigured CRD (e.g. snapshot class etc)
- 16. Kubernetes Persistence Kubernetes Cluster Namespace Pod Container 1 Container 2 Volume Volume Mount Volume Device âactualâ storage Persistent Volume Volume Claim Spec Spec 2 PVC with SC 5 Pod reference PVC 1 Storage Class Storage Class Provisioner 3 Provision storage Create PV 4 PVC bound to PV
- 17. Demo: RBD and CephFS Data pool Image Data pool mon mon mon config data raw data osd raw data osd raw data mds osd Data pool Image Image Ceph Filesystem Data pool (data) Data pool (md) PV Pod PVC PV Pod PVC PV Pod PVC PV PVC Pod Pod Pod sub-volumes Rook Operators CephBlockPool CephBlockPool CephFilesystem StorageClass StorageClass
- 18. Demo: Mirroring Data pool Image mon mon mon config data raw data osd raw data osd raw data osd Data pool Image PV Pod PVC rbd-mirror Rook Operators Data pool Image mon mon mon config data raw data osd raw data osd raw data osd Data pool Image PV Pod PVC rbd-mirror Rook Operators primary replay CephBlockPool mirroring: enabled: true mode: image peers: ... CephBlockPool mirroring: enabled: true mode: image peers: ... VolumeReplicationClass VolumeReplicationClass VolumeReplication VolumeReplication
- 21. Beyond the Demo â Optimization: Resources, Throughput, ... â Management: Quotas, Topology, ... â Ceph: Object Store, NFS, ... â BCDR: Velero â Connectivity: VPN, Perring, Submariner, ...
- 23. Q&A