David Steiman - Getting serious with private kubernetes clusters & cloud native storage - Codemotion Berlin 2018
0 likes44 views
David Steiman's presentation discusses the implementation of private Kubernetes clusters and cloud-native storage. It covers motivations for private clusters, deployment strategies, and various tools such as kubeadm, Hetzner-Kube, and storage solutions like OpenEBS and Rook. The session highlights cost benefits, architecture, and high availability considerations for private cloud environments.
David Steiman - Getting serious with private kubernetes clusters & cloud native storage - Codemotion Berlin 2018
- 1. Getting serious with private kubernetes clusters & cloud-native storage David Steiman Berlin | November 20 - 21, 2018
- 2. About Me ➢ Name: David Steiman aka. xetys ➢ Owner of github.com/xetys/hetzner-kube ➢ Twitter: @theOnlyScrippi ➢ GitHub: github.com/xetys ➢ Blog: stytex.de ➢ Working @ K-TEL Communications ➢ JHipster core developer
- 3. Private cluster = ? ➢ Bare-metal ➢ Virtual instances on own infrastructure ➢ Custom cloud provider ➢ Raspberry PI?
- 4. Motivation
- 5. Motivation 1. Custom machine providers 2. Own “cloud” a. IaaS (Block/Object Storage, TLS Certs, VMs, networks) b. FaaS (OpenFaaS, Kubeless, …) c. PaaS (fabric8, deis, …) d. CI/CD (jx, drone, Gitlab) 3. Cost-reduction a. 82 cores, 328GB RAM, 4TB of resilient storage => ~500 EUR b. Similar setup at AWS ~12k EUR 4. No GAWA?
- 6. Three base stands of private clusters Compute Network Storage
- 8. CoreOS ➢ Pure container philosophy ➢ PXE boot provisioning clusters with matchbox ➢ Enterprise support with Tectonic ➢ Good for large clusters with thousands of nodes
- 9. Ansible powered and driven Kismatic Enterprise Toolkit ➢ Powered by an ansible playbook, extended with go ➢ No “real” HA support ➢ Persistent storage with GlusterFS out-of-the-box Kube-spray ➢ Full ansible based ➢ Large feature base ○ HA support ○ Self-hosted ○ Many Linux distros ➢ kubernetes-incubator project
- 10. Rancher 1.x & Rancher 2.0 Rancher 1.x ➢ Focussed on Cattle ➢ k8s as catalog app ➢ Most easy install ➢ Least correct install ➢ User Support! Rancher 2.0 / RKE ➢ Focussed on k8s ➢ Real HA mode ➢ Yet, quite simple install ➢ User support ➢ Early and little UI
- 11. kubeadm ➢ In-tree component of k8s ➢ Quite simple install ➢ Ready to production ➢ Simplifies ugly parts
- 13. hetzner-kube ➢ Go tool for deploying k8s on hetzner cloud ➢ Uses kubeadm under the hood ➢ Ships default with flannel ➢ Bundles addons like helm, ingress, cert-manager, kube-prometheus, OpenEBS, rook ➢ E2e tested
- 14. hetzner-kube High Availability ➢ External etcd cluster ➢ Decentralized apiserver proxy using nginx ➢ Tested with evil tools like comcast ○ and Falkenstein DC outtake
- 16. Network & Load Balancing
- 17. type: LoadBalancer? ➢ Most commonly not available to private clusters ➢ Exception: Rancher 1.X with cloud-provider Rancher ➢ Should be realized using --cloud-provider=<custom> ➢ MetalLB
- 18. nginx ingress controller on edge nodes ➢ Label nodes as edge routers ➢ Deploy nginx-ingress-controller with nodeSelector ➢ Multiple A-Records per domain
- 19. Node networking ➢ Encrypted private network VPN tool “Wireguard” ➢ https://github.com/xetys/wgctl ○ Operate wireguard networks
- 20. Pod Networking ➢ Simple setup: flannel, weave ➢ Simple + Pod Network policies: canal ➢ Integrate with existing network infrastructure: calico ➢ Alternative approaches: kube-router
- 22. Persistence
- 23. Needs ➢ Storage Class support ➢ High Availability & Fault resistance ➢ High Performance in Throughput & IOPS ➢ RWO + RWX ➢ Backup tools
- 24. Kubernetes driven solutions ➢ OpenEBS ○ Uses containers for every storage ○ Uses iSCSI ➢ Rook ○ Leverages ceph as backing storage cluster ○ Simplifies ceph operation via CRD ➢ GlusterFS ○ Supports RWX ootb ○ Supports Storage Class with heketi
- 25. Dedicated Ceph cluster ➢ Manageable in operation ➢ One ceph for several clusters ➢ Storage Class Support with RWX, and object storage from kubernetes-incubator/external-storage
- 27. Demo Time