Storage os kubernetes clusters need persistent data
- 1. Kubernetes Clusters Need Persistent Data
- 2. © StorageOS, Inc. About Me: James Spurin • StorageOS Product Evangelist, formerly a founding StorageOS Advisory Board member • 20+ years of Engineering experience covering Storage, Automation, DevOps and Software Development • Previously worked at Nomura, Goldman Sachs, Dell EMC and many more - Connect with me on LinkedIn • Published Technical Author: Dive Into Ansible
- 3. • How organisations are using Kubernetes • Problems + missed opportunities with ephemeral workloads • Use of StorageClasses for Kubernetes multi tenancy • How to simplify the transition of Legacy apps to Kubernetes • Features like Data Encryption • Features like Read-Write-Many volumes • GitOps design patterns with StorageClasses + Demo © StorageOS, Inc. Agenda
- 4. • Self Managed Kubernetes (On-Prem / Cloud) • Rancher – Kubernetes as a Service (On-Prem / Cloud)| • OpenShift – Kubernetes with a PaaS (On-Prem / Cloud) © StorageOS, Inc. How is Kubernetes Deployed and Consumed Common Approaches for Hosting and Managing Kubernetes • Hosted Kubernetes (Amazon EKS, Microsoft AKS, Google GKE) • Developer Environments (Self Hosted / Minikube / Kind)
- 5. © StorageOS, Inc. How are Organisations Using Kubernetes Kubernetes Market Adoption – Heavily utilised for Ephemeral Workloads • Very easy to deploy and use without storage • Application can sync/download content on init • Pod can be destroyed/recreated for site updates • Very easy to deploy and use without storage • Instances can access keys/values from other nodes on failure
- 6. © StorageOS, Inc. Ephemeral First Problems When a Kubernetes environment does not factor for the use of persistent data, we solve for ephemeral environments, this results in - Legacy Environments running alongside Kubernetes Applications may be running in less than optimal ways • Natively supports persistent storage • Improves recovery time • Re-deploying the container is not the only answer Scaffolding to work around non-persistent storage … no really …
- 7. © StorageOS, Inc. Ephemeral First Problems PersitentVolume: NodeAffinity has been used to restrict this workload so that it can only runs on k8snode1 This achieves “Persistent Storage” but sacrifices the benefits, of Kubernetes! What happens if k8snode1 fails? Example ”Workaround”
- 8. © StorageOS, Inc. Treating Persistent Data like Networking Technologies A typical setup procedure or consideration when deploying a Kubernetes cluster is the Installation of a networking technology as a DaemonSet StorageOS installs as an Operator and runs as a DaemonSet
- 9. © StorageOS, Inc. Unlocking the Full Potential of Kubernetes Adding an advanced dataplane like unlocks huge potential across the entire Kubernetes cluster + =
- 10. StorageClasses • Development: Not concerned about the number of copies, data to be highly available across all nodes • Production: Highly available with 2 replicas © StorageOS, Inc. • Top Secret: Encryption Enabled and 2 replicas • Archived: Compression with 1 replica StorageClasses Promote Multi-Tenancy and Agility
- 11. © StorageOS, Inc. Using Storage Classes and Persistent Volume Claims * PV, Persistent Volume automatically created as part of the claim!
- 12. © StorageOS, Inc. Data Encryption at Rest Data Encryption enabled via a single label With the increased market use of Kubernetes as a cloud managed service and strict data compliance regulations that include the Payment Card Industry Standard (PCI DSS) and the Health Insurance Portability and Accountability Act (HIPPA), organisations are dependent on encrypted data as a requirement for Kubernetes workloads using persistent storage
- 13. © StorageOS, Inc. Transition of Legacy Applications is Simplified Applications that rely on Persistent Data can just be run, as you’d expect them to run without scaffolding or legacy environments
- 14. Pool • Allow a filesystem to be shared by multiple containers • Useful for legacy applications (e.g. filesystem as message bus) that rely on a shared filesystem • ReadWriteOnce semantics – 1 application per volume • Some applications require ReadWriteMany (RWX) semantics • Orchestrates a user space NFS layer to provide RWX volumes © StorageOS, Inc. Alternative Approaches for Solution Architects – RWX Volumes N1 N2 N3 N4 N5 • All pods, share same data • Updates to the main volume updates all • No restarts of Pods, needed • Additional benefit of replicas
- 15. © StorageOS, Inc. Containers Can Run More Effectively Unlock intended functionality! Redis natively supports persistent data, improving recovery time Without persistent storage, Redis running in a Kubernetes environment will not make use of this functionality See – https://storageos.com/redis-with-kubernetes- persistent-storage
- 16. GitOps Approach myapp1 (OnPrem): Highly Available with 2 replicas © StorageOS, Inc. myapp1 (Cloud): Highly Available with 2 replicas + Encryption StorageClass shares the same name (myapp1) but with different configurations
- 18. Further Reading © StorageOS, Inc. StorageOS Platform Architecture Overview https://bit.ly/3r7cx6g StorageOS Documentation https://docs.storageos.com Supporting Civo’s Lightning-Fast Managed Kubernetes Development and Deployment https://bit.ly/3r4S1nD Speak to StorageOS Engineers on Slack https://slack.storageos.com Performance Benchmarking Cloud Native Storage Solutions https://bit.ly/3l8cTqQ