![In-tree plugins for Windows
q File based cloud volumes:
o Azure File through SMB
q Block based cloud volumes with attach support:
o Azure Disk
o GCE PD
o EBS [WIP]](https://image.slidesharecdn.com/storagewindowskube-190506174200/85/Storage-for-Windows-workloads-in-Kubernetes-19-320.jpg)
Storage for Windows workloads in Kubernetes
- 1. Anusha Ragunathan Senior Software Engineer, Docker Inc. Persistent Storage for Windows workloads in Kubernetes Deep Debroy Engineering Manager, Docker Inc.
- 2. o Kubernetes and Windows o Persistent Storage in Kubernetes o Storage Plugins in Windows Agenda
- 4. Kubernetes and Windows o Windows support is stable in Kubernetes v1.14 o Worker nodes need to run Windows Server 2019 o Kubernetes master nodes need to be Linux o Docker engine is the supported container runtime
- 5. Kubernetes and Windows q Active Directory integration through GMSA o Alpha in v1.14 q Support for ContainerD through CRI o In design phase q Support for multiple CNI plugins o Azure CNI, Docker CNI, Flannel q Persistent Storage
- 6. Kubernetes Architecture Kubernetes Master Scheduler API Server Controller Node Kubelet Kube-proxy CNI/CRI/CSI plugins Cloud Datacenter Network Fabric Datacenter Storage Fabric Node Kubelet Kube-proxy CNI/CRI/CSI plugins Node Kubelet Kube-proxy CNI/CRI/CSI plugins
- 7. q Cluster-wide controller loops in master nodes o PV controller o Attach/Detach controller o Node controller q OS-specific logic in worker nodes o Kubelet o Container runtimes o Flexvolume and CSI node plugins Key components of Kubernetes
- 9. Kubernetes Jargon ● A storage claim made by a user ● Just like how Pods consume Compute resources, PVC consume Volume resources. ● Just like how Pods can request specific levels CPU & memory, PVCs can request specific sizes and access modes. ● Pods reference PVC Persistent Volume Claim (PVC) Persistent Volume (PV) ● Storage resources in a cluster ● Lifecycle independent of a Pod Pods ● A set of running containers representing a workload
- 10. Lifeof a PV CreateVolume AttachVolume (to a Node) UnMountVolume (from Pod) MountVolume (to Pod and format if necessary) DetachVolume (from Node) DeleteVolume (after use) Storage Service PodPod PodPod PodPod
- 11. ● Provisioning is the creation/allocation of Persistent Volumes ● Static and Dynamic Provisioning ○ Static: pre creation of PV ○ Dynamic: automatic creation of PV based on size, permissions requested. ● Dynamic Provisioning through Storage Classes ○ Provides a way for Admins to describe “classes” of storage available. Eg, different performance SLAs, value-add features such as replication, backup, etc ○ Backed by a provisioner PV Provisioning
- 12. Provisioning Workflow (static) 1. Cluster Admin pre-provisions volumes and registers PVs 2. Developer claims a PV from the pool 4. Developer references the claim in a Pod 3. Controller BINDS PV to PVC Pool of Persistent Volumes NFS PV iSCSI PV EBS PV Claim `Claim ClaimClaim Pod Claim 5. Controller inspects Claim and MOUNTs the PV into the Pod.
- 13. Provisioning Workflow (dynamic) 1. Cluster Admin registers Storage Classes 2. Developer defines a claim by referring a storage class 5. Developer references the claim in a Pod 4. Controller BINDS PV to PVC 6. Controller inspects Claim and MOUNTs the PV into the Pod. SSD Pod Claim Claim Persistent Volumes Storage Classes Slow Fast 3. Controller provisions volumes on-demand
- 14. • In-tree volume plugins • Flexvolume plugins • CSI plugins • External provisioners Persistent storage interfaces
- 15. Plugin Type Pros Cons Future InTree No extra installation necessary Release cycles tied to k8s releases Superseded by CSI External Provisioner Code maintained independently Limited customization for attach and mount ops Superseded by CSI FlexVolume Customizable code for attach/mount maintained independently Host based exec model reduces portability. Plugin lifecycle is non- native to k8s. Deprecated for Linux Will be used for Windows CSI Highly customizable code maintained independently and based on standard Requires installation and configuration The future of storage plugins Persistent Storage Interfaces
- 17. Considerations for Windows q Disk device and Volume enumeration interfaces o Object based (rather than file based) q File System support o NTFS for block o SMB for shared q Powershell cmdlets/scripts for many actions
- 18. In-tree storage support for Windows q Core support in kubelet.exe to: o Partition disk and format volumes with NTFS o Link volumes to container's file system o Link SMB shares to container's file system
- 19. In-tree plugins for Windows q File based cloud volumes: o Azure File through SMB q Block based cloud volumes with attach support: o Azure Disk o GCE PD o EBS [WIP]
- 20. In tree Plugin Architecture Worker Worker Master Storage Service Kubelet Kubelet AttachVolum e DetachVolu me CreateVolum e DeleteVolum e Kube Core Components MountVolume UnmountVolume AttachVolume DetachVolume CreateVolume Delete Volume In-tree Plugins PV Claim API Server
- 21. Demo 1: In-tree storage with Azure File Worker Worker Master Azure File Storage
- 22. Flexvolume plugins for Windows q Attach/mount pre-provisioned volumes over: o SMB (shared file system) o iSCSI (dedicated block) q Dynamic provisioning with external provisioners: o Set PV source to supported Flexvolume plugins
- 23. ExternalProvisionerArchitecture Worker Worker Master Storage Service Kubelet Kubelet External Provisioner (StatefulSet/Deployment Pod) Kube Core Components External components API Server MountVolume UnmountVolume AttachVolume DetachVolume CreateVolume Delete Volume In-tree Plugins PV Claim PV FlexVolume Plugin
- 24. Demo 2: External Provisioning + iSCSI Worker Worker Master ISCSI Target Server iSCSI LUN
- 25. CSI plugins for Windows q CSI support for Windows is a Work in Progress. q No privileged support for containers in Windows yet q Options for deploying plugin o Host process o Container with privileged host-based proxy
- 26. Summary q In-tree and Flexvolume plugins ready to be used. q External provisioners coming soon. q CSI plugins for Windows node plugins in future.
- 27. What's next Support for Windows workloads in Kubernetes clusters coming in Docker Enterprise Tech Preview!!
- 28. Thank You Q&A
- 29. Backup slides
- 30. CSI node plugins as host process q Plugin process runs directly on host q Handles plugin registration with kubelet o Cannot use node driver registrar sidecar o Other side cars can be scheduled on Linux masters. q Independent life-cycle and log management
- 31. CSI node plugins as container q Plugin process packaged and deployed as containers q Use node driver registrar for plugin registration q Life-cycle and log management through Kubernetes q Requires a privileged proxy in host for operations
- 32. CSIPlugins Worker Worker Storage Service Kubelet Kubelet External Provisioner (Deployment/ StatefulSet) CreateVolume DeleteVolume External Attacher (Deployment/ StatefulSet) CSI Controller (Deployment/ StatefulSet) CSI Node (DaemonSet) Kube Core Components Kube Sidecar containers CSI plugin components ControllerPublish ControllerUnPublish Master API Server NodeStage/NodeUnstage NodePublish/NodeUnpublish PV PV Claim volume attachment CSI InTree