Kubernetes the Very Hard Way. Lisa Portland 2019
7 likes2,644 views
The document elaborates on the complexities and challenges faced in deploying Kubernetes at scale by Datadog, including the necessity for a resilient control plane, management of certificates, and efficient networking strategies. It discusses incidents related to certificate management, networking inefficiencies, and the importance of graceful termination in services while highlighting learnings from practical experiences. The presentation emphasizes that self-managed Kubernetes is challenging and encourages using managed services, underscoring the necessity for team building and user training.
Kubernetes the Very Hard Way. Lisa Portland 2019
- 1. Kubernetes the Very Hard Way Laurent Bernaille Staff Engineer, Infrastructure @lbernail
- 2. lbernail Datadog Over 350 integrations Over 1,200 employees Over 8,000 customers Runs on millions of hosts Trillions of data points per day 10000s hosts in our infra 10s of k8s clusters with 50-2500 nodes Multi-cloud Very fast growth
- 3. lbernail Why Kubernetes? Dogfooding Improve k8s integrations Immutable Move from Chef Multi Cloud Common API Community Large and Dynamic
- 4. The very hard way?
- 5. It was much harder
- 6. lbernail This talk is about the fine prints “Of course, you will need a HA master setup” “Oh, and yes, you will have to manage your certificates” “By the way, networking is slightly more complicated, look into CNI / ingress controllers”
- 7. lbernail What happens after “Kube 101” 1. Resilient and Scalable Control Plane 2. Securing the Control Plane a. Kubernetes and Certificates b. Exceptions? c. Impact of Certificate Rotation 3. Efficient networking a. Giving pod IPs and routing them b. Accessing services: Client-side load-balancing: c. Ingresses: Getting data in the cluster
- 8. lbernail What happens after “Kube 101” 1. Resilient and Scalable Control Plane 2. Securing the Control Plane a. Kubernetes and Certificates b. Exceptions? c. Impact of Certificate Rotation 3. Efficient networking a. Giving pod IPs and routing them b. Accessing services: Client-side load-balancing: c. Ingresses: Getting data in the cluster
- 9. Resilient and Scalable Control Plane
- 10. lbernail Kube 101 Control Plane kubelet kubectl etcd apiserver controllersscheduler Master
- 11. lbernail Making it resilient etcd apiserver controllersscheduler kubelet kubectl Master etcd apiserver controllersscheduler Master etcd apiserver controllersscheduler Master LoadBalancer
- 12. lbernail Separate etcd nodes apiserver controllersscheduler kubelet kubectl Master apiserver controllersscheduler Master apiserver controllersscheduler Master LoadBalancer etcdetcdetcdetcdetcd
- 13. lbernail Single active Controller/scheduler apiserver controllersscheduler kubelet kubectl Master apiserver controllersscheduler Master apiserver controllersscheduler Master LoadBalancer etcdetcdetcdetcdetcd
- 14. lbernail Split scheduler/controllers apiserver kubelet kubectl apiserver apiserver LoadBalancer etcdetcdetcdetcdetcd scheduler scheduler controllers controllers
- 15. lbernail 1. Resilient and Scalable Control Plane 2. Securing the Control Plane a. Kubernetes and Certificates b. Exceptions? c. Impact of Certificate Rotation 3. Efficient networking a. Giving pod IPs and routing them b. Accessing services: Client-side load-balancing: c. Ingresses: Getting data in the cluster What happens after “Kube 101”
- 17. lbernail From “the hard way”
- 18. lbernail “Our cluster broke after ~1y”
- 19. lbernail Certificates in Kubernetes ● Kubernetes uses certificates everywhere ● Very common source of incidents ● Our Strategy: Rotate all certificates daily
- 20. lbernail Certificate management etcd apiserver Vault etcd PKIPeer/Server cert Etcd Client cert
- 21. lbernail Certificate management etcd apiserver controllers scheduler Vault etcd PKIPeer/Server cert Etcd Client cert kube PKI Apiserver/kubelet client cert Controller client cert Scheduler client cert kubelet Kubelet client/server cert
- 22. lbernail Certificate management etcd apiserver controllers scheduler Vault etcd PKIPeer/Server cert Etcd Client cert kube PKI Apiserver/kubelet client cert kube kv SA public key SA private key Controller client cert Scheduler client cert In-cluster app SA token kubelet Kubelet client/server cert
- 23. lbernail Certificate management etcd apiserver controllers scheduler apiservice webhook... Vault etcd PKIPeer/Server cert Etcd Client cert apiservice PKI Apiservice cert (proxy/webhooks) kube PKI Apiserver/kubelet client cert kube kv SA public key SA private key Controller client cert Scheduler client cert In-cluster app SA token kubelet Kubelet client/server cert
- 24. lbernail Certificate management etcd apiserver controllers scheduler apiservice webhook... Vault etcd PKIPeer/Server cert Etcd Client cert apiservice PKI Apiservice cert (proxy/webhooks) kube PKI Apiserver/kubelet client cert kube kv SA public key SA private key Controller client cert Scheduler client cert OIDC provider kubectl OIDC auth In-cluster app SA token kubelet Kubelet client/server cert
- 26. lbernail Kubelet: TLS Bootstrap apiserver controllers Vault kube PKI kube kv3- Get signing key admin 1- Create Bootstrap token 2- Add Bootstrap token to vault
- 27. lbernail Kubelet: TLS Bootstrap apiserver controllers Vault kube PKI kube kv kubelet 5- Verify RBAC for CSR creator 6- Sign certificate 1- Get Bootstrap token 2- Authenticate with token 4- Create CSR 7- Download certificate 8- Authenticate with cert 9- Register node 3- Verify Token and map groups
- 28. lbernail Kubelet certificate issue 1. One day, some Kubelets were failing to start or took 10s of minutes 2. Nothing in logs 3. Everything looked good but they could not get a cert 4. Turns out we had a lot of CSRs in flight 5. Signing controller was having a hard time evaluating them all CSR resources in the cluster Lower is better!
- 29. lbernail Why? Initial creation 1. Authenticate with bootstrap token, mapped to group “system:bootstrappers” 2. Create CSR 3. “system:bootstrappers” has role “system:certificates.k8s.io:certificatesigningrequests:nodeclient” Renewal 1. Authenticate with current node certificate, mapped to group “system:nodes“ 2. Create CSR 3. Not allowed for auto-sign Also needed for “system:nodes”
- 31. lbernail Temporary solution apiserver webhook Vault kube kv Get cert and key admin Create webhook with self-signed cert as CA Add self-signed cert + key to Vault One day, after ~1 year ● Creation of resources started failing (luckily only a Custom Resource) ● Cert had expired...
- 32. lbernail Take-away ● Rotate server/client certificates ● Not easy But, “If it’s hard, do it often” > no expiration issues anymore
- 34. lbernail Apiserver restartsapiserver restarts etcd slow queries etcd traffic We have multiple apiservers We restart each daily Significant etcd network impact (caches are repopulated) Significant impact on etcd performances
- 35. lbernail Apiserver restarts, continued apiserver restarts coredns memory usage ● Apiserver restarts ● clients reconnect and refresh their cache > Memory spike for impacted apps No real mitigation today
- 36. lbernail Unbalanced apiserver traffic Number of connections / traffic very unbalanced Because connections are very long-lived More clients => Bigger impact clusterwide 15MB/s 2.5MB/s 2300 connections 300 connections
- 37. lbernail Take-away Restarting components is not transparent ○ Not limited to apiservers, some issues with the Kubelet too It would be great if ○ Components could transparently reload certs (server & client) ○ Clients could wait 0-Xs to reconnect to avoid thundering herd ○ Reconnections did not trigger memory spikes ○ Connections were rebalanced (kill them after a while?)
- 38. lbernail What happens after “Kube 101” 1. Resilient and Scalable Control Plane 2. Securing the Control Plane a. Kubernetes and Certificates b. Exceptions? c. Impact of Certificate Rotation 3. Efficient networking a. Giving pod IPs and routing them b. Accessing services: Client-side load-balancing: c. Ingresses: Getting data in the cluster
- 40. lbernail Throughput Trillions of data points daily Scale 1000-2000 nodes clusters Network challenges Latency End-to-end pipeline Topology Multiple clusters Access from standard VMs
- 41. Giving pods IPs & Routing them
- 42. lbernail From “the Hard Way” node IP Pod CIDR for this node
- 43. lbernail Small cluster? Static routes Node 1 IP: 192.168.0.1 Pod CIDR: 10.0.1.0/24 Routes (local or cloud provider) 10.0.1.0/24 => 192.168.0.1 10.0.2.0/24 => 192.168.0.2 Node 2 IP: 192.168.0.2 Pod CIDR: 10.0.2.0/24
- 44. lbernail Mid-size cluster? Overlay Node 1 IP: 192.168.0.1 Pod CIDR: 10.0.1.0/24 Node 2 IP: 192.168.0.2 Pod CIDR: 10.0.2.0/24 VXLAN VXLAN Tunnel traffic between hosts Examples: Calico, Flannel
- 45. lbernail Large cluster with a lot of traffic? Native pod routing Performance Datapath: no Overlay Control plane: simpler Addressing Pod IPs are accessible from ● Other clusters ● VMs
- 46. lbernail In practice On premise BGP Calico Kube-router AWS Additional IPs on ENIs AWS EKS CNI plugin Lyft CNI plugin Cilium ENI IPAM GCP IP aliases
- 47. lbernail A bit more complex on AWS eth1 agent Pod 1 Pod 2 kubelet cni containerd CRI CNI eth0 Attach ENI Allocate IPs Create veth ip 1 ip 2 ip 3 Routing rule “From IP1, use eth1” Routing eth0 ip 1
- 48. lbernail Take-away ● Native pod routing has worked very well at scale ● A bit more complex to debug ● Much more efficient datapath ● Topic is still dynamic (Cilium introduced ENI recently) ● Great relationship with Lyft / Cilium
- 50. lbernail Kube-proxy default: iptables # Mid size cluster iptables -S -t nat | wc -l 48688
- 51. lbernail Alternative: IPVS Pod X Pod Y Pod Z Service ClusterIP:Port S Backed by pod:port X, Y, Z Virtual Server S Realservers - X - Y - Z kube-proxy apiserver Watches endpoints Updates IPVS
- 52. lbernail New connection Pod X Pod Y Pod Z Virtual Server S Realservers - X - Y - Z kube-proxy apiserver IPVS conntrack App A => S >> A => X App establishes connection to S IPVS associates Realserver X
- 53. lbernail Pod X deleted Pod X Pod Y Pod Z Virtual Server S Realservers - Y - Z kube-proxy apiserver IPVS conntrack App A => S >> A => X Apiserver removes X from S endpoints Kube-proxy removes X from realservers Kernel drops traffic (no realserver)
- 54. lbernail Pod X deleted Pod X Pod Y Pod Z Virtual Server S Realservers - Y - Z kube-proxy apiserver IPVS conntrack App A => S >> A => X Pod X sends FIN on exit Conntrack entry deleted Connection from A terminates FIN
- 55. lbernail What if X doesn’t send FIN? Pod Y Pod Z Virtual Server S Realservers - Y - Z kube-proxy apiserver IPVS conntrack App A => S >> A => X Traffic blackholed until App detects issue > tcp_retries2 (default 15) > ~15mn
- 56. lbernail Mitigation Pod Y Pod Z Virtual Server S Realservers - Y - Z kube-proxy apiserver IPVS conntrack App A => S >> A => X net/ipv4/vs/expire_nodest_conn Delete conntrack entry on next packet Forcefully terminate (RST) RST
- 57. lbernail Limit? ● No graceful termination ● As soon as a pod is Terminating connections are destroyed ● Addressing this took time
- 58. lbernail Graceful termination Pod X Pod Y Pod Z Virtual Server S Realservers - X Weight:0 - Y - Z kube-proxy apiserver IPVS conntrack App A => S >> A => X Apiserver removes X from S endpoints Kube-proxy sets Weight to 0 No new connection
- 59. lbernail Garbage collection Pod X Pod Y Pod Z Virtual Server S Realservers - Y - Z kube-proxy apiserver IPVS conntrack App A => S >> A => X Pod exits and sends FIN Kube-proxy removes realserver when it has no active connection FIN
- 60. lbernail What if X doesn’t send FIN? Pod Y Pod Z Virtual Server S Realservers - X Weight:0 - Y - Z kube-proxy IPVS conntrack App A => S >> A => X Conntrack entries expires after 900s If A sends traffic, it never expires Traffic blackholed until App detects issue ~15mn > Mitigation: lower tcp_retries2
- 61. lbernail Take-away ● IPVS has been great for us ● IPVS is in a good state now ● Several improvements in the works ● But harder than we expected ● I ended up reviewer/approver for kube-proxy/IPVS
- 62. Ingresses
- 63. lbernail Ingress: cross-clusters, VM to clusters A A A B B B C C D D Cluster 1 Cluster 2Classic (VM) C? C? B?
- 64. lbernail Master Kubernetes default: LB service External Client Load-Balancer pod pod pod kube-proxy kube-proxy kube-proxy NP NP NP Healthchecker data path health checks configuration (from watching ingresses on apiservers) service-controller
- 65. lbernail Master Inefficient Datapath & cross-application impacts Web traffic Load-Balancer web-1 web-2 web-3 kube-proxy kube-proxy kube-proxy NP NP NP Healthchecker data path health checks configuration (from watching ingresses on apiservers) service-controller kafka
- 66. lbernail Master ExternalTrafficPolicy: Local? Web traffic Load-Balancer web-1 web-2 web-3 kube-proxy kube-proxy kube-proxy NP NP NP Healthchecker data path health checks configuration (from watching ingresses on apiservers) service-controller kafka
- 67. lbernail L7-proxy ingress controller data path health checks configuration from watching ingresses/endpoints on apiservers (ingress-controller) from watching LoadBalancer services (service-controller) External Client Load-Balancer l7proxy l7proxy kube-proxy kube-proxy kube-proxy NP NP NP Heathchecker ingress-controller pod pod pod pod Create l7proxy deployments Update backends using service endpoints Master service-controller
- 68. lbernail Limits All nodes as backends (1000+) Inefficient datapath Cross-application impacts Alternatives? ExternalTrafficPolicy: Local? > Number of nodes remains the same > Issues with some CNI plugins K8s ingress > Still load-balancer based > Need to scale ingress pods > Still inefficient datapath Challenges
- 69. lbernail Our target: native routing External Client ALB pod pod pod Healthchecker data path health checks alb-ingress-controller configuration (from watching ingresses/endpoints on apiservers)
- 70. lbernail Limited to HTTP ingresses No support for TCP/UDP Ingress v2 should address this Remaining challenges Registration delay Slow registration with LB Pod rolling-updates much faster Mitigations - MinReadySeconds - Pod ReadinessGates
- 71. lbernail Workaround External Client Load-Balancer l7proxy l7proxy Heathchecker pod pod pod pod Not managed by k8s Dedicated nodes Pods in host network TCP / Registration delay not manageable > Dedicated gateways
- 72. lbernail Take-away ● Ingress solutions are not great at scale yet ● May require workarounds ● Definitely a very important topic for us ● The community is working on v2 Ingresses
- 73. Conclusion
- 74. lbernail A lot of other topics ● DNS (it’s always DNS!) ● Challenges with Stateful applications ● How to DDOS <insert ~anything> with Daemonsets ● Node Lifecycle ● Cluster Lifecycle ● Deploying applications ● ...
- 75. lbernail You want more horror stories? “Kubernetes the very hard way at Datadog” https://www.youtube.com/watch?v=2dsCwp_j0yQ “10 ways to shoot yourself in the foot with Kubernetes” https://www.youtube.com/watch?v=QKI-JRs2RIE “Kubernetes Failure Stories” https://k8s.af
- 76. lbernail Key lessons Self-managed Kubernetes is hard > If you can, use a managed service Networking is not easy (especially at scale) The main challenge is not technical > Build a team > Transforming practices and training users is very important