Best practices for Highly available apps on k8s.pptx
Download as PPTX, PDF0 likes35 views
This document outlines best practices for achieving high availability in Kubernetes applications. Key topics include handling container processes, implementing liveness and readiness probes, managing pod shutdowns, and employing various deployment strategies like blue-green and canary deployments. It also addresses disturbance types and how to mitigate impacts from both voluntary and involuntary disturbances in containerized environments.
![preStop hook more than LB Health internal
lifecycle:
preStop:
exec:
# Sleeping before shutdown allows app to process requests that
were
# in-flight before the node is removed from load-balancing.
# If using an external load balancer, you may need to increase
this
# duration to be greater than the LB's health check interval.
command: ["sleep", "${lb_healthcheck_internal + 5}"]](https://image.slidesharecdn.com/highlyavailableappsonk8s-241219091432-5920362f/85/Best-practices-for-Highly-available-apps-on-k8s-pptx-20-320.jpg)
Best practices for Highly available apps on k8s.pptx
- 1. Highly available apps on k8s - Prashant Kalkar
- 2. Agenda ● Get your container process right. ● Starting up healthy - Liveness Readiness Probes. ● Pod graceful shutdowns. ● Working with Cloud Load balancer. ● Scheduling pods ● Deployment strategies ● Disturbances ○ Voluntary ○ involuntary
- 4. Unix Processes ● Manage child processes’ lifecycle. ● Forwarding OS signals (SIGTERM, SIGKILL)
- 5. Container PID = 1 Command: bash -c "java $JAVA_OPTS -jar /opt/app/app.jar"
- 6. Container PID = 1. Graceful shutdown ● PID = 1 should forward OS signals to container process. ● Wait for child termination ● Exit with 0. Bash does not forward any OS signals to child process. So no SIGTERM send to application. The container is simply killed. No Graceful shutdown.
- 8. Fixing the PID 1 issue ● Avoid shell format : ● Use exec format: ● Use exec: ● Replace bash with better init process (See tini) Ensure your app does handle the SIGTERM signal (all modern frameworks do).
- 10. Liveness and Readiness Probe Readiness probe ● Ensures application running before traffic is send to it. ● Reduces possibilities of errors when application is not ready. ● Can mark pod as not ready to avoid new traffic under high load. ● Readiness plays role in rolling deployments Liveness probe ● Tries to restore failed container by killing it. ● Ensure Liveness probe is not aggressive to prevent excessive restarts. ● Not recommended for clustered applications.
- 14. Kubelet pod termination in more details
- 16. preStop hook to delay Pod termination
- 17. Ensure High availability ● preStop hook and terminationGracePeriodSeconds controls the pod termination process (kubelet). ● Ensure preStop hook >= Time taken to stop traffic + Time required to complete ongoing request. ● Can slow down rolling deployments
- 18. What about Cloud Load Balancer
- 19. Consider scenario ● Start a rolling update of the deployment ● Rollout of new pods takes less time than it takes the Load Balancer controller to register the new pods and for their health state turn »Healthy« in the target group ● So even when pod is ready and deployment mark the pod as available. LB is not sending traffic. ● Since deployment available replica count is satisfied one of the healthy older pod start terminating. ● Thus making less pods available for the service from the LB point of view.
- 20. preStop hook more than LB Health internal lifecycle: preStop: exec: # Sleeping before shutdown allows app to process requests that were # in-flight before the node is removed from load-balancing. # If using an external load balancer, you may need to increase this # duration to be greater than the LB's health check interval. command: ["sleep", "${lb_healthcheck_internal + 5}"]
- 22. Scheduling pods
- 23. Inter-pod Anti-affinity for high availability
- 24. Topology Spread Across Node and Zone
- 30. Disturbances
- 31. Voluntary disturbance ● Draining a node ● Rolling deployments ● Deleting a pod
- 32. Pod Disruption Budgets (PDB)
- 33. Involuntary disturbance Evictions ● Node out of resources ● Hardware failure Preemption ● Pod replaced due to high priority pod.
- 34. How to reduce Impact for Involuntary disturbance ● QoS - Guaranteed, Bustable, BestEffort ● Cluster autoscaling (and low priority empty pods) ● PDB can not prevent but count Involuntary disturbances.
- 36. Guaranteed
- 37. Bustable
- 38. BestEffort
- 39. Fast scaling with Overprovisioning and Preemption
- 40. Fast scaling with Overprovisioning and Preemption
- 41. How to reduce Impact for Involuntary disturbance ● QoS - Guaranteed, Bustable, BestEffort ● Cluster autoscaling (and low priority empty pods) ● PDB can not prevent but count Involuntary disturbances.
- 42. Thank you!