Why Kubelet is Crucial in Kubernetes: Pod Scheduling, Health Checks & More
Chaitanya Sawant
Full Stack Developer || 4x Kubernetes Certified || CKA | CKAD | KCNA | KCSA || Docker || Nodejs || MongoDB || Reactjs || Nextjs || Remixjs || JavaScript || Typescript
Kubelet is an essential component of the Kubernetes control loop. It is the primary agent that runs on each node in the cluster and is responsible for ensuring that the containers specified in a PodSpec are running as expected.
1. Overview of Kubelet
Kubelet is an agent running on every node (worker & control plane nodes).
It communicates with the Kubernetes API Server.
It ensures that the containers described in Pod manifests are running.
It interacts with the Container Runtime Interface (CRI) to start and stop containers.
It collects metrics and logs from running containers.
It supports Pod health checks and readiness probes.
2. Kubelet Workflow
The kubelet operates in a continuous loop:
Retrieves PodSpecs Fetches Pod definitions from the API server or local manifest files.
Validates PodSpecs Ensures the requested resources are available.
Interacts with the Container Runtime Calls the Container Runtime Interface (CRI) to schedule containers. Supports container runtimes like Docker, containerd, and CRI-O.
Monitors Container Health Periodically checks the health of running containers. Uses liveness, readiness, and startup probes.
Reports Status to the API Server Reports back if a pod is running, failed, or needs to be restarted.
3. Key Features of Kubelet:
a. Enforces the Desired State on Each Node
Kubernetes follows a declarative model where users define how their applications should run.
The Kubernetes API server assigns workloads (pods) to nodes, but the API server itself doesn’t execute them.
Kubelet is responsible for ensuring that the assigned pods are running.
If a pod crashes or is deleted, kubelet ensures it is restarted as per the PodSpec.
🔹 Without kubelet: The control plane could schedule pods, but they wouldn't actually start running on nodes.
b. Manages Pod Lifecycle
Kubelet monitors the entire lifecycle of a pod on a node:
Starts pods using the Container Runtime Interface (CRI).
Periodically checks pod health.
Stops or restarts pods if needed.
🔹 Without kubelet: Pods would not be monitored, and failures would go undetected, leading to unreliable applications.
c. Communicates with the API Server
Kubelet acts as the node’s interface to the Kubernetes API.
It reports: The status of pods running on the node. The health of the node itself. Resource usage (CPU, memory, disk, network).
🔹 Without kubelet: The control plane would have no visibility into the node’s health, leading to scheduling issues and blind spots in monitoring.
d. Interacts with the Container Runtime
Kubernetes does not run containers directly; it delegates this to container runtimes (like containerd or CRI-O).
Kubelet uses the Container Runtime Interface (CRI) to instruct the runtime to:
Pull container images.
Start and stop containers.
Manage container networking.
🔹 Without kubelet: The containers inside a pod would never be created, since the API server doesn’t interact directly with the container runtime.
e. Ensures Pod Networking & Storage
Kubelet works with CNI (Container Network Interface) plugins to: Assign IP addresses to pods. Ensure network policies are applied.
It also integrates with CSI (Container Storage Interface) to:
Mount persistent storage.
Ensure storage volumes are available to pods.
🔹 Without kubelet: Pods wouldn't get network connectivity or access to persistent storage, making many workloads unusable.
f. Handles Health Checks & Probes
Kubelet manages: Liveness probes (restart containers if they become unresponsive). Readiness probes (determine if a pod is ready to receive traffic). Startup probes (ensure a pod has fully started before sending traffic).
🔹 Without kubelet: Kubernetes wouldn’t know if a container is healthy or needs restarting, leading to increased downtime.
g. Collects Metrics & Logs
Kubelet integrates with cAdvisor to collect CPU, memory, and network statistics.
It enables kubectl logs and kubectl top commands.
🔹 Without kubelet: Metrics and logs wouldn’t be available, making monitoring and debugging difficult.
h. Runs Static Pods (For Control Plane Components)
On control plane nodes, kubelet can run static pods (e.g., API server, etcd).
These are defined via local manifest files and do not depend on the API server.
🔹 Without kubelet: The control plane itself might not function in certain Kubernetes deployments.
4. Kubelet Configuration
a. Running Kubelet as a System Service
On most Kubernetes distributions, kubelet runs as a systemd service:
b. Key Configuration Files
/etc/kubernetes/kubelet.conf → Configuration for kubelet.
/var/lib/kubelet/config.yaml → Main kubelet config.
c. Command-line Flags
Kubelet accepts various flags for customization:
Common flags:
-container-runtime-endpoint → Defines the CRI endpoint.
-fail-swap-on=false → Allows running on a node with swap enabled.
5. Kubelet & Static Pods
Static pods are directly managed by kubelet without the API server.
They are defined using local files (/etc/kubernetes/manifests).
Used for control plane components like API Server, Scheduler, etc.
Example:
6. Troubleshooting Kubelet
a. Checking Logs
b. Restarting Kubelet
c. Debugging CRI Issues
7. Kubelet Security
a. Authentication & Authorization
Uses client certificates (/etc/kubernetes/kubelet.conf).
Authenticates with API server using TLS.
b. RBAC (Role-Based Access Control)
Kubelet permissions are managed via ClusterRoles.
c. Pod Security Policies
Enforces security constraints on pods.
Summary
Kubelet is the backbone of Kubernetes nodes, ensuring that pods run as expected. It works with the API server, CRI, CNI, and CSI to manage container lifecycles, networking, and storage. Understanding kubelet is crucial for troubleshooting Kubernetes at the node level.
