Right, let’s talk about pod startup latency. You’ve deployed your masterpiece, hit that kubectl apply -f command, and are now waiting. And waiting. And… why is this taking so long? It feels like your pod is waiting for a background check before it can run a simple web server. I’ve been there. The truth is, a pod’s journey from “Pending” to “Running” is a gauntlet of bureaucratic checks, and our job is to grease the wheels.

The startup sequence isn’t just your container starting. It’s a multi-stage relay race where any runner can drop the baton. It involves the kube-scheduler finding a node, the kubelet pulling your image and starting the container runtime, and finally, your own startup probes and entrypoints doing their thing. Our goal is to instrument each leg of that race.

Image Pulling: The Biggest Bottleneck

This is, by far, the most common culprit. By default, Kubernetes will pull your image with an IfNotPresent policy. This sounds sane until you realize that “present” means “present on that specific node.” If you’ve rolled out a new version of your app (my-app:v1.0.1) to a 100-node cluster, the kubelet on each node is going to decide it needs to pull that image. This is where things get slow and chatty.

The solution is to embrace image caching. For public images, this isn’t a huge deal, but for your private registry, the latency adds up. The real pro move is to use a imagePullPolicy: Always in your development manifests to avoid maddening caching issues, but switch to IfNotPresent for production once you’re confident in your rollout patterns. For truly massive images, consider using a DaemonSet to pre-pull images onto your nodes, though that’s a bit of a nuclear option.

# A snippet from a Pod spec. This is the first thing to check.
containers:
- name: my-app
  image: my-registry.com/my-app:v1.2.3
  imagePullPolicy: IfNotPresent # The sensible default. Change to 'Always' to force pulls for latest.

But the biggest performance gain comes from making your images smaller. A 1.5 GB image will always pull slower than a 15 MB Alpine-based image. Use multi-stage builds to avoid shipping your build toolchain to production. It’s like moving house; you don’t bring the lumberyard, you bring the finished bookcase.

# A multi-stage Dockerfile example. This is non-negotiable for serious work.
FROM golang:1.21 AS builder
WORKDIR /app
COPY . .
RUN go build -o my-app .

FROM alpine:latest
RUN apk --no-cache add ca-certificates
WORKDIR /root/
COPY --from=builder /app/my-app .
CMD ["./my-app"]

The Scheduling Dance

Before the kubelet even hears about your pod, the kube-scheduler has to find it a home. This is usually fast, but it can become a traffic jam. If your nodes are already resource-starved—meaning they’re low on allocatable CPU or memory—the scheduler has to work harder to find a node that can fit your pod’s requests.

This is why your resource requests are so critical. They’re not just for fairness; they’re a direct input to the scheduler’s algorithm. If you omit them, the scheduler assumes your pod is a no-cost free-loader and can place it anywhere, which sounds great until your pod gets evicted for actually using resources. Always set requests. Be realistic.

# Good pod citizens specify resources. This makes the scheduler's job easy.
resources:
  requests:
    memory: "64Mi"
    cpu: "250m"
  limits:
    memory: "128Mi"
    cpu: "500m"

If you find scheduling is consistently slow, check your kube-scheduler metrics for scheduler_binding_duration_seconds and scheduler_pending_pods. A high number of pending pods usually indicates a cluster that is simply too full or a set of requests that are too greedy.

Kubelet and Container Runtime Overhead

Once the pod is bound to a node, the kubelet takes over. It’s got to create the pod’s sandbox (usually a pause container), set up networking (CNI plugins, I’m looking at you), and finally tell the container runtime (containerd, Docker, etc.) to pull the image and start your container.

The choice of CNI plugin has a non-trivial impact here. Some plugins are faster than others at assigning IP addresses and setting up network rules. If you’re using a complex service mesh like Istio or Linkerd, their sidecar injection adds significant overhead to this phase. It’s a trade-off for the features they provide.

You can see the breakdown of where time is spent by describing a pod after it has started. Look for the ContainerStatuses and its State:

kubectl describe pod my-pod-name

Look for lines like:

  Normal   Scheduled