Right, let’s talk about getting your container images onto your nodes. This is one of those things you blissfully ignore until it isn’t working, and then it becomes the single most infuriating bottleneck in your entire deployment. A slow ImagePull can turn a rapid, 30-second rollout into a minutes-long agonizing wait, or worse, cause your shiny new Pod to fail and get stuck in ImagePullBackOff hell. We’re going to fix that. We’re going to make your image pulls so efficient it’ll make the container registry blush.

The core problem is simple: Kubernetes needs the image to run the container. By default, the kubelet on each node only pulls the image when a Pod scheduling decision tells it to. This is a “just-in-time” model, and like its supply chain counterpart, it’s brilliant until it’s catastrophically not. If the image is large, your registry is slow, or the network is having a bad day, your Pod just sits there in ContainerCreating purgatory.

The Default: imagePullPolicy: IfNotPresent

This is the sensible, conservative default. The kubelet checks its local container runtime (containerd, Docker, etc.) and asks, “Hey, do you already have my-app:v1.2.3 cached?” If the answer is yes, it gets on with its life. If not, it goes and pulls it. This is fine for development where you’re bouncing on a single node. It’s a disaster in production when you need to scale a deployment to 10 new nodes simultaneously. All 10 nodes will suddenly realize they don’t have the image and will proceed to hammer your registry at the exact same time, creating a thundering herd of pull requests. Don’t be that person.

The Nuclear Option: imagePullPolicy: Always

This tells the kubelet, “I don’t care what you have cached, go ask the registry if there’s a new one every single time.” This is useful for tags like :latest in a development environment where you want absolute certainty you’re running the newest build. In production? It’s a performance nightmare and a stability risk. You are guaranteeing a network call to your registry for every single Pod startup. If your registry goes down, your entire cluster grinds to a halt because no Pod can ever be created again. Use this with extreme prejudice, or better yet, don’t use mutable tags like :latest in prod at all. Use immutable tags (e.g., git SHA, build timestamp) and rely on IfNotPresent.

Pre-Pulling: Beating the Scheduler to the Punch

This is the first big weapon in our arsenal. The idea is brutally simple: get the image onto the node before any Pod needs it. This completely decouples the image distribution problem from the Pod scheduling problem.

The most robust way to do this is a DaemonSet. You create a DaemonSet whose sole job is to run a Pod that does nothing but pull your image. Since it’s a DaemonSet, it runs on every node (or every node with a specific label), effectively pre-warming the cache across your entire cluster.

Here’s a sample manifest. Note the clever bits: the imagePullPolicy: IfNotPresent (we don’t need to re-pull it if it’s already there), the command that just sleeps forever (its job is just to force the pull, then exist), and the tolerations to ensure it runs on all nodes, even control planes.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: image-pre-puller-my-app
  namespace: my-app
spec:
  selector:
    matchLabels:
      name: image-pre-puller-my-app
  template:
    metadata:
      labels:
        name: image-pre-puller-my-app
    spec:
      # This toleration is key for running on all nodes
      tolerations:
        - operator: Exists
      containers:
      - name: puller
        image: my-registry.com/my-app:immutable-sha-abc123
        command: ["/bin/sh", "-c"]
        args: ["echo 'Image pulled successfully'; sleep infinity"]
        imagePullPolicy: IfNotPresent
      terminationGracePeriodSeconds: 0

You apply this before your main deployment. Once its Pods are Running, you know every node has the image cached. Now, when your real Deployment rolls out, its Pods will start near-instantly because the kubelet finds the image already present. It’s beautiful.

The Black Magic of Image Streaming

Now, let’s talk about something genuinely cool that you probably aren’t using but should be: image streaming. This is where the container runtime (containerd with CRI-plugin since v1.5) gets smart. Instead of waiting for the entire image tar.gz to be downloaded before starting the container, it can start the container almost immediately while the image layers are still being pulled in the background.

This is a game-changer for large images. Your app can be initializing, connecting to databases, and loading classes while the last 20% of the image (maybe docs, maybe seldom-used binaries) is still trickling in. It’s like starting a video game while it’s still installing.

The best part? You don’t need to do anything to enable it! If you’re on a reasonably modern Kubernetes cluster (and you should be), containerd does this by default. The kubelet and CRI just handle it. Your Pod will transition to Running state much earlier than the image pull is technically “complete.”

But—and there’s always a but—your application needs to be written to handle this. If your app’s entrypoint immediately tries to exec a binary that happens to be in one of the last layers to be downloaded, it’ll fail with a “file not found” error. So the best practice is to have your entrypoint script be something simple and early in the image layers, and have it do a quick check for any critical files it needs before proceeding with the real startup.

Best Practices and Pitfalls

  1. Use Immutable Tags: This is non-negotiable. Pre-pulling my-app:latest is a waste of time. What does latest mean? It’s a moving target. Pre-pull a specific, immutable digest like my-app@sha256:abc123.... This is deterministic and safe.

  2. Clean Up Your Pre-Pullers: That DaemonSet is now a permanent resident on your node. If you need to pre-pull a new version, you update the DaemonSet. The old Pods will terminate, but the images they pulled remain cached. You might need a separate cron job or tool (like docker-gc) to clean out old, unused images to avoid filling up the node’s disk.

  3. Understand Your Registry’s Limits: Pre-pulling from a single node is polite. A thundering herd of nodes pulling at once can DDoS a poorly configured registry. If you can’t pre-pull, consider a pull-through cache or a registry with a massive bandwidth budget.

  4. imagePullSecrets: Don’t forget these! If your registry is private, your pre-puller DaemonSet needs the same imagePullSecrets as your application Pods to authenticate. Nothing fails harder than a DaemonSet full of ImagePullBackOff because it has no permissions.

The goal here is to make the image a non-issue. By pre-pulling, you turn a variable, network-dependent operation into a predictable, node-local one. Combine that with the magic of image streaming, and you’ve effectively neutered one of the most common performance killers in Kubernetes. Now your deployments can be as fast as your nodes can start processes, not as fast as your internet connection allows.