Alright, let’s talk about Jobs. You’ve got your long-running services (Deployments, StatefulSets) that just hum along forever, and then you’ve got the stuff you actually want to finish. That’s what a Job is for. Think of it as a disposable, single-shot Pod with a very specific purpose: run this container, let it do its work, and when its main process exits with a code of zero, declare victory and go home. Backup scripts, database migrations, data processing batches—this is their home.

The beauty is in the cleanup. A completed Job doesn’t just hang around cluttering up your get pods output. It’s a run-to-completion workload, not a conversation. But, and this is a crucial but, the Job resource itself sticks around afterwards so you can see if it succeeded or failed. The Pods get tidied up.

The Absolute Minimum Job

Let’s start with the simplest possible Job. You want to run echo "Hello, Jobs!" and be done with it. Here’s your weapon of choice.

apiVersion: batch/v1
kind: Job
metadata:
  name: hello-world-job
spec:
  template:
    spec:
      containers:
      - name: hello
        image: busybox:1.35
        command: ["echo", "Hello, Jobs!"]
      restartPolicy: Never
  backoffLimit: 4

Apply this with kubectl apply -f job.yaml. Now, run kubectl get pods and you’ll see something like hello-world-job-abcx1. Once it’s done, its status will change to Completed. Run kubectl logs job/hello-world-job to see your triumphant message. The key here is restartPolicy: Never. This tells the kubelet, “If this container exits, for any reason, do not try to restart it on the same node.” The Job controller is the one in charge of retries by creating entirely new Pods, which is a much more robust way to handle failures.

Handling Failure Like a Pro

So what happens when your script has a bad day and exits with a non-zero code? That’s where spec.backoffLimit comes in. It defines how many times the Job controller should try to create a new Pod before finally giving up and marking the Job as failed. The “backoff” part means it waits longer between each retry (e.g., 10s, 20s, 40s…) to avoid hammering a broken system.

Let’s look at a Job that’s destined to fail.

apiVersion: batch/v1
kind: Job
metadata:
  name: failure-demo
spec:
  template:
    spec:
      containers:
      - name: failer
        image: busybox:1.35
        command: ["sh", "-c", "echo 'I am about to ruin your day...'; exit 1"]
      restartPolicy: Never
  backoffLimit: 3

After you apply this, run kubectl get job/failure-demo -w to watch the drama unfold. You’ll see it create Pods, they’ll fail, and after the third attempt, the Job itself will be marked as Failed. Use kubectl describe job/failure-demo and look at the events; it’s a perfect log of what the Job controller was thinking. This is your first stop for debugging.

Parallelism and Completions

This is where Jobs get powerful. You don’t just have to run one Pod; you can run a whole squad of them, either sequentially or in parallel.

  • spec.completions: How many Pods need to successfully complete for the entire Job to be considered… well, complete. The default is 1.
  • spec.parallelism: How many Pods are allowed to run at the same time. The default is also 1.

Need to process 100 items, but you don’t want to overwhelm your database? Run with completions: 100 and parallelism: 10. It’ll create 10 Pods, wait for them all to finish, create 10 more, and so on, until 100 have succeeded.

Want to run a single task but just get it done faster by throwing multiple instances at it until one wins? That’s a pattern for completions: 1 and parallelism: 5. The first Pod to exit successfully will cause the others to be terminated. This is great for idempotent, high-availability tasks.

apiVersion: batch/v1
kind: Job
metadata:
  name: parallel-processor
spec:
  completions: 10
  parallelism: 2
  template:
    spec:
      containers:
      - name: worker
        image: my-app:latest
        command: ["./process.sh"]
      restartPolicy: OnFailure

The OnFailure Restart Policy

I used restartPolicy: Never above for clarity, but OnFailure is often more practical. The critical difference is this: Never means the Job controller creates a brand new Pod on failure. OnFailure means the kubelet on the node will restart the container inside the existing Pod.

Use OnFailure for transient errors that might be solved by a simple restart, like a dependency that wasn’t quite ready. Use Never when a restart on the same node is unlikely to help, like a bug in your code. The Job controller’s retry (governed by backoffLimit) is a much heavier and more robust operation. Mixing them is an art. A high backoffLimit with OnFailure means you’re telling Kubernetes “try really hard on this node, and if that doesn’t work, try a few other nodes too.”

Common Pitfalls and Rough Edges

  1. Dead Man’s Switch: The biggest gotcha. If the entire control plane or your node pool dies while a Job is running, when the cluster comes back online, the Job controller might not restart the Pod. It thinks the Pod is still running somewhere. You need to set spec.ttlSecondsAfterFinished to automatically clean up done Jobs, but for true fault tolerance across cluster outages, you need an external orchestrator. It’s a admitted weak spot.
  2. Zombie Pods: Speaking of, always set ttlSecondsAfterFinished. Without it, completed and failed Jobs (the resource) and their Pods accumulate forever. It’s a tax on your API server.
  3. Resource Requests: This isn’t a Deployment. Your Pods aren’t being restarted constantly. You must set CPU/Memory requests and limits for a Job. A greedy Job that starts on a small node can easily cause havoc. Don’t be that person.
  4. Active Deadlines: Use spec.activeDeadlineSeconds if you absolutely must cap how long a Pod can run. This is a brilliant kill switch for jobs that might hang indefinitely. It’s a much cleaner solution than hoping your code has its own timeout logic.