Right, so you’ve got your API server up and running, and you’re starting to feel like you understand the whole request flow. Then you hear about this thing called an “Admission Controller” and your brain immediately jumps to bouncers at a club. It’s not a bad analogy, honestly. But the real story is a bit more nuanced and, frankly, more interesting. Think of it less as a single bouncer and more like a multi-stage security and compliance checkpoint for every single request that wants to create, update, or delete something in your cluster.

Before any admission controller even gets to see the request, two critical things happen: authentication and authorization. I know, they sound similar, but the distinction is everything. Authentication (AuthN) is about who you are. The API server figures out if the request is coming from a valid user, a service account, or maybe a pod. It’s checking your ID. Authorization (AuthZ) is about what you’re allowed to do. Now that it knows you’re “jane-doe,” does “jane-doe” have permission to delete a namespace? It’s checking your access card.

Only after you’ve passed both of those checks does your request enter the admission control phase. This is where the real “should this happen?” logic lives. Admission control is split into two phases that run sequentially: mutating admission first, then validating admission. This order is crucial. You wouldn’t want to validate an object before something else has had a chance to mutate it into its correct form, right? That would be like doing a quality check on a car before it’s even been painted.

The Mutating Stage: The Helpful Editors

This is where admission controllers can modify the incoming object. They’re your helpful, if sometimes overly opinionated, editors. A classic example is injecting a sidecar container into your Pod spec if it has a certain label. You send in a Pod, and the mutating webhook says, “Ah, I see you’re part of the ‘monitoring’ group, you get a Prometheus sidecar for free!” and patches your request before it’s persisted to etcd.

Here’s a simplified look at what a mutating webhook configuration might target. Notice how it uses operations: ["CREATE"] and object for a new Pod.

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: "inject-sidecar.example.com"
webhooks:
- name: "inject-sidecar.example.com"
  rules:
  - operations: ["CREATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
    scope: "Namespaced"
  clientConfig:
    service:
      namespace: "webhooks"
      name: "sidecar-injector"
      path: "/inject-pod"
  admissionReviewVersions: ["v1"]
  sideEffects: "None"
  failurePolicy: "Fail"

The failurePolicy: Fail here is a big deal. If our webhook service is down, this policy tells the API server to outright reject the request. For a critical mutating webhook, that’s often the right call. You don’t want pods running without their security sidecar.

The Validating Stage: The Strict Librarians

After all the mutators have had their fun, the validators step in. Their job is to say “yes” or “no,” with no modifications allowed. They are the strict librarians of the API server. This is where you enforce your organization’s policies: “All containers must have resource limits,” “No containers can run as root,” “Ingress hostnames must follow a naming convention.”

A validating webhook configuration looks similar but is, obviously, for validation.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: "require-limits.example.com"
webhooks:
- name: "require-limits.example.com"
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
    scope: "Namespaced"
  clientConfig:
    service:
      namespace: "webhooks"
      name: "limit-validator"
      path: "/validate-pod"
  admissionReviewVersions: ["v1"]
  sideEffects: "None"
  failurePolicy: "Fail"

Why This Two-Phase Design is Brilliant (and Annoying)

The designers got this right. The separation of concerns is clean. Mutators prepare the object, validators judge the final product. The annoying part? You can’t mutate in the validating phase. If you need to do both, you must write two separate webhooks and ensure your mutator runs first (often controlled by the webhookTimeoutSeconds and a nifty trick called reinvocationPolicy on mutating webhooks, but that’s a story for another day). It feels a bit clunky, but it prevents some truly horrifying circular logic.

The Pitfalls They Don’t Tell You About

  1. Failure Mode Hell: Your choice of failurePolicy is a critical design decision. Fail means if your webhook is down, the request is rejected. Ignore means if your webhook is down, the request is allowed. Neither is perfect. Use Fail for must-have security policies and Ignore for nice-to-have policies, but be prepared for the operational burden of running a highly available webhook service. This is why you see projects like Kyverno and OPA Gatekeeper—they handle this complexity for you.
  2. Performance is Your Problem: The API server will wait for your webhook to respond. If your admission logic is slow, you are slowing down everyone’s kubectl apply commands. You must make your webhook logic incredibly efficient. No blocking network calls in there. Seriously.
  3. The Object You Get Isn’t Always The Object You Save: You’re working with the incoming object. For UPDATE operations, you need to explicitly ask for the old object in your webhook configuration if you want to do compare-and-contrast logic, and even then, you’re working with the proposed object, not the final stored one. This trips everyone up.

The admission chain is the beating heart of policy enforcement in Kubernetes. It’s what transforms a dumb API server into a smart, self-governing system. But with great power comes great responsibility—and a non-trivial amount of YAML and operational overhead. Get it right, and you sleep soundly. Get it wrong, and you’re the one waking up at 3 a.m. because your webhook panicked and nobody can deploy anything. No pressure.