Alright, let’s get our hands dirty. You’ve defined your fancy Custom Resource (CR) with a Custom Resource Definition (CRD). It’s sitting there in your cluster, a beautiful, empty JSON-shaped promise. But right now, it’s about as useful as a car without an engine. It looks like a car, you can describe the car you want, but turning the key does absolutely nothing.

That’s where the Operator Pattern comes in. It’s the engine. More precisely, it’s the controller that acts as the engine for your CR. The core idea is gloriously simple, and it’s the same pattern that makes all of Kubernetes work: reconcile desired state with observed state.

Think of it like a thermostat. You set a temperature (the desired state). The thermostat constantly checks the room’s actual temperature (the observed state). If the room is too cold, it kicks on the heat. Too hot? The A/C whirs to life. It keeps doing this in a loop until the desired and observed states match. A Kubernetes controller is just a very sophisticated, programmable thermostat for your custom resources.

The Reconciliation Loop: It’s Just a While True Loop

At its heart, every controller runs a reconciliation loop. It’s essentially this, but with a lot more Go and a lot more waiting on the Kubernetes API:

for {
    desiredState := getDesiredState() // From your Custom Resource
    currentState := getCurrentState() // From the actual cluster (Pods, Deployments, etc.)

    if currentState != desiredState {
        doSomethingToMakeThemMatch() // This is your business logic!
    }

    time.Sleep(10 * time.Second) // Be polite, don't hammer the API server.
}

The real magic, and complexity, lives inside that doSomethingToMakeThemMatch() function. This is where you write the code that says, “If my CronJob CR spec says ‘run every hour,’ then I need to ensure a Kubernetes CronJob resource exists with that schedule. If it doesn’t, create it. If it does but the schedule is wrong, update it. If my CR is deleted, clean up the underlying CronJob.”

Anatomy of a Controller

A controller isn’t a magical black box. It’s made of a few key parts that work together:

  1. Informer/SharedIndexInformer: This is your controller’s window into the world. It watches for changes (CREATE, UPDATE, DELETE) to specific resource types, like your precious CRs and any other resources it manages (e.g., Pods, Services). Its genius is that it maintains a local cache of these objects and only gets deltas (changes) from the API server, preventing you from doing expensive kubectl get operations every few seconds.
  2. Workqueue: When the Informer sees a change, it doesn’t process it immediately. Instead, it shoves a “key” (like <namespace>/<name> of the changed object) into a workqueue. This is crucial for handling bursts of events and for retries. If your reconciliation logic fails spectacularly, you can just put the key back in the queue to try again later.
  3. Reconciler: This is the star of the show. It’s the function that pops a key off the workqueue, uses the Informer’s cache to look up the actual object, and then runs your business logic to make the world look the way the CR spec says it should.

A Concrete Example: The Event Hander

Let’s look at some pseudo-Go code to see how the Informer and Workqueue connect. This is what you’d set up in a framework like controller-runtime.

// This function sets up the whole watch party.
func (r *MyAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        // Watch for changes to our primary custom resource, 'MyApp'
        For(&myapiv1alpha1.MyApp{}).
        // And also watch for changes to Pods owned by our 'MyApp' resources.
        // This is critical: if someone deletes a Pod we created, we need to know!
        Owns(&corev1.Pod{}).
        Complete(r)
}

// This is the Reconcile function that gets called for each event.
func (r *MyAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := log.FromContext(ctx)

    // 1. Fetch the MyApp instance that triggered this event.
    myApp := &myapiv1alpha1.MyApp{}
    if err := r.Get(ctx, req.NamespacedName, myApp); err != nil {
        // This usually means the resource was deleted, so we need to handle that.
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // 2. Check if the object is being deleted.
    if !myApp.GetDeletionTimestamp().IsZero() {
        // Our CR is being deleted! Do cleanup here.
        return r.cleanupResources(ctx, myApp)
    }

    // 3. YOUR RECONCILIATION LOGIC GOES HERE.
    // This is the "doSomethingToMakeThemMatch" function.

    // Example: Ensure a Deployment exists for this MyApp.
    found := &appsv1.Deployment{}
    err := r.Get(ctx, types.NamespacedName{Name: myApp.Name, Namespace: myApp.Namespace}, found)
    if err != nil && errors.IsNotFound(err) {
        // It doesn't exist? Create it.
        dep := r.deploymentForMyApp(myApp) // Helper function that builds the Deployment object.
        log.Info("Creating a new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
        err = r.Create(ctx, dep)
        if err != nil {
            return ctrl.Result{}, err
        }
        // Creation was successful! Let's requeue to check on it later.
        return ctrl.Result{Requeue: true}, nil
    } else if err != nil {
        // Some other error happened while trying to fetch the Deployment.
        return ctrl.Result{}, err
    }

    // 4. Compare the found Deployment's spec with what we *want* it to be.
    // This is a deep, semantic comparison, not a naive reflect.DeepEqual.
    if !r.compareDeployments(found, desired) {
        log.Info("Updating Deployment", "Deployment.Namespace", found.Namespace, "Deployment.Name", found.Name)
        updatedDeployment := found.DeepCopy()
        updatedDeployment.Spec = desiredDeployment.Spec
        if err := r.Update(ctx, updatedDeployment); err != nil {
            return ctrl.Result{}, err
        }
    }

    // 5. Update the status of the MyApp CR to reflect the current world state.
    myApp.Status.Ready = true
    if err := r.Status().Update(ctx, myApp); err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil
}

Pitfalls and Battle-Scarred Wisdom

  1. Idempotency is Non-Negotiable: Your Reconcile function must be idempotent. It might be called five times in a row with the exact same input. The outcome must always be the same. This is why you use Create and Update calls that are declarative. Never, ever code a function that says “increment this value” inside a reconciler. It will end in tears.
  2. Own Your Children: Use Kubernetes’ ownerReferences field. When your controller creates a Pod or Deployment, set the CR as its owner. This is Kubernetes-native garbage collection. When the CR is deleted, the Kubernetes API server will automatically clean up all the owned objects. It’s free real estate. Don’t try to manage deletion yourself.
  3. Status Matters: The .spec is for the user’s desired state. The .status is for your observed state. Use it! Update it every reconciliation cycle with the current state of the world. Is the underlying deployment ready? How many pods are running? This is what shows up in kubectl get myapp. A CR without an updated status is a black box and a nightmare to debug.
  4. Requeue, But Be Polite: Sometimes you need to re-check things. Maybe you’re waiting for a Pod to become ready. Return ctrl.Result{RequeueAfter: 10 * time.Second} to check back in 10 seconds. Don’t write a busy-wait loop; that will get your controller evicted for consuming too many resources.
  5. Watch All The Things: Your controller must watch not only your CR but also every resource it creates. If you create a Deployment, you must watch Deployments. If you create a Service, watch Services. If you don’t, and someone goes and deletes that Service manually, your controller will be blissfully unaware until another change to the CR forces a reconciliation. This is what the Owns() method in the example above does for you.