Right, so you’ve got ConfigMaps and Secrets down. You’re manually kubectl create secret generic-ing your life away. It works, but it feels a bit… medieval. You’re duplicating secrets into Kubernetes, which is a fantastic way to have them rot in two different places instead of one. And let’s be honest, you’re probably not rotating them as often as you should. Nobody is.

The real grown-ups keep their secrets in a proper vault—Hashicorp Vault, AWS Secrets Manager, Google Cloud Secret Manager, Azure Key Vault, you name it. These tools are built for this job: tight access controls, auditing, rotation, the whole nine yards. The question is, how do you bridge that world with the frantic, YAML-obsessed world of Kubernetes?

You could bake custom logic into your apps to fetch secrets directly from Vault. But now you’ve just traded one problem for another. Now every app needs its own Vault client and authentication logic. Yuck. This is where the External Secrets Operator (ESO) swoops in like a superhero, if that superhero’s power was writing impeccably detailed YAML manifests.

ESO is a Kubernetes operator—a fancy term for a controller that watches for custom resources you define. In this case, you define a SecretStore (or a ClusterSecretStore) which tells ESO how to connect to your external secret manager (e.g., your Vault server URL and auth details). Then, you define an ExternalSecret which says what secrets to fetch from the external manager and how to shape them into a regular Kubernetes Secret.

The magic is that ESO does the heavy lifting of authentication, fetching, and—crucially—auto-refreshing. It reconciles your desired state (the ExternalSecret) with the actual state (the Kubernetes Secret) continuously.

Connecting to Your Secret Store: The SecretStore

First, you gotta tell ESO how to talk to your vault. This is the most sensitive part because you’re storing access credentials. For Vault, using Kubernetes Service Account authentication is the way to go. It’s secure and doesn’t require you to hardcode a token.

# This is a ClusterSecretStore, usable across namespaces.
# Use a regular SecretStore if you want it namespace-scoped.
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      # Address of your Vault server. Don't screw this up.
      server: "https://vault.example.com:8200"
      # Path for Kubernetes auth, usually /v1/auth/kubernetes/login
      path: "kubernetes"
      # We're using the Kubernetes auth method
      auth:
        kubernetes:
          # This is the service account token path mounted in the ESO pod.
          # You almost never need to change this.
          serviceAccountRef:
            name: "external-secrets"
          # Mount path for the Kubernetes service account token
          mountPath: "/var/run/secrets/kubernetes.io/serviceaccount"
          # The role we configured in Vault that this ServiceAccount can use
          role: "external-secrets-role"

Why this is brilliant: The permissions for ESO to read secrets are defined and locked down in Vault itself, following the principle of least privilege. ESO just authenticates with its service account token; Vault decides what secrets that token is allowed to access.

Defining What to Fetch: The ExternalSecret

Now for the fun part. This is where you map your external secrets to a nice, boring, standard Kubernetes Secret. ESO will create and keep this secret updated.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  # Which SecretStore to use (refers to the one we made above)
  refreshInterval: "1h" # How often to check for updates. '1h' is sensible.
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    # Optional: you can tweak the name of the final Kubernetes Secret
    name: prod-database-creds
    # Optional: sometimes you need to put it in a different namespace
    # creationPolicy: Owner
  data:
  - secretKey: username # The key in the final K8s Secret
    remoteRef:
      key: secret/data/prod/app-db # The path in Vault
      property: data.username      # The specific property within the Vault secret
  - secretKey: password
    remoteRef:
      key: secret/data/prod/app-db
      property: data.password

Apply this, and watch the magic happen. ESO will create a Secret named prod-database-creds with username and password keys, populated with the latest values from Vault. Change the secret in Vault, and within an hour (or your specified interval), ESO will update the Kubernetes Secret. Pods that reference this secret will eventually get the update, though how they handle that depends on the app (a topic for another day).

The Gotchas: Because Nothing is Perfect

  1. Permission Denied: This is the big one. 99% of your problems will be here. The identity ESO uses (its service account) must have the correct permissions in your external secret manager. For Vault, this means the Kubernetes auth role must have the correct Vault policies attached. Test your policies with vault policy test before you drive yourself mad.
  2. Network Connectivity: Can the ESO pod actually reach your Vault server or AWS endpoint? Obvious, but it’ll get you every time. Check network policies and firewall rules.
  3. Refresh Interval: It’s not real-time. If you update a secret in Vault, there’s a delay before it’s in Kubernetes. Don’t panic. You can manually trigger a reconcile by deleting the ExternalSecret and recreating it, but that’s a bit hacky.
  4. Secret Format: External systems often return JSON. ESO is great at parsing it (using the property field), but you have to get the path right. secret/data/prod/app-db is for the V2 KV secret engine. If you’re on the old V1 engine, it’s just secret/prod/app-db. This tripped me up for a solid 30 minutes once, and I’m still annoyed about it.

The beauty of ESO is that it turns a complex, custom integration into a declarative resource you can manage with GitOps. You can see exactly what secrets your applications require right there in the cluster, without the actual secret data being in your git repo. It’s the best of both worlds: central management with Kubernetes-native consumption. Now go forth and stop hardcoding things.