Alright, let’s talk about secrets. Not the salacious kind, but the ones that make your cluster go. API tokens, database passwords, TLS certificates—the digital crown jewels. The problem is, by default, Kubernetes doesn’t treat them with the gravitas they deserve. A Secret is just a base64-encoded string sitting in etcd. It’s like writing your password on a post-it note and then just turning the note upside-down. We’re going to do a lot better than that.

The Base64 “Obfuscation” Farce

First, let’s demystify the most common misconception. People see kubectl get secret my-secret -o yaml and see a garbled mess under the data key and think, “Ah, encrypted!” No. It’s just base64-encoded. This is not encryption. This is the equivalent of putting a cheap lock on a diary—it keeps your little sister out, but not anyone with five minutes and an internet connection to find a base64 decoder.

Let’s prove it. You create a secret:

apiVersion: v1
kind: Secret
metadata:
  name: test-secret
type: Opaque
data:
  password: c3VwZXJzZWNyZXRwYXNzd29yZCAgICA= # "supersecretpassword" base64'd

Any clown with kubectl access can decode it instantly:

kubectl get secret test-secret -o jsonpath='{.data.password}' | base64 --decode
# Output: supersecretpassword

The stringData field is a bit nicer for creating secrets, as it handles the encoding for you, but the result in etcd is the same. The takeaway? Never rely on base64 as a security measure. Its only job is to make binary data safe for YAML/JSON. Your actual security layers are Encryption at Rest and tight RBAC.

Encrypting Secrets at Rest

This is the first, non-negotiable step. You must ensure that the etcd database, where all your secrets are stored, is encrypted. Otherwise, anyone with access to the etcd storage backend can just read all your secrets in plaintext. It’s a nightmare.

You enable this by providing an EncryptionConfiguration file to the Kubernetes API server. This file defines a hierarchy of encryption providers (like a secret box, or better yet, using your cloud provider’s KMS). Here’s a basic example using a locally-generated key:

# encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <your-32-byte-base64-encoded-key> # Generate this properly!
      - identity: {} # This allows unencrypted reads for migration. Remove it later.

You’d then start your kube-apiserver with --encryption-provider-config=/path/to/encryption-config.yaml. The moment you do this, all new secrets will be encrypted before being written to etcd. Old secrets are still sitting there in plaintext. You have to manually rewrite them to get the benefits:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

This command reads all secrets and writes them back, forcing the API server to encrypt them on the new write. Warning: This is a disruptive operation. Do it in a maintenance window. And for the love of all that is holy, do not use a local key in production. Use a real KMS like AWS KMS, Azure Key Vault, or GCP Cloud KMS in your EncryptionConfiguration. The configuration is more complex, but it means your keys are managed by a real service, not a file on a disk.

The Principle of Least Privilege with RBAC

Encryption at rest protects against someone stealing your etcd backups. RBAC protects against someone inside your cluster doing something stupid (or malicious). Your default posture should be absolute zero trust. A pod, and by extension the service account it uses, should only have permission to access the secrets it explicitly needs. No more.

This means you’ll be writing a lot of RoleBindings and ClusterRoleBindings. Here’s a classic pitfall: giving a pod get and list access on secrets in its namespace. Bad. Now any vulnerability in your app could lead to it dumping every secret in the namespace. Instead, create a specific Role that grants get access to only one specific secret.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: my-app
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["app-db-password"] # This is the critical part
  verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: my-app
  name: read-secrets
subjects:
- kind: ServiceAccount
  name: my-app-service-account
  namespace: my-app
roleRef:
  kind: Role
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

Now, the pod using my-app-service-account can only kubectl get secret app-db-password. It can’t list other secrets. This drastically contains the blast radius of a compromise.

Mounting Secrets the Right Way

So your pod has permission to get the secret. How does it actually use it? The worst way is to environment variables. envFrom might seem convenient, but it’s a terrible idea. Why? Because environment variables are visible in the process table, they get logged accidentally more often than you’d think, and they’re passed down to child processes. It’s like shouting your password across a crowded room.

The right way is to mount them as volumes. Kubernetes will handle this beautifully, creating a tmpfs (in-memory filesystem) volume for you, which is never written to disk.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      serviceAccountName: my-app-service-account
      containers:
      - name: app
        image: my-app:latest
        volumeMounts:
        - name: secret-volume
          mountPath: "/etc/secrets"
          readOnly: true
        # ... and NEVER do this:
        # env:
        # - name: DB_PASSWORD
        #   valueFrom:
        #     secretKeyRef:
        #       name: app-db-password
        #       key: password
      volumes:
      - name: secret-volume
        secret:
          secretName: app-db-password

Now your application can just read the file at /etc/secrets/password. It’s cleaner, safer, and more in line with how Unix systems have handled secrets for decades.

Going Beyond Built-Ins: External Secrets Operators

Let’s be honest: managing thousands of secrets natively in Kubernetes can become a pain. The built-in secret store is fairly rudimentary. This is where the ecosystem comes to the rescue. Tools like the External Secrets Operator (ESO) are brilliant.

The pattern is simple: you keep your canonical secrets in a real secret manager like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault. Then, you define an ExternalSecret manifest in Kubernetes. The ESO controller running in your cluster has permission to talk to your external store. It reads the manifest, fetches the actual secret from the external store, and creates a standard Kubernetes Secret object for you.

This gives you the best of both worlds: the robust management and auditing of a dedicated secrets manager, with the standard Kubernetes API that your applications expect. You version your ExternalSecret manifests in Git, and your actual secret values stay safely in Vault. It’s a clear win.