Right, let’s talk about keeping your secrets secret. Because by default, Kubernetes doesn’t. I know, I was disappointed too. When you create a Secret, the API server slaps it into etcd, the cluster’s brain, and just… leaves it there. In plain text. It’s the digital equivalent of writing your database password on a post-it note and sticking it to the monitor. We can, and must, do better. This involves two key concepts: encryption at rest (so the post-it note is in a locked drawer) and RBAC (so only certain people have the key to that drawer).

The Default (and Terrible) Situation

Out of the box, etcd holds all your cluster data, including every Secret you’ve ever created, in plain text. Anyone with direct access to the etcd data store (a backup, a disk snapshot, a mischievous admin) can see everything. Let’s prove it to ourselves because seeing is believing.

Create a simple secret:

kubectl create secret generic my-secret --from-literal=password='S0perS3kret!'

Now, if you have etcdctl and the right certs (ask your cluster admin nicely, or do this on a local kind cluster), you can peer directly into the abyss:

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/path/to/ca.crt \
--cert=/path/to/etcd-client.crt \
--key=/path/to/etcd-client.key \
get /registry/secrets/default/my-secret | hexdump -C

You’ll see a bunch of binary data, but nestled right in there, clear as day, you’ll find S0perS3kret!. This is not ideal. This is why we enable encryption at rest.

Enabling Encryption at Rest

This is a cluster-wide configuration that you set on the kube-apiserver. You provide it with a configuration file that defines a list of encryption providers and their corresponding keys. The API server will use the first provider in the list to encrypt secrets on their way into etcd and try each one in order to decrypt them on the way out.

Here’s a typical encryption-config.yaml using the aescbc provider, which is a good balance of performance and security:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key-1
          secret: <a-base64-encoded-32-byte-key> # You must generate this!
    - identity: {} # This fallback allows reading unencrypted secrets

The critical part is that secret value. You must generate a proper 32-byte random key and base64 encode it. Do not just type “password” in there. Use the tools your OS gives you:

# This gives you the raw bytes. Now base64 encode it.
head -c 32 /dev/urandom | base64

You then start your kube-apiserver with the --encryption-provider-config flag pointing to this file. After restarting the API server, any new secrets you create will be encrypted. The old ones? Still sitting there in plain text. You have to explicitly rewrite them:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

This command fetches all secrets and replaces them, forcing the API server to write their encrypted versions back to etcd.

The Crucial Role of RBAC

Encryption at rest protects your data from someone stealing the etcd disks. RBAC (Role-Based Access Control) protects your data from someone who has access to the cluster right now. If you don’t lock down access to Secrets, any developer with basic pod permissions can just kubectl get secret my-database-creds -o yaml and instantly decode the base64-encoded values within. Encryption at rest is useless against this.

You must be ruthlessly minimalist with Secret access. The principle of least privilege isn’t a suggestion here; it’s the law. This means:

  1. Don’t use wildcards. A RoleBinding granting get and list on secrets in a namespace is a catastrophe waiting to happen.
  2. Use specific resourceNames. If an app only needs one secret, only grant it access to that one secret by name.

Here’s an example of a good, restrictive Role and RoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: app-secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["app-database-creds"] # Specifically THIS secret
  verbs: ["get"] # Specifically ONLY this verb, not 'list' or 'watch'

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: production
  name: read-app-creds
subjects:
- kind: ServiceAccount
  name: my-app-service-account # The specific app that needs it
  namespace: production
roleRef:
  kind: Role
  name: app-secret-reader
  apiGroup: rbac.authorization.k8s.io

This policy is airtight. The service account can get only the secret named app-database-creds. It can’t list all the other secrets in the namespace to go on a fishing expedition.

The Pitfalls and The Gotchas

First, key management. That aescbc key we generated? You are now responsible for it. Lose it, and you lose the ability to decrypt every secret encrypted with it. Rotate it? You’ll need to add a new key to the config (moving it to the top of the keys list), restart the API server, and rewrite all secrets again. It’s a manual and potentially disruptive process. This is why many organizations opt for external KMS solutions (like AWS KMS, Azure Key Vault, or GCP KMS) as encryption providers—they handle the key rotation and management for you.

Second, encryption is not magic. The secret is encrypted in etcd, but the moment it’s mounted into a pod as a volume or environment variable, it’s decrypted and in plain text on the node’s filesystem. RBAC protects the API access, but you also need to secure your nodes. If an attacker can exec into your pod or read the node’s disk, the jig is up. This is why controlling host-level access and using pod security standards are non-negotiable parts of the whole secret security picture.