Right, let’s talk storage. Because your fancy pods are ephemeral, and while that’s great for cattle, not pets, your precious application data needs to live somewhere more permanent than a container’s short, brutal life. You can’t just chmod 777 your way out of this one. In the old, barbaric days of EKS, you’d use the in-tree aws-ebs and aws-efs volume plugins that were baked into Kubernetes itself. Those are now deprecated and scheduled for a not-so-tearful goodbye. The future, and frankly the present, is the Container Storage Interface (CSI).

CSI is a standard that lets storage vendors write their own drivers and plug them right into Kubernetes, without having to wait for a full K8s release cycle. It’s cleaner, more feature-rich, and it’s how AWS delivers its storage magic to your cluster. You need two drivers: one for the workhorse (EBS) and one for the sharpshooter (EFS).

The EBS CSI Driver: Your Pod’s Personal Hard Drive

Think of EBS as giving a single pod its own dedicated block storage device. It’s fast, it’s reliable, and it’s locked to a single Availability Zone (AZ). This last point is not a suggestion, it’s the law. This is the most important thing to remember: An EBS volume can only be attached to an EC2 instance in the same AZ. If you try to schedule a pod that uses an EBS volume in a different AZ, it will sit there staring at you, Pending, until you fix your topology or it gets deleted out of sheer frustration.

The driver itself needs IAM permissions to make API calls on your behalf (to create volumes, attach them, etc.). You’d think this would be a simple IAM role, but because the driver runs on your nodes, we use IAM Roles for Service Accounts (IRSA). This is the secure, right way to do it.

First, create an IAM OIDC provider for your cluster (if you haven’t already) and then an IAM policy with the necessary permissions. Here’s the policy that AWS provides; it’s a kitchen-sink policy, but it works:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AttachVolume",
                "ec2:CreateSnapshot",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:DeleteSnapshot",
                "ec2:DeleteTags",
                "ec2:DeleteVolume",
                "ec2:DescribeInstances",
                "ec2:DescribeSnapshots",
                "ec2:DescribeTags",
                "ec2:DescribeVolumes",
                "ec2:DetachVolume"
            ],
            "Resource": "*"
        }
    ]
}

Then, create a ServiceAccount for the driver with the annotated role. Finally, deploy the driver. The easiest way is via the AWS-managed Helm chart:

helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver
helm upgrade --install aws-ebs-csi-driver \
  --namespace kube-system \
  --set controller.serviceAccount.create=false \
  --set controller.serviceAccount.name=ebs-csi-controller-sa \
  aws-ebs-csi-driver/aws-ebs-csi-driver

Now, for a StorageClass. The default one they provide is fine, but let’s make our own that’s explicitly ext4 and sets a delete policy. This is a best practice.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
fsType: ext4
parameters:
  type: gp3
reclaimPolicy: Delete

Note volumeBindingMode: WaitForFirstConsumer. This is crucial. It tells the EBS provisioner to wait until a pod is scheduled before creating the volume. This ensures the volume is created in the correct AZ—the AZ where the pod lands. Without this, it might create the volume in a random AZ and then your pod can’t use it. Rookie mistake.

The EFS CSI Driver: The Network File Share for Everyone

EFS is the opposite of EBS. It’s a managed NFS share that can be accessed by pods across all your AZs simultaneously. It’s for when you need many pods to read and write the same data. Think shared configs, asset directories, or that one monolithic application that still dumps everything into a single wp-content/uploads folder.

The setup is similar but simpler in one key way: no IRSA. Since EFS uses standard NFS protocols, the driver pods don’t need IAM permissions to create the file system itself. You pre-provision the EFS filesystem and then the driver just helps mount it. You do need to configure security groups to allow NFS traffic (port 2049) from your nodes to the EFS mount targets.

Deploying the driver is also a Helm affair:

helm repo add aws-efs-csi-driver https://kubernetes-sigs.github.io/aws-efs-csi-driver
helm upgrade --install aws-efs-csi-driver \
  --namespace kube-system \
  aws-efs-csi-driver/aws-efs-csi-driver

Your StorageClass needs to know your pre-created EFS filesystem ID.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-12345678
  directoryPerms: "700"
  gid: "1000"
  uid: "1000"

The efs-ap (Access Point) provisioning mode is the modern way. It creates EFS Access Points for each PV, which are fantastic for enforcing a specific POSIX user, group, and directory on the shared filesystem, giving you some semblance of multi-tenant security on a shared drive.

Common Pitfalls and The Gotcha Hall of Fame

  1. The AZ Blunder: I said it already, but it’s the #1 support ticket. EBS volume and pod must be in the same AZ. WaitForFirstConsumer is your shield against this.
  2. IRSA Misconfiguration: For EBS, if your controller pods are crashing with permission errors, you botched the IRSA setup. Double-check the IAM role’s trust policy and the service account annotation.
  3. EFS Security Groups: This is the EFS equivalent of the AZ blunder. Your node security groups must allow inbound traffic from the EFS mount target’s security group on NFS port 2049. Not outbound. It’s a stateful connection. Get it wrong and your mount will hang forever.
  4. Performance: EBS is low-latency and high-IOPS. EFS is… not. It’s a network filesystem. It has its place, but don’t try to run your database on it unless you enjoy pain. Use it for what it’s good at: shared, sequential reads and writes.

So there you have it. EBS for your single-pod, high-performance needs. EFS for your multi-AZ, shared access needs. Use the right tool for the job, set up the permissions correctly, and for the love of all that is holy, mind your AZs.