Alright, let’s talk networking. This is where the rubber meets the road in EKS, and frankly, where most people get their knickers in a twist. You’ve got your shiny new cluster, but until your pods can actually talk to each other and the outside world, it’s just a very expensive, very abstract art installation. The two big players here are the VPC CNI plugin and Security Groups for Pods. One provides the fundamental plumbing, the other gives you a much-needed security scalpel. Let’s get our hands dirty.

The VPC CNI: Your Pod’s Network Identity Crisis

When you launch a pod on an EC2 node in your EKS cluster, it needs an IP address. The VPC Container Network Interface (CNI) plugin is Amazon’s way of solving this. Here’s the crucial part: it does NOT use an overlay network. Forget flannel, Calico’s IPIP modes, or Weave. The VPC CNI assigns a real, honest-to-goodness VPC IP address from the subnet your node is in directly to your pod.

Why should you care? Because it means your pods are first-class citizens on your VPC network. They can talk to your RDS instances, your Elasticache clusters, and your on-premises servers (via Direct Connect/VPN) natively, without any fancy routing tricks. The security groups attached to the worker node’s ENI are, by default, the ones that govern the traffic for all pods on that node. This is simple but also incredibly blunt.

The main gotcha? IP address exhaustion. Each node has a finite number of IPs it can hand out, defined by its ENI limits and the number of IPs per ENI (which varies by instance type). You can quickly find yourself with plenty of CPU/RAM free on a node but unable to schedule new pods because it’s out of IPs. You must plan your subnets and instance types with this in mind.

Here’s how you’d install the VPC CNI if it wasn’t already there (it usually is by default):

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/master/config/v1.7/aws-k8s-cni.yaml

And to see how many IPs a t3.large node (which has 3 ENIs and 12 IPs per ENI) can actually offer pods:

# Calculate total IPs per node: (Number of ENIs - 1) * (IPs per ENI) + 2
# The '-1' is because the primary ENI for the host itself uses one IP.
# The '+2' is for the host's primary IP and the network/broadcast addresses? Wait, no, that's not right.

# Let's use the AWS calculator instead. For a t3.large:
# Max ENIs: 3
# IPv4 Addresses per ENI: 12
# Total IPs for Pods: (3 ENIs * 12 IPs) - 1 (for the primary ENI's primary IP) = 35
# But wait, the primary ENI gets its primary IP and then secondary IPs. It's messy.

# The truth is, you should just use the AWS CLI. This is why the joke exists.
aws ec2 describe-instance-types --instance-types t3.large --query "InstanceTypes[].NetworkInfo.{MaxENIs: MaxNetworkInterfaces, IPv4perENI: Ipv4AddressesPerInterface}"

# Output: [ { "MaxENIs": 3, "IPv4perENI": 12 } ]
# So a node can have 3 ENIs, each with 12 IPs. The primary ENI uses one IP for the host, so the total for pods is (3 * 12) - 1 = 35.
# See? Absurd. Just use the `max-pods` value in your bootstrap script and let AWS do the math.

Security Groups for Pods: Finally, a Proper Firewall

The default VPC CNI behavior is a security nightmare. All pods on a node share the node’s security groups. Your frontend web pod and your backend payment processor pod have the same network permissions. Yikes.

Security Groups for Pods (SG4Pods) fixes this by allowing you to attach AWS Security Groups directly to a Kubernetes Pod, not the node. This is a game-changer. Now your payment processor pod can have a SG that only allows outbound traffic to PCI-compliant services, while your frontend pod has a SG that allows inbound HTTP/S.

The magic behind this is the AWS VPC Resource Controller, which manages a special “trunk” interface on your node. Your pods then get their own “branch” interfaces from this trunk, and you can attach SGs directly to those branch interfaces.

To use it, first enable it on your cluster. The easiest way is to use the eksctl tool:

# cluster-config.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: my-cluster
  region: us-west-2
vpc:
  clusterEndpoints:
    publicAccess: true
    privateAccess: true
managedNodeGroups:
  - name: ng-1
    instanceType: m5.large
    desiredCapacity: 2
    # This is the crucial part
    securityGroups:
      attachIDs: ['sg-0a1b2c3d4e5f67890'] # Your node SG
    labels:
      # This label is often required for the k8s controller to manage nodes
      vpc.amazonaws.com/trunk-eni: "true"

Then, you define a Kubernetes resource that maps to an AWS Security Group. This is done via a SecurityGroupPolicy custom resource.

# payment-processor-sgp.yaml
apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: payment-processor-sg
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: payment-processor
  securityGroups:
    groupIds:
      - sg-paymentdbaccess # Replace with your actual SG ID
      - sg-outboundonly

Now, any pod in the production namespace with the label app: payment-processor will automatically have those two security groups attached. The VPC CNI and Resource Controller handle the rest.

The Rough Edges and Best Practices

First, the rough edges. SG4Pods is brilliant, but it’s not perfect. The time between pod creation and the attachment of the security group can create a brief window where the pod might not have the correct network rules. For most applications, this is negligible, but for ultra-paranoid, compliance-heavy workloads, it’s something to note.

Best practice? Use it everywhere. Seriously. The granular security is worth the minimal operational overhead. Always use separate security groups for your nodes and your pods. The node SG should be locked down to only allow SSH, node-exporter, and Kubernetes API communication. All application-level rules should be on the pod security groups.

Also, remember that assigning a security group to a pod is a disruptive operation. The VPC CNI has to tear down the old branch ENI and create a new one. This will briefly interrupt the pod’s network connectivity. So don’t go changing SGs on a pod every five minutes.

The VPC CNI is the foundation, and SG4Pods is the elegant, secure house you build on top of it. Master both, and your EKS cluster will be not only functional but genuinely secure. Ignore them, and you’re just hoping nobody notices your packets are all wearing the same ill-fitting security outfit.