14.7 Visualizing and Testing Network Policies
Right, so you’ve written a NetworkPolicy. You’ve stared at the YAML, you’ve run kubectl apply -f, and it returned without an error. Fantastic. Now for the multi-million dollar question: is it actually doing what you think it’s doing?
This is where most people’s eyes glaze over and they just hope for the best. Don’t be that person. Hope is not a strategy; it’s a prelude to a 3 AM page. Let’s get tactical and verify our work.
The “Show Me” Commands
First, you need to see what you’ve actually deployed. kubectl get is your friend, but it’s a bit of a minimalist.
kubectl get networkpolicy -o wide
This gives you the basics: the policy’s name, its pod selector, and its age. Useful, but about as informative as a road sign in a language you don’t speak.
For the real details, you need to ask Kubernetes to describe it. This is where you see the full spec.
kubectl describe networkpolicy my-restrictive-policy
This output is your single source of truth. Scrutinize the PodSelector field. This is the most common point of failure. Does it match the pods you think it’s matching? A typo here means your policy is either governing the wrong pods or, more likely, governing nothing at all. I’ve seen an entire policy rendered useless because someone used app: nginx instead of app: nginx-server. Kubernetes is brutally literal. It doesn’t guess your intentions.
Testing with a Kitchen Sink Pod
Theory is great, but networking is about packets moving (or not moving). You need a throwaway pod to act as your probe. My go-to is a simple busybox or curlimages/curl pod. We’re going to use kubectl run to create a temporary pod and then exec into it to run tests.
Let’s say you have a policy that should only allow traffic to a redis database from pods with the label role: api. Here’s how you test the allow case:
# Create a test pod that SHOULD be allowed
kubectl run allowed-tester --image=curlimages/curl --labels="role=api" --rm -it -- /bin/sh
# Now, from inside the allowed-tester shell:
curl -v telnet://redis-service:6379
If you get a connection, great. Now, let’s test the deny case, which is arguably more important.
# Create a test pod that should be BLOCKED
kubectl run denied-tester --image=curlimages/curl --labels="role=some-other-thing" --rm -it -- /bin/sh
# Now, from inside the denied-tester shell:
curl -v --connect-timeout 5 telnet://redis-service:6379
If your policy is working, this command should hang for a bit and then time out. This is the beautiful sound of your policy working correctly. No packet has been harmed; it was simply, and politely, shown the door.
The Cruel Reality of Defaults and Namespaces
Here’s a classic “gotcha” that has tripped up more engineers than I can count. Network Policies are namespaced. This is obvious when you say it, but it’s easily forgotten in the heat of battle. A policy in the web namespace does absolutely nothing for pods in the database namespace.
Furthermore, remember the default behavior: if no policies exist in a namespace, all traffic is allowed. The moment you create any policy in a namespace, it switches to a default-deny model for all traffic not explicitly whitelisted by your policies. This is a crucial design choice. It means your first policy might accidentally break everything else in the namespace that relied on that default allow. It’s the Kubernetes equivalent of putting up a single stop sign and suddenly requiring every other car to have an explicit written permission to drive. Always apply policies incrementally.
Visualizing the Madness
Staring at YAML is a terrible way to understand a mesh of connections. This is where tools like cilium-cli (if you’re using Cilium CNI) absolutely shine.
cilium connectivity test
This command does something magical: it deploys a whole suite of test pods and systematically checks connectivity between them based on the policies you have installed. It gives you a full report, telling you exactly which tests passed and which failed, and why. It’s like having a dedicated QA engineer for your network security. If you have the option, use it. It’s a game-changer.
For other CNIs, you might be stuck with the older, manual netcat/curl method, which is why I detailed it above. It’s the pickaxe to Cilium’s excavator—less efficient, but it’ll still get the job done.
The bottom line is this: never assume your network policy works. Verify it. Test the happy path, but more importantly, test the unhappy path. Prove that the bad traffic is being dropped. Your future self, who is not debugging a production outage at midnight, will thank you.