Right, so we’ve got labels and selectors for the stuff that matters to us—finding and grouping our Pods. Annotations are the metadata we slap on there for the machinery. Think of them as the sticky notes you leave for the various robots and automated systems in Kubernetes. They don’t impact how the core system groups or identifies objects; instead, they’re instructions, comments, or configuration for external tools, operators, or even your own automation.

The key difference between a label and an annotation is this: labels are for selecting. If you can imagine writing a selector: block in a Service to find a set of Pods, that’s a label. Annotations are for storing arbitrary data. They’re a key-value store bolted onto every API object, perfect for things that are too big, too structured, or simply not meant for identifying the object itself.

What Do We Even Put in Here?

Good question. The classic, almost cliché example is for your continuous deployment tool. Let’s say Jenkins just built a new image and is deploying a Pod. It might annotate the Pod with the exact git commit hash it built from. This is data you might want for debugging, but you’d never use it to select a Pod.

apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
  annotations:
    build.jenkins.io/commit: f4c3d2e1a5b6  # Not for selecting, just for info.
    build.jenkins.io/build-url: https://jenkins.example.com/job/my-app/42/
spec:
  containers:
  - name: app
    image: my-registry.com/my-app:f4c3d2e1a5b6

Other common uses include:

  • Load Balancer Configuration: Cloud providers often use annotations on Services to let you specify parameters their cloud controller doesn’t have first-class support for. Want a specific SSL policy or a backend health check timeout on your GCP Load Balancer? You’re probably annotating the Service.
  • Custom Sidecar Injection: Service meshes like Istio or Linkerd use annotations on Pods to control the behavior of their injected sidecar proxies. You can tell the injector, “Hey, don’t inject the proxy for this Pod,” or “Configure the proxy with this specific tracing sample rate.”
  • Description and Ownership: While kubectl describe will show you this, it’s a common practice to add description, owner, or support-contact annotations for human consumers.

The Practical Limits: Size and Syntax

Here’s where the designers’ “questionable choice” rears its head. The metadata API is a bit of a mess. Both labels and annotations have character limits, but for very different reasons.

Labels are meant to be used in queries, so they have strict limits: keys can have a prefix and a name, with a total length of 253 characters. Their values are capped at 63 characters. This feels arbitrary, but it’s to prevent etcd (Kubernetes’ backing database) from choking on massive selectors.

Annotations, however, are the wild west… with a catch. Their keys have the same 253-character limit as label keys. But their values? They can be up to 256 kilobytes each. That’s 262,144 characters. You could store a small novel in a single annotation value.

Why would you do this? Please don’t. But you could.

annotations:
  # This is a valid, if utterly ridiculous, key name.
  com.example.very-long-and-descriptive-key-name-with-version.v2.alpha1: "This is the value. It can be massive. We're talking entire chunks of XML or JSON configuration. This is both a blessing and a curse. A blessing because you can store complex, structured data without a separate system. A curse because you can store complex, structured data without a separate system. It's a foot-gun waiting to go off."

The real pitfall here isn’t the size, it’s the temptation to abuse it. If you find yourself storing a multi-kilobyte JSON blob in an annotation, you should probably stop and ask if a ConfigMap or a custom resource would be a better home for that data. Annotations are best for small, simple pieces of metadata.

Best Practices: Don’t Be That Person

  1. Prefix Your Annotations: If you’re writing a tool that uses annotations, use a domain prefix like cooltool.example.com/. This prevents conflicts with other tools. The Kubernetes system itself uses kubernetes.io/ and k8s.io/ prefixes.
  2. Treat Them as Ephemeral: Annotations can be changed freely. Unlike a Pod’s spec, which has some protections, annotations are fair game. Any tool or controller can come along and modify them. Don’t build critical logic that assumes an annotation’s value is permanent or immutable.
  3. They Are Not for Application Config: This is the big one. Your application inside the container should not be relying on Kubernetes annotations for its runtime configuration. The app has no easy, native way to read them. That’s what environment variables, ConfigMaps, and Secrets are for. Annotations are for the orchestrator, not the orchestrated.

In short, use annotations as the versatile scratchpad they are. They’re how you make Kubernetes play nicely with the rest of your ecosystem. Just remember: with great power (and 256KB values) comes great responsibility. Don’t be the person who brings the API server to its knees with a Pod manifest containing the complete works of Shakespeare in an annotation.