Alright, let’s talk about why you’d actually use a DaemonSet. You don’t just deploy them for fun; they solve a very specific, infrastructure-level problem: when you need a piece of software running on every single node in your cluster, come hell or high water. It’s the Kubernetes way of saying, “I don’t care what’s scheduled here, this pod is non-negotiable.” Think of them as the mandatory background services of your operating system, but for your cluster.

The classic, textbook use cases are log collectors, monitoring agents, and network plugins. Let’s break them down.

The Log Collector

Every application pod on a node writes its logs to that node’s filesystem. If a pod dies, gets rescheduled, or a node bursts into flames, those local logs are gone. Poof. This is a terrible way to run a railroad. You need a central log aggregator like Fluentd, Filebeat, or Logstash.

But here’s the catch: you can’t just run one log collector pod and call it a day. How would a pod on node-07 get its logs to a collector running on node-01? You’d be hauling logs across the network for no reason, creating a single point of failure, and generally making a mess.

A DaemonSet solves this elegantly. You deploy your log collector as a DaemonSet. Now, every node gets its own dedicated collector. It mounts the host’s /var/log directory (or wherever your container runtime and kubelet are writing logs) and tails those files, shipping them off to your central Elasticsearch, Loki, or whatever black hole you use for logs. It’s efficient, resilient, and makes perfect sense.

Here’s a brutally simplified example of a Fluentd DaemonSet. Notice the critical parts: the nodeSelector to target all nodes (we’ll get to that), the hostPath volume mount, and of course, the DaemonSet itself.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: fluentd
  template:
    metadata:
      labels:
        name: fluentd
    spec:
      # This toleration is key. It allows the pod to be scheduled on control-plane nodes.
      # Many clusters taint the control-plane to avoid user workloads.
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

The Monitoring Agent

The exact same logic applies here. Tools like Prometheus need to scrape metrics from every node’s kubelet and from the cAdvisor running on each node. You could try to point a single Prometheus server at all your nodes, but that’s clunky and doesn’t scale. The modern way is to use the Prometheus Node Exporter.

You run the Node Exporter as a DaemonSet. Each pod exposes the metrics of its host node on a local port. Then, your central Prometheus server simply scrapes every pod in the DaemonSet. Each node’s metrics are collected locally and efficiently. It’s a beautiful, decentralized model.

The YAML looks almost identical to the log collector, just a different container image. The key insight is the pattern: one agent per node, collecting local data.

The Network Plugin

This one is a bit more special. Kubernetes is famously “plug-and-play” for networking, but that plug isn’t optional. Every node must run a network plugin pod (like Calico’s calico-node or Cilium’s agent) to handle the complex networking magic that allows pods on different nodes to talk to each other. This is not a suggestion; it’s a requirement for the cluster to function.

So how do you ensure this absolutely critical pod is on every node, including ones that join the cluster later? You guessed it: a DaemonSet. The network plugins you install are almost always deployed this way. They often require higher privileges (hostNetwork: true, elevated capabilities) to mess with the node’s networking stack, which is a DaemonSet’s bread and butter.

The Devil’s in the Details

Now, the gotchas. First, tolerations. Your cluster’s control-plane nodes are almost certainly tainted with something like node-role.kubernetes.io/control-plane:NoSchedule. This is to keep your application pods off them. But your DaemonSet for logging or monitoring? You probably do want it on the control-plane nodes. You want those logs and metrics too! So you must add the appropriate tolerations to your Pod spec, like in the Fluentd example above. If you forget this, your DaemonSet will politely ignore the control-plane nodes, which is rarely what you want.

Second, node selection. By default, a DaemonSet targets every node. But what if you have a special pool of nodes with GPUs that don’t need your standard log collector? You can use a nodeSelector in the Pod spec to target only nodes with specific labels. This is how you make DaemonSets behave themselves in complex, heterogeneous clusters.

The bottom line: Use a DaemonSet when the workload is intrinsically tied to the node itself, not to your application. It’s for the plumbing, not the furniture. And get those tolerations right, or you’ll be left wondering why your master nodes are mysteriously silent.