Right, so you’ve got your application instrumented, your spans are flying, and your OpenTelemetry Collector is dutifully collecting. Fantastic. But that telemetry data has to go somewhere. You can’t just shout your traces into the void and hope for the best (though I’ve seen teams try). This is where Tempo comes in. Think of it as Grafana’s purpose-built, highly scalable, and refreshingly simple parking garage for your trace data. It’s not trying to be a general-purpose database; it’s built from the ground up to do one thing incredibly well: store and retrieve traces, fast.

Why Tempo Exists (The “Duh, Obvious” Moment)

You could shove your traces into a general-purpose database like PostgreSQL or even an index-heavy monster like Elasticsearch. For a toy project, you might get away with it. In production, you’ll watch it catch fire in real-time. Traces are not like metrics or logs. A single user request can generate a trace comprised of dozens of spans, which is a deeply nested, connected graph of data. Querying for a specific trace means finding one specific graph in a haystack of billions. Relational databases choke on this. Index-based systems become horrifically expensive, both in storage and compute, because you’d have to index nearly every field to make arbitrary tracing queries work.

Tempo sidesteps this entirely with a brutal, genius insight: the primary access pattern for a trace is almost always by its trace ID. You find a trace ID in your logs or from your metrics, and then you want to see the whole darn thing. Tempo is optimized for this exact query. It’s a deep, cheap object store for your traces, and it lets other, more index-happy systems (like Loki for logs or Prometheus for metrics) do what they’re good at: finding which trace ID you should be looking for. This separation of concerns is its superpower.

The Absolute Guts of How Tempo Works

Tempo’s architecture is a thing of beauty in its directness. It breaks a trace into pieces, writes them in a structured way to object storage (like S3, GCS, or Azure Blob), and uses a minimal index to glue it all together.

  1. The Ingester: This is the front door. It receives spans (usually via OTLP), groups them by trace ID, and batches them up. Once a batch is full or a timeout is hit, it writes two things:
    • The Trace Data: A complete batch of spans is compressed and written as a block to your object storage backend. This is the main event, the payload.
    • The Index: A tiny, pared-down index (often using Apache Parquet) that maps the trace ID to the exact object where its data lives. This is written to a separate index storage backend (which can also be object storage).

This means when you query Tempo for trace ID abc123, it doesn’t scan all your data. It first checks the index to find which block file contains trace abc123, fetches only that one block from object storage, and decompresses it for you. It’s brutally efficient.

Here’s a minimal docker-compose.yml to get Tempo running locally and see this in action. We’ll point an OpenTelemetry Collector at it.

# docker-compose.yml
version: "3.8"

services:
  tempo:
    image: grafana/tempo:latest
    command: ["-config.file=/etc/tempo.yaml"]
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml
      - ./data/tempo:/tmp/tempo
    ports:
      - "3200:3200"   # tempo
      - "9411:9411"   # zipkin ingest (optional)

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
    depends_on:
      - tempo

And the crucial tempo.yaml config. Notice the complete lack of fuss.

# tempo.yaml
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
        http:
    zipkin:

ingester:
  trace_idle_period: 10s
  max_block_bytes: 1_000_000
  max_block_duration: 5m

storage:
  trace:
    backend: local
    local:
      path: /tmp/tempo
    pool:
      max_workers: 100
      queue_depth: 10000

Finally, the Collector config to bridge your app to Tempo:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  logging:
    loglevel: debug
  otlp/http:
    endpoint: "http://tempo:3200"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging, otlp/http]

Querying: It’s a Team Sport

Here’s the part everyone misses at first: You don’t usually query Tempo directly. Remember that separation of concerns? Tempo is the deep storage. To find a trace, you use Grafana. Grafana talks to Tempo’s Query Frontend, which is the component that knows how to take a trace ID, find its block in the index, and retrieve it.

You paste a trace ID into Grafana’s Explore view, and it handles the rest. The real magic is when you use Grafana’s built-in correlations to click from a log line in Loki that contains a trace ID and jump directly to that trace in Tempo. That’s the workflow. Tempo is the silent partner in that operation.

The Rough Edges and Pitfalls (Because Nothing is Perfect)

  • The “No Arbitrary Querying” Trade-off: This is the big one. You can’t just ask Tempo, “show me all traces where the http.status_code was 500 and the service.name was checkout-service.” That’s not what it’s for. For that, you need a metrics system that can derive that information from your traces as they pass through the collector (e.g., creating a metric from span data) or a dedicated trace query engine like Jaeger’s. This catches people off guard.
  • Cold Starts: Since Tempo uses object storage, a query for a trace that hasn’t been accessed in a while might be slightly slower as it pulls the data from a “colder” storage tier. For the vast majority of recent traces, it’s blazingly fast.
  • Configuration is… Spartan: The config files are simple, which is great, but the options for tuning the ingester, block sizes, and caching are critical for high-scale deployments. Don’t just copy a config from a blog post and call it a day. Understand the max_block_duration and max_block_bytes settings—they control the trade-off between query latency and overall efficiency.

Tempo’s elegance is in its specialization. It doesn’t try to be everything. It’s the best-in-class vault for your trace data, and it knows it. Use it as the backbone of your tracing pipeline, let other tools handle the finding, and you’ll have a robust, scalable, and frankly, affordable system that won’t melt down when your traffic spikes.