Right, so you’ve got your service humming along, doing its little job, and you think it’s all going well. But how do you know? A hunch? A feeling? A user screaming in all caps in your support tickets? We can do better. We’re going to instrument this thing, which is a fancy way of saying we’re going to teach it to tattle on itself. We’re going to use Prometheus, the de facto standard for metric collection in the cloud-native world, and we’re going to do it the right way from the start.

The first thing you need to wrap your head around is that Prometheus is a pull-based system. This isn’t some agent on your service constantly spamming a metrics backend; instead, Prometheus itself periodically reaches out (or “scrapes”) a designated HTTP endpoint on your service to collect the latest metrics. This is a brilliant design for simplicity and reliability, but it means our job is to expose that endpoint cleanly.

The promhttp.Handler() is Your Best Friend

Forget manually building your metrics endpoint. The prometheus/promhttp package is here to do the heavy lifting. Your main function should look something like this. Notice how we namespace our metrics with our service name? This isn’t just tidy; it’s a lifesaver when you’re staring at a graph with 50 different request_total metrics.

package main

import (
	"net/http"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
	// Create a new registry. This is where your metrics will be registered.
	registry := prometheus.NewRegistry()

	// The standard process and Go runtime metrics are insanely useful.
	// Not adding them is like flying blind. Don't be that person.
	registry.MustRegister(prometheus.NewProcessCollector(prometheus.ProcessCollectorOpts{}))
	registry.MustRegister(prometheus.NewGoCollector())

	// ... (You'll register your custom metrics here later)

	// This handler is specifically for our metrics, using our custom registry.
	metricsHandler := promhttp.HandlerFor(registry, promhttp.HandlerOpts{})
	http.Handle("/metrics", metricsHandler)

	// This is your actual application handler
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte("Hello, World!"))
	})

	http.ListenAndServe(":8080", nil)
}

Run this, curl localhost:8080/metrics, and behold! A beautiful wall of text detailing everything about your process’s memory, goroutines, and GC pauses. It’s like getting a full-body MRI for your application, for free.

Counting Things: The Humble Counter

The workhorse of observability. You use a counter for anything that only goes up: total requests, tasks completed, errors, times you’ve muttered “it worked on my machine.” Let’s create one to track HTTP requests.

// Define our metrics in a struct to keep things organized.
type metrics struct {
	httpRequestsTotal *prometheus.CounterVec
}

func newMetrics(reg prometheus.Registerer) *metrics {
	m := &metrics{
		// We use a CounterVec to have labels (method, path, status_code)
		httpRequestsTotal: prometheus.NewCounterVec(prometheus.CounterOpts{
			Namespace: "myapp",
			Name:      "http_requests_total",
			Help:      "Total number of HTTP requests.", // This Help string is mandatory. Yes, Prometheus is judging you if you write something useless.
		}, []string{"method", "path", "status_code"}),
	}

	// Register the metric with the provided registry
	reg.MustRegister(m.httpRequestsTotal)
	return m
}

// In your main function, after creating the registry:
m := newMetrics(registry)

// Then inside your actual HTTP handler:
http.HandleFunc("/api", func(w http.ResponseWriter, r *http.Request) {
	// ... handle the request ...
	statusCode := 200

	// Increment the counter with the relevant labels.
	// Note: We use the pattern here, not the full URL, to avoid cardinality explosion.
	m.httpRequestsTotal.WithLabelValues(r.Method, "/api", fmt.Sprintf("%d", statusCode)).Inc()

	w.Write([]byte("Done!"))
})

The Cardinal Sin: Label Cardinality

See what I did with the path label above? I used a constant pattern ("/api") instead of the actual r.URL.Path. This is the most important best practice to internalize. Do NOT use user-controlled or high-cardinality data as a label value. If you label a metric by user_id, and you have ten million users, you just created ten million new time series. Prometheus will eat your RAM for breakfast and then die. It’s the number one way people blow up their monitoring systems. Use labels for bounded sets of values: method (GET, POST, etc.), status_code (200, 404, 500), or predefined route names.

Timing Things: The Insightful Histogram

Knowing how many requests you get is useless if you don’t know how long they take. For this, we use a Histogram. A histogram doesn’t just give you the average; it buckets the durations, allowing you to calculate useful percentiles later (like the all-important p95 or p99).

func newMetrics(reg prometheus.Registerer) *metrics {
	m := &metrics{
		httpRequestsTotal: prometheus.NewCounterVec(...), // from before
		httpDuration: prometheus.NewHistogramVec(prometheus.HistogramOpts{
			Namespace: "myapp",
			Name:      "http_response_time_seconds",
			Help:      "Duration of HTTP requests.",
			Buckets:   prometheus.DefBuckets, // This uses the default buckets. They're good. Start with these.
		}, []string{"method", "path"}),
	}
	reg.MustRegister(m.httpDuration)
	return m
}

// Using it with a timer is elegantly simple:
http.HandleFunc("/api", func(w http.ResponseWriter, r *http.Request) {
	timer := prometheus.NewTimer(m.httpDuration.WithLabelValues(r.Method, "/api"))
	defer timer.ObserveDuration() // This records the time when the function returns.

	// ... handle the request ...
})

The beauty of this is that in PromQL, you can now query for the 99th percentile latency of your POST requests to /api. That’s powerful stuff.

The Global Registry Trap

You might see older examples use prometheus.MustRegister(...) directly. This uses a hidden global registry. It’s convenient for a tiny example but becomes a nightmare for testing. How do you isolate tests from each other? You can’t. It’s like using a global variable for your database connection. Our approach above, injecting a registry, is far superior. You can create a fresh registry for every unit test, ensuring no state leaks between them. Trust me, your future self will thank you for avoiding the global mess.