Right, let’s talk about iostat. You’ve probably felt it—that creeping suspicion that your application is just… waiting. Waiting on what? The disk. It’s the slowest part of the whole party, and when it’s struggling, everything grinds to a halt. top might show you a sky-high %wa (I/O wait), but that’s just the headline. iostat is the full, gritty investigative report. It tells you which disk is the problem child and exactly what kind of tantrum it’s throwing.

Think of iostat as your dashboard for the spinning platters (or flash memory) that hold all your precious data. It breaks down the raw, unfiltered truth about throughput, utilization, and latency. We’re going to use the version that comes with the sysstat package on Linux, which is the good one. If you just type iostat, you’ll get a single, lonely snapshot of CPU and device stats since boot. That’s about as useful as a single frame from a movie. You need to see the story unfold.

The Basic Incantation: Watching the Story Unfold

To make iostat actually useful, you need to run it with an interval. This tells it to report continuously, giving you a live view of the action.

iostat -dx 2

Let’s break this down because these options are non-negotiable:

  • -d: Shows only the device statistics, sparing you the CPU info you probably don’t need right now.
  • -x: This is the magic switch. It gives you the extended statistics, which is where all the truly insightful data lives. Never run iostat without -x. Just don’t.
  • 2: The interval in seconds. It will report every 2 seconds. I find 1-3 seconds is the sweet spot for diagnosing most issues.

Here’s what a typical output might look like (I’ve trimmed the columns for clarity):

Device            r/s     w/s    rkB/s    wkB/s  await  %util
sda              0.00   55.00     0.00  4688.00   8.73  48.00
dm-0             0.00   55.00     0.00  4688.00   9.12  48.20

Now, let’s translate this from sysadmin-ese.

Decoding the Hieroglyphics: What Actually Matters

The columns are the heart of it. You don’t need to memorize them all, but these are the big ones:

  • r/s and w/s: Read and write requests issued to the device per second. A high number here means a lot of I/O operations. But are they big or small?
  • rkB/s and wkB/s: The actual throughput in kilobytes per second. This tells you the volume of data moving. Multiply by 1024 if you need bytes. Combine this with the requests per second to understand the I/O pattern. 1000 requests/sec at 4kB/s is a lot of tiny, inefficient requests. 10 requests/sec at 400MB/s is large, sequential, efficient ones.
  • await: This is the average time (in milliseconds) for an I/O request to be served. This includes time spent in queue and time being serviced. This is your user’s perception of slowness. If this number is high, your users are waiting. It’s the most direct measure of pain.
  • %util: This is the percentage of time the device was busy processing requests. It’s a rough gauge of saturation. If this is near 100%, your device is screaming for mercy and is unquestionably your bottleneck. Here’s a designer’s questionable choice: for modern SSDs and RAID arrays, this can be misleadingly low even under heavy load because they can handle massive parallel queues. It’s still a great first signal, but don’t trust it blindly.

The Gotchas and The Glory

A common pitfall is not checking the right device. On most modern systems, your actual disk (sda) is wrapped in a logical device mapper (dm-0), which is where your filesystem actually lives. You need to be looking at the dm-0 line, not the sda line, to see the true workload hitting your OS. If sda has high await but dm-0 doesn’t, the problem is in the virtual device layer or the filesystem itself, not the physical hardware.

Another edge case: high await but low %util. This is a classic sign of an overloaded external storage array or a disk that’s about to fail and is taking forever to respond. The OS is sending requests, but the device itself is struggling to keep up, so the queue isn’t full, but the service time is terrible.

The best practice? Always run iostat -dx [interval] and watch for trends. Is await spiking? Is %util pegged at 100%? Is the throughput (rkB/s/wkB/s) what you’d expect from your hardware? A SATA SSD should do tens of thousands of IOPS and hundreds of MB/s. If you’re seeing 100 IOPS and 5MB/s on that hardware, something in your application or OS configuration is deeply, profoundly wrong. iostat gives you the data to prove it.