Right, let’s talk about strace. You’ve probably hit a wall where your application is doing… something… but you have no earthly idea what. It’s not logging, it’s not crashing, it’s just sitting there, taunting you. This is where strace becomes your best friend. It’s the debugger of last resort, the ultimate truth-teller. It shows you the raw conversation between your program and the Linux kernel—every file it opens, every network call it makes, every time it asks the system for the time of day. It doesn’t lie.

Think of it as putting a wiretap on your process. It intercepts and prints all the system calls and signals it encounters. This is the low-level, nitty-gritty stuff that most high-level languages try to abstract away from you. When your fancy Python script hangs, strace will show you it’s actually stuck on a read() call from a socket that’s never going to respond. It cuts through the abstraction and tells you what’s really happening.

The Absolute Basics: Following a Program’s Trail

The simplest way to use strace is to just launch a program with it. Let’s start with something trivial, like finding out what ls actually does under the hood.

strace ls

You’ll be immediately bombarded with a firehose of output. This is normal. Your terminal will scream with lines like:

execve("/usr/bin/ls", ["ls"], 0x7ffc445f5c80 /* 58 vars */) = 0
brk(NULL)                               = 0x55a1a1b2e000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
...

Each line is a system call. It shows the name of the call (e.g., openat), its arguments (often in a human-readable form, thanks to strace), and its return value (e.g., = 3). That = 3 is crucial—it’s a file descriptor. The next call that uses 3 is probably going to be reading from that file.

But what if your program is already running? You don’t want to restart it and potentially lose its state. That’s where the -p (pid) flag comes in.

# First, find the PID of your rogue process
ps aux | grep my_app

# Then, attach strace to it
strace -p 1234

Now you’ll see a live feed of what that process is doing. To detach, hit Ctrl+C.

Cutting Through the Noise: Useful Flags You Actually Need

The raw output is overwhelming. You need filters. The two most important flags in your arsenal are -e (for events) and -f (for follow).

The -e flag lets you focus on only the system calls you care about. Want to see only file operations?

strace -e trace=open,read,write,close ls /tmp

Maybe your issue is network-related. Focus on those calls:

strace -e trace=network curl -s http://example.com > /dev/null

The -f flag is non-negotiable if your process spawns children (which almost any web server, compiler, or complex tool will do). Without -f, you only trace the parent process and miss all the interesting work the children are doing. -f follows fork/clone.

strace -f -e trace=read,write my_app

Another lifesaver is -o to write the output to a file. Trust me, you do not want to read gigabytes of strace output scrolling through your terminal.

strace -f -o /tmp/strace_my_app.log ./my_app

Reading the Output: It’s Not Just Gibberish

Let’s break down a real line. You’ll see a lot of this:

write(1, "hello world\n", 12)          = 12

This is the C library’s printf function ultimately making a write system call.

  • 1 is the file descriptor (1 is standard output).
  • "hello world\n" is the pointer to the data it wants to write.
  • 12 is the number of bytes it wants to write.
  • = 12 is the return value, meaning it successfully wrote all 12 bytes.

Sometimes, you’ll see failure. This is often more informative:

connect(3, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("192.168.0.10")}, 16) = -1 ECONNREFUSED (Connection refused)

Boom. There’s your problem. The process tried to connect to port 8080 on 192.168.0.10 and got rejected. No more guessing.

The Performance Killer: Context Switches

Here’s the dirty secret nobody tells you: strace is horrifically slow. It’s a debugging tool, not a production profiling tool. The reason is that every single system call triggers a context switch into the strace process itself, which logs the call, then another context switch back to your program. This can slow your application down by orders of magnitude.

If you see a performance problem while running under strace, it’s almost certainly because of strace itself. Don’t trust timing information from a straced process. For actual performance profiling, you’d use perf or bpftrace, which are much more efficient. strace is for debugging correctness, not performance.

Best Practices and Pitfalls

  1. Don’t Run It on Production. Seriously. The performance overhead is so bad it can take down a live service. If you absolutely must, attach it for a few seconds to catch a specific event and then detach immediately.
  2. Combine with -c for a Summary. If you just want a high-level count of which calls are being made, use strace -c ./my_program. It will run the program and then print a neat table of system call counts, errors, and time spent. It’s a great first recon tool.
  3. Beware of “Deferred” Knowledge. strace shows you the request, not the implementation. A write() returning successfully doesn’t mean the data hit the disk; it just means the kernel copied it into its write cache. The actual disk I/O happens later. This is why you often need to also trace fsync calls to understand true durability.
  4. It’s a Sledgehammer. The output is vast. Always use -e to narrow your focus. Start broad, then get specific. Trying to understand everything at once is a recipe for a headache.

strace is one of those tools that feels like a superpower once you get comfortable with it. It demystifies the black box of your running application and gives you a concrete, undeniable log of what it’s asking the operating system to do. It’s the first tool I reach for when something is hanging, crashing mysteriously, or accessing files it shouldn’t.