Right, let’s talk about lsof. It’s one of those tools you’ll use once a year and think, “Why don’t I use this magnificent beast more often?” Its name, lsof, is a classic piece of Unix utility naming: brutally literal. It stands for “list open files.” But on a Unix-like system, everything is a file. And I mean everything. Your hard drive? File. Your network connection? File (a socket, technically). That running process? You get the idea. This means lsof is your ultimate backstage pass to see what every process on your system is actually doing—what it’s reading, writing, and talking to. It’s the debugger’s equivalent of a truth serum.

The Absolute Basics: What’s Open Right Now?

Running lsof with no arguments is a terrible idea. It will list every open file for every process on the system, which is usually thousands of lines. You’ll immediately scroll past anything useful. Don’t do it. Instead, start by asking a specific question. The most common one: “What is process ID 1234 up to?”

lsof -p 1234

This spits out everything that process has open. You’ll see its current working directory (cwd), the binary it’s running (txt), loaded libraries, and any files it’s reading or writing. The output columns are mostly self-explanatory: COMMAND, PID, USER, FD (file descriptor), TYPE, DEVICE, SIZE/OFF, and NAME. The FD column is where the magic is. cwd is the current working directory, txt is the program text (code), mem is a memory-mapped file, and a number like 3u is a genuine file descriptor (3) with its mode (u for read/write).

Finding What’s Holding That Pesky File Hostage

You’ve been there. You try to delete a file or unmount a filesystem and the OS yells at you: “Device or resource busy.” You curse. Then you remember lsof.

lsof /path/to/annoying/file
lsof +f -- /mount/point

The first command lists any process that has that specific file open. The second is for dealing with filesystems mounted at /mount/point; the +f flag tells lsof to treat the path as a mounted filesystem, which is crucial for finding what’s preventing an umount. Find the PID, and you can then politely ask (kill -SIGTERM) or violently force (kill -SIGKILL) the process to let go.

Network Detective Work: What’s Listening and Who’s Connected?

This is where lsof truly shines and leaves other network tools in the dust. You want to see all network connections? Easy.

lsof -i

But again, be specific. The -i flag can take arguments to filter. Want to see everything listening on a port?

lsof -i :80

Want to see all TCP connections to a specific host?

lsof -i @192.168.1.10

Combine filters. To see all UDP traffic on port 53 (DNS)? lsof -i UDP:53. The beauty here is that you immediately see the process behind the port. netstat -tulpn will give you similar info, but lsof’s output is often clearer and its filtering syntax is, in my opinion, more intuitive once you get the hang of it.

The “Why” Behind the Weirdness: Deleted Files

Here’s a classic head-scratcher that lsof explains perfectly. You check your disk space with df and it says you’re at 99% capacity. You run du -sh / to find the big files and… the numbers don’t add up. There’s free space missing. This is almost always because a large file was deleted while still being held open by a process. The disk space won’t be freed until that process closes the file descriptor. lsof to the rescue:

lsof +L1

The +L1 flag lists all open files that have a link count less than 1—meaning the filename has been deleted from the filesystem, but a process still has a handle to it. The SIZE/OFF column will show how large the file was. Find the PID, restart the process, and your disk space magically reappears. It feels like witchcraft.

Best Practices and Pitfalls

First, always run lsof with sudo. Without elevated privileges, it can only see files owned by your user. You’ll get a partial, often misleading, view of the system. sudo lsof is the only way to see everything.

Second, its output is wide. Really wide. Be ready to pipe it to less -S to scroll horizontally, or use awk and cut to parse out the specific columns you need for scripting.

Finally, remember that lsof is a snapshot. It shows you the state of the world the instant you ran it. For something that changes rapidly, like network connections, you might want to watch -n 2 sudo lsof -i @some.host to get a repeating, updating view.

It’s not a tool you need every day, but when the problem is “what is this thing doing,” it’s often the only tool that can give you the direct, unvarnished answer.