30.7 Continuous Fuzzing in CI
Right, so you’ve got your fuzzer working on your machine. It’s finding some gnarly stuff. You feel like a wizard. Don’t get too comfortable. The real magic—and the real pain—happens when you stop running this thing manually and shove it directly into the cold, unforgiving heart of your CI pipeline. This is where we move from a cool party trick to a relentless, 24/7 bug-hunting cyborg that works while you sleep. The goal is to make the pipeline so angry it emails you at 3 AM. You’ll thank me later.
The Core Concept: It’s Just a Job
At its simplest, continuous fuzzing in CI is just another job in your ci.yml or whatever config file you’re using. It’s not special. You check out the code, build it (this is the most critical part, get this wrong and you’re wasting everyone’s time), and then run the fuzzer for a fixed amount of time or executions. The key is that you’re doing this on every merge to your main branch, or better yet, on a scheduled cron job to fuzz the latest code every night.
Here’s a brutally simple GitHub Actions example to get the idea across. We’ll use the excellent google/oss-fuzz infrastructure, which is basically the industry standard way to do this properly.
# .github/workflows/fuzz.yml
name: Fuzzing
on:
schedule:
- cron: '0 0 * * *' # Run daily at midnight UTC
push:
branches: [ main ]
jobs:
fuzz:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Build Fuzzers
run: |
# This is the part that will be unique to YOUR project.
# You need to compile your code and the fuzz target with the fuzzing engine (e.g., libFuzzer, AFL++)
# This often means using `-fsanitize=fuzzer,address` or similar.
make clean
CC=clang CXX=clang++ CFLAGS="-fsanitize=fuzzer-no-link,address" make all
- name: Run Fuzzer for a while
run: |
# Run for 60 seconds, or until a crash is found
./my_fuzz_target -max_total_time=60
Notice the -max_total_time flag. This is crucial. You can’t let a fuzzing job run indefinitely in a typical CI job; you’d burn through your credits and annoy your team. You give it a fixed budget of time (like 10 minutes) and let it do its best. The value is in the aggregate. Ten minutes today, ten minutes tomorrow, ten minutes the day after… that adds up to a lot of coverage.
Why Building Correctly is Your #1 Problem
I cannot stress this enough: if your build is not identical to your production build but with the fuzzing instrumentation slapped on, you are fuzzing a different program. You will find bugs that don’t exist in production and, worse, you will miss bugs that do. The most common pitfall is forgetting to include all the necessary sanitizers. You want AddressSanitizer (ASAN) and UndefinedBehaviorSanitizer (UBSAN) enabled at a minimum. This is what turns a simple memory overwrite into a beautifully detailed crash report.
Your build command should look something like this monstrosity:
clang -fsanitize=fuzzer,address,undefined -g -O1 -o my_fuzz_target my_fuzz_target.c
-O1 is a sweet spot for fuzzing. You want some optimization so the code is fast, but not so much that it mangles the stack traces and makes debugging impossible. -g includes debug symbols, which are non-negotiable if you ever want to understand where the crash happened.
Corpus Management: The Secret Sauce
Running a fuzzer with an empty seed corpus every time in CI is like trying to start a fire by rubbing two cold, wet sticks together. It’s dumb and inefficient. The fuzzer spends its first precious minutes just discovering basic valid inputs. The solution is to persist the corpus between runs. OSS-Fuzz does this automatically, but if you’re rolling your own, you need to think about it.
You can use your CI’s caching mechanism to store the corpus. The next run downloads the corpus, runs the fuzzer, which generates new interesting inputs, and then uploads the new, larger corpus back to the cache.
- name: Download Corpus Cache
uses: actions/cache@v3
with:
path: ./corpus
key: ${{ runner.os }}-fuzz-corpus-${{ hashFiles('my_fuzz_target.c') }}
restore-keys: |
${{ runner.os }}-fuzz-corpus-
- name: Run Fuzzer
run: |
# The fuzzer will use the ./corpus directory and update it with new finds
./my_fuzz_target -max_total_time=300 ./corpus
This is a bit more advanced, but it multiplies the effectiveness of your short CI runs by orders of magnitude. The fuzzer immediately starts mutating from a set of known-good (and known-interesting) inputs.
Triage: The 3 AM Email
Okay, the fuzzer found a crash. Now what? Your CI job shouldn’t just fail silently. You need to capture the output, especially the sanitizer stack trace, and get it to a human. Immediately.
The best practice is to have the job, upon a non-zero exit code (which libFuzzer does on a crash), package up the crashing input and the log output and create a bug report. You can use the CI API to automatically file an issue in your tracker or send a message to a Slack channel. The report must include the exact input that caused the crash. You can save it as a file and attach it.
# After the fuzzer run command in your CI step
if [ $? -ne 0 ]; then
echo "Fuzzer crashed! Saving input and stack trace."
# The fuzzer usually saves the crashing input automatically, but let's be sure.
# The stack trace is in the stdout/logs, which CI will capture.
# Here you'd add commands to use the GitHub API to create an issue.
exit 1
fi
The beauty of this system is that it’s automated and relentless. It doesn’t get tired. It doesn’t forget. It just constantly probes the darkest corners of your code and screams when it finds something that shouldn’t be there. Your job is to build the system, and then listen for the scream.