Shrink a Failing LLVM Test Case with bugpoint-18

Got a 40,000-line LLVM IR file that crashes the optimiser? bugpoint-18 will whittle it down to something you can actually read. You finish with a smaller LLVM IR or bitcode testcase that still shows the failure. This guide uses the installed bugpoint-18 from Ubuntu's LLVM 18 package, version 1:18.1.3-1ubuntu1. Allow 15 to 30 minutes for a small, repeatable failure; a large testcase or a slow compiler can take much longer.

You need:

Security warning: running bugpoint on untrusted input can execute programs and load shared objects. Use a disposable working directory and normal user privileges.

1. Confirm the installed command

Check the package and executable before starting. This catches the classic mistake of reading documentation for one LLVM release while invoking another.

$ dpkg-query -W -f='${Package} ${Version}\n' llvm-18
llvm-18 1:18.1.3-1ubuntu1
$ command -v bugpoint-18
/usr/bin/bugpoint-18
$ bugpoint-18 -help | head -n 5
OVERVIEW: LLVM automatic testcase reducer. See

Your package revision or help text may differ after an update. What matters is that the command exists and prints its own help.

Checkpoint: if the executable is missing, stop here. Install or select the matching LLVM package through your normal system process; do not run a guessed path as root.

2. Put the original testcase in a disposable directory

Make a directory and copy the input without replacing the source file. Substitute a real path for /path/to/failure.ll.

$ mkdir -p "$HOME/bugpoint-work"
$ cp -- /path/to/failure.ll "$HOME/bugpoint-work/original.ll"
$ cd "$HOME/bugpoint-work"
$ test -s original.ll && echo 'original testcase is readable'
original testcase is readable

If your input is bitcode, keep the .bc suffix and use that filename in later commands. Avoid putting generated files beside a source tree that is under version control.

Checkpoint: confirm that original.ll fails with the exact command you are investigating. Bugpoint is a reducer, not a general-purpose fault finder: it needs a test that is already interesting.

3. State the failure in a command bugpoint can repeat

For an optimiser or pass failure, give bugpoint the input followed by the pass names. This example investigates instcombine; swap in the pass from your report.

$ bugpoint-18 original.ll -instcombine

Bugpoint repeatedly removes parts of the module and tests whether the failure remains. It normally uses internal cleanup passes while reducing. If the suspected fault is in dead-code elimination or control-flow simplification, disable the relevant reducer so bugpoint does not rely on the pass being examined:

$ bugpoint-18 original.ll -instcombine --disable-dce --disable-simplifycfg

Options may be written in the forms shown by the installed manual. The command can take a while and may leave reduced files in the current directory. Read its progress rather than assuming the first line is the result.

The process exits with status 0 when bugpoint finds a problem and non-zero when an error occurs. Capture the status immediately:

$ status=0
$ bugpoint-18 original.ll -instcombine || status=$?
$ printf 'bugpoint status: %s\n' "$status"
bugpoint status: 0

Tip: a zero status means bugpoint found the problem according to the test it was given. It does not prove the reduced file has the same root cause, so compare the diagnostic and rerun the original tool yourself.

4. Separate program arguments from bugpoint options

If the testcase is run as a program, put its arguments after --args. If an argument begins with a hyphen, add another -- straight after --args; otherwise bugpoint can read that argument as its own option.

$ bugpoint-18 reduced-input.ll --args -- --mode --input sample.dat

The first -- belongs to bugpoint's argument separator. The later values go to the test program. For arguments that do not begin with a hyphen, the simpler form is enough:

$ bugpoint-18 reduced-input.ll --args sample.dat

Use --tool-args -- ... for options meant for the LLVM tool under test, such as llc or lli. Do not confuse that with --args, which is for the generated test program.

5. Supply a stable reference result when needed

When the test produces output, pass a known-good reference file with --output. Bugpoint compares the test program's standard output with that file. Without this option it tries to generate a reference using its safe backend, which may not match a test that depends on external behaviour.

$ bugpoint-18 original.ll -instcombine \
    --output expected.txt \
    --args -- --mode sample.dat

For a code-generator investigation, choose the execution and safe backends explicitly:

$ bugpoint-18 original.bc --run-custom --safe-custom \
    --exec-command /path/to/run-test \
    --output expected.txt

Safety warning: a custom command, input file and loaded plugin can execute arbitrary code. Inspect scripts and shared objects before passing them to --exec-command, --load or --additional-so. Do not use elevated privileges to make a failing test run.

6. Check the reduced result and keep a way back

When bugpoint reports a reduced testcase, list the files and inspect the relevant output. The exact filenames depend on the failure and tool, so do not hard-code a guessed result path into automation.

$ find . -maxdepth 1 -type f -printf '%f\n' | sort
$ opt-18 -verify -disable-output /path/to/the-reduced-file.ll

Replace the last path with the filename bugpoint reports. The opt-18 -verify command checks LLVM IR validity; it does not reproduce the original failure. Re-run the failing tool or test with the reduced input and compare the diagnostic with the original. Keep original.ll and the command line in a small text note until that comparison is done.

Two options help when the output is ambiguous:

Recovery: if a reduction goes the wrong way, stop it with your terminal's interrupt key, remove only generated files after checking their names, and rerun from the untouched original in a new directory. Do not delete the original testcase as a shortcut.

Done means