Write Precise LLVM Tests with FileCheck-20

A test that greps for one word will pass on almost anything, so FileCheck-20 checks order, adjacency and consistency instead. You will build a small LLVM-style regression test that checks tool output in order, tolerates values that legitimately change, and rejects an unwanted line.

FileCheck-20 reads patterns from a file and verifies text supplied on standard input. Allow about 15 minutes if LLVM is already installed. No elevated privileges are needed.

1. Confirm the installed tool

This guide uses the Ubuntu LLVM 20 package installed on this machine: llvm-20, version 20.1.8~++20250804090239+87f0227cb601-1~exp1~20250804210352.139. The executable is /usr/bin/FileCheck-20. Check your own host before copying a test into a project:

$ FileCheck-20 --version
Ubuntu LLVM version 20.1.8
  Optimized build.

FileCheck is a verifier, not a generator. Another command produces the output; FileCheck compares it with directives in a match file. A successful comparison returns status 0. A mismatch, malformed directive or other error returns non-zero.

2. Create a minimal ordered check

Make a temporary match file and an input file. The ordinary CHECK: directive matches a fixed string somewhere after the previous match. So the matches must appear in the order written, though unrelated lines may sit between them.

$ cat > /tmp/filecheck-input.txt <<'EOF'
compiler: llvm
target: x86_64
status: ready
EOF
$ cat > /tmp/filecheck-check.txt <<'EOF'
CHECK: compiler: llvm
CHECK: target: x86_64
CHECK: status: ready
EOF
$ FileCheck-20 /tmp/filecheck-check.txt < /tmp/filecheck-input.txt
$ printf 'status=%s\n' "$?"
status=0

Checkpoint: the final status must be 0. Swap the target and status lines in the input and the test fails, because FileCheck checks a sequence, not whether each word exists somewhere.

3. Make line boundaries and whitespace explicit

By default a positive pattern can match part of a line, and horizontal spaces and tabs count as alike. That suits assembly and formatted diagnostics, but it can hide an accidental suffix or spacing change.

$ FileCheck-20 --match-full-lines /tmp/filecheck-check.txt < /tmp/filecheck-input.txt
status=0
$ FileCheck-20 --match-full-lines --strict-whitespace /tmp/filecheck-check.txt < /tmp/filecheck-input.txt
status=0

Tip: these options do not resize, normalise or rewrite the input file. They change what the verifier accepts, so use them only where the output format genuinely needs that precision.

4. Capture values that should stay consistent

Do not hard-code a register number or temporary identifier when the test only needs it to be consistent. A named variable definition uses double square brackets, and the text between the colon and the closing brackets is a regular expression. Reuse the captured value by name.

$ cat > /tmp/filecheck-vars.txt <<'EOF'
CHECK: result [[REG:r[0-9]+]] = add
CHECK: use [[REG]]
EOF
$ cat > /tmp/filecheck-vars-input.txt <<'EOF'
result r7 = add
use r7
EOF
$ FileCheck-20 /tmp/filecheck-vars.txt < /tmp/filecheck-vars-input.txt
status=0

Example: the first pattern captures r7 and the second requires that exact value, so a later use r8 fails. That is stronger than a broad wildcard, which could let an internally inconsistent result through.

5. Choose the right relationship between checks

Use the specialised directives when adjacency or absence matters:

Tip: CHECK-NEXT: and CHECK-SAME: cannot be the first directive, so put an ordinary CHECK: before them. Keep CHECK-NOT: bounded by useful positive checks where you can, because an unbounded negative assertion can cover a much larger region than intended.

$ cat > /tmp/filecheck-negative.txt <<'EOF'
CHECK: start
CHECK-NOT: warning:
CHECK: done
EOF
$ printf '%s\n' 'start' 'all good' 'done' | FileCheck-20 /tmp/filecheck-negative.txt
status=0
$ printf '%s\n' 'start' 'warning: retrying' 'done' | FileCheck-20 /tmp/filecheck-negative.txt
status=1

Warning: do not treat a failing check as a safe pass in a shell pipeline. Preserve the verifier's status and make the surrounding test runner fail when FileCheck returns non-zero.

6. Share one match file between configurations

Use --check-prefixes=CPU32,COMMON when one match file holds directives for more than one run. FileCheck reads both CPU32: and COMMON: directives in that invocation.

$ cat > /tmp/filecheck-prefixes.txt <<'EOF'
COMMON: shared output
CPU32: 32-bit output
EOF
$ printf '%s\n' 'shared output' '32-bit output' | FileCheck-20 --check-prefixes=CPU32,COMMON /tmp/filecheck-prefixes.txt
status=0

FileCheck also reads options from FILECHECK_OPTS. That environment variable can silently affect a local or CI run, so inspect it when two apparently identical commands behave differently:

$ printf 'FILECHECK_OPTS=%s\n' "${FILECHECK_OPTS-}"

7. Diagnose a failing comparison

On failure, LLVM 20 normally dumps the relevant input with annotations. Ask for more context when the mismatch is buried in generated output:

$ FileCheck-20 --dump-input=fail --dump-input-context=10 /tmp/filecheck-check.txt < /tmp/filecheck-input.txt
status=$?

Use -v for successful directive matches and -vv for extra detail about DAG matches, implicit end-of-file checks and negative patterns. These write diagnostics to the terminal and do not alter either file. Check the exit status immediately, before running another command.

Recovery: the usual culprits are a missing match file, a directive prefix that does not match the selected --check-prefixes, a first directive of CHECK-NEXT or CHECK-SAME, and a regex wider or narrower than intended. If empty input is a valid test case, pass --allow-empty, since empty input is rejected by default.

Done means