Test Compiler Output with FileCheck-18

Your compiler output changed by one register number and forty tests went red, so you need FileCheck-18 to check what matters and ignore what does not. By the end you will have a small test that verifies generated text in order, rejects an unwanted line, and tolerates values that legitimately vary.

The installed command is FileCheck-18 from the llvm-18 package, reporting LLVM version 18.1.3 on this machine. Allow about fifteen minutes.

• You need: a shell and the LLVM 18 tools package.
• It will not touch: anything outside your working directory. No package installs, no compiler settings, no elevated privileges.

1. Confirm the installed verifier

Check the executable and version before copying a test into a project:

$ command -v FileCheck-18
/usr/bin/FileCheck-18
$ FileCheck-18 --version
Ubuntu LLVM version 18.1.3
  Optimized build.

• Input. The match file is the positional argument. The text being checked comes from standard input unless you pass --input-file.
• Exit status. A successful verification exits with 0. A failed match or any other error returns non-zero.

Checkpoint: keep FileCheck-18 in scripts when this exact major version matters. An unversioned FileCheck may refer to another LLVM installation.

2. Write an ordered check file

FileCheck looks for directives in the match file. The default prefix is CHECK:. A normal check can match text anywhere on a line, but successive checks must be found in order. Create this harmless example:

$ cat > output.txt <<'EOF'
begin: compile
instruction: add
instruction: ret
end: compile
EOF
$ cat > checks.txt <<'EOF'
; CHECK: begin: compile
; CHECK: instruction: add
; CHECK: instruction: ret
; CHECK: end: compile
EOF

The semicolon is ordinary text. FileCheck searches each line for the default directive, so this layout suits LLVM source files where semicolons start comments. The patterns themselves are fixed strings, apart from the substitution blocks in step 4.

Run the verifier by piping the candidate output into it:

$ FileCheck-18 checks.txt < output.txt
$ printf 'exit status: %s\n' "$?"
exit status: 0

Tip: there is nothing to undo. Both files are disposable, and if you point FileCheck at existing files it only reads them.

3. Make adjacency and absence explicit

Use CHECK-NEXT: when the next match must be on the very next line. Use CHECK-NOT: between two positive checks when a string must not appear in that range:

$ cat > checks-strict.txt <<'EOF'
; CHECK: begin: compile
; CHECK-NEXT: instruction: add
; CHECK-NOT: warning:
; CHECK-NEXT: instruction: ret
; CHECK-NEXT: end: compile
EOF
$ FileCheck-18 checks-strict.txt < output.txt
$ printf 'exit status: %s\n' "$?"
exit status: 0

• CHECK-NEXT: cannot be first. It requires exactly one newline after the previous match, so it catches an inserted line.
• CHECK-NOT: covers a region. That is the space between its neighbouring positive checks, including the area before the first or after the last positive check where applicable.
• CHECK-EMPTY: matches a blank line.
• CHECK-SAME: stays on the current line.

Tip: these directives are stricter than a plain CHECK:. Use them only when that relationship is part of the output contract.

4. Allow a controlled variable value

Literal checks turn brittle the moment a compiler picks a different register or numeric identifier. A double-bracket block captures a regular expression and lets you reuse the captured text:

$ cat > variable-output.txt <<'EOF'
function: add
register: r7
use: r7
EOF
$ cat > variable-check.txt <<'EOF'
; CHECK: function: add
; CHECK: register: [[REG:r[0-9]+]]
; CHECK: use: [[REG]]
EOF
$ FileCheck-18 variable-check.txt < variable-output.txt
$ printf 'exit status: %s\n' "$?"
exit status: 0

The expression r[0-9]+ is regular-expression syntax. The name REG stores whatever it matched, and the later [[REG]] demands the same text. That checks the relationship between the two lines without guessing the register number.

Warning: do not make expressions broader than the contract. A pattern such as .* can hide a broken result. FileCheck also supports single-brace regular-expression blocks and numeric variables, but reach for those only when the output genuinely needs them.

5. Use prefixes for more than one configuration

A single source file can hold checks for different runs. Put the chosen prefix before each directive, then select it with --check-prefix:

$ cat > modes.txt <<'EOF'
mode: fast
; FAST: mode: fast
; SAFE: mode: safe
EOF
$ FileCheck-18 --check-prefix=FAST modes.txt < modes.txt
$ printf 'exit status: %s\n' "$?"
exit status: 0

• No trailing colon. The prefix argument leaves it off.
• Several prefixes. Pass a comma-separated list to --check-prefixes.
• Unused prefixes are rejected. By default FileCheck fails a selected prefix that has no directives in the match file. --allow-unused-prefixes changes that, so add it only when an optional configuration is deliberate.
• Unordered output. Consecutive CHECK-DAG: directives match within a region in any relative order. Separate DAG groups with an ordinary check when order between groups matters.

Warning: be careful with captured variables in a DAG group. A definition can match later than its use, which may let a weak test pass.

6. Tighten whitespace and diagnose failures

• Whitespace. By default FileCheck treats horizontal spaces and tabs as equivalent. Use --strict-whitespace when indentation or exact spacing is part of the interface.
• Whole lines. Use --match-full-lines when a positive pattern must cover the whole line. Leading and trailing whitespace is still relaxed unless strict whitespace is also enabled.
• Case. Matching is case-sensitive by default. --ignore-case changes that, but it can turn a useful check into a false pass when capitalisation carries meaning. Keep the default unless the producer's case is deliberately irrelevant.
• Failures. FileCheck writes a diagnostic to standard error and, by default, dumps the relevant input with annotations. Re-run with -v for successful directive matches, or -vv for extras such as discarded overlapping DAG matches. These options explain a failure; they do not make a failing test pass.

Recovery: the usual culprits are checking two values without anchoring them to the same block, using CHECK-NEXT: after a directive that is not actually adjacent, and relying on whitespace the default mode ignores. Start with distinctive labels, then add the smallest constraint that expresses the behaviour you need.

Done means

• Right tool. FileCheck-18 --version identifies the expected LLVM 18 installation.
• Green path. A match file with ordered checks returns status 0 for the intended output.
• Red path. An altered or missing line returns non-zero and produces a useful diagnostic.
• No over-checking. Variable, adjacency, absence and whitespace checks appear only where the output contract requires them.
• Read-only. The test needs no sudo.