Test Compiler Output with FileCheck-18

Your compiler output changed by one register number and forty tests went red, so you need FileCheck-18 to check what matters and ignore what does not. By the end you will have a small test that verifies generated text in order, rejects an unwanted line, and tolerates values that legitimately vary.

The installed command is FileCheck-18 from the llvm-18 package, reporting LLVM version 18.1.3 on this machine. Allow about fifteen minutes.

• You need: a shell and the LLVM 18 tools package.
• It will not touch: anything outside your working directory. No package installs, no compiler settings, no elevated privileges.

1. Confirm the installed verifier

Check the executable and version before copying a test into a project:

$ command -v FileCheck-18
/usr/bin/FileCheck-18
$ FileCheck-18 --version
Ubuntu LLVM version 18.1.3
  Optimized build.

• Input. The match file is the positional argument. The text being checked comes from standard input unless you pass --input-file.
• Exit status. A successful verification exits with 0. A failed match or any other error returns non-zero.

2. Write an ordered check file

FileCheck looks for directives in the match file. The default prefix is CHECK:. A normal check can match text anywhere on a line, but successive checks must be found in order. Create this harmless example:

$ cat > output.txt <<'EOF'
begin: compile
instruction: add
instruction: ret
end: compile
EOF
$ cat > checks.txt <<'EOF'
; CHECK: begin: compile
; CHECK: instruction: add
; CHECK: instruction: ret
; CHECK: end: compile
EOF

The semicolon is ordinary text. FileCheck searches each line for the default directive, so this layout suits LLVM source files where semicolons start comments. The patterns themselves are fixed strings, apart from the substitution blocks in step 4.

Run the verifier by piping the candidate output into it:

$ FileCheck-18 checks.txt < output.txt
$ printf 'exit status: %s\n' "$?"
exit status: 0

3. Make adjacency and absence explicit

Use CHECK-NEXT: when the next match must be on the very next line. Use CHECK-NOT: between two positive checks when a string must not appear in that range:

$ cat > checks-strict.txt <<'EOF'
; CHECK: begin: compile
; CHECK-NEXT: instruction: add
; CHECK-NOT: warning:
; CHECK-NEXT: instruction: ret
; CHECK-NEXT: end: compile
EOF
$ FileCheck-18 checks-strict.txt < output.txt
$ printf 'exit status: %s\n' "$?"
exit status: 0

• CHECK-NEXT: cannot be first. It requires exactly one newline after the previous match, so it catches an inserted line.
• CHECK-NOT: covers a region. That is the space between its neighbouring positive checks, including the area before the first or after the last positive check where applicable.
• CHECK-EMPTY: matches a blank line.
• CHECK-SAME: stays on the current line.

4. Allow a controlled variable value

Literal checks turn brittle the moment a compiler picks a different register or numeric identifier. A double-bracket block captures a regular expression and lets you reuse the captured text:

$ cat > variable-output.txt <<'EOF'
function: add
register: r7
use: r7
EOF
$ cat > variable-check.txt <<'EOF'
; CHECK: function: add
; CHECK: register: [[REG:r[0-9]+]]
; CHECK: use: [[REG]]
EOF
$ FileCheck-18 variable-check.txt < variable-output.txt
$ printf 'exit status: %s\n' "$?"
exit status: 0

The expression r[0-9]+ is regular-expression syntax. The name REG stores whatever it matched, and the later [[REG]] demands the same text. That checks the relationship between the two lines without guessing the register number.

5. Use prefixes for more than one configuration

A single source file can hold checks for different runs. Put the chosen prefix before each directive, then select it with --check-prefix:

$ cat > modes.txt <<'EOF'
mode: fast
; FAST: mode: fast
; SAFE: mode: safe
EOF
$ FileCheck-18 --check-prefix=FAST modes.txt < modes.txt
$ printf 'exit status: %s\n' "$?"
exit status: 0

• No trailing colon. The prefix argument leaves it off.
• Several prefixes. Pass a comma-separated list to --check-prefixes.
• Unused prefixes are rejected. By default FileCheck fails a selected prefix that has no directives in the match file. --allow-unused-prefixes changes that, so add it only when an optional configuration is deliberate.
• Unordered output. Consecutive CHECK-DAG: directives match within a region in any relative order. Separate DAG groups with an ordinary check when order between groups matters.

6. Tighten whitespace and diagnose failures

• Whitespace. By default FileCheck treats horizontal spaces and tabs as equivalent. Use --strict-whitespace when indentation or exact spacing is part of the interface.
• Whole lines. Use --match-full-lines when a positive pattern must cover the whole line. Leading and trailing whitespace is still relaxed unless strict whitespace is also enabled.
• Case. Matching is case-sensitive by default. --ignore-case changes that, but it can turn a useful check into a false pass when capitalisation carries meaning. Keep the default unless the producer's case is deliberately irrelevant.
• Failures. FileCheck writes a diagnostic to standard error and, by default, dumps the relevant input with annotations. Re-run with -v for successful directive matches, or -vv for extras such as discarded overlapping DAG matches. These options explain a failure; they do not make a failing test pass.