Run and Debug LLVM Test Suites with lit-18

A CI job reports 12 failures and gives you nothing else to go on: lit is the tool that turns that into one reproducible command per test. This guide runs a whole suite, narrows a failure down to a single test, and tells apart a real failure from a broken discovery step. Allow 15 to 30 minutes for a first run, depending on suite size.

The local reference is the lit-18(1) manual from package llvm-18, version 18.1.3-1ubuntu1. This machine has the manpage but no lit executable on PATH, so the examples below follow the documented output format: check them against the binary your own LLVM test package or build actually supplies.

1. Find the executable and the suite marker

Start in the project that owns the tests. The executable is normally called lit; use whatever path the project provides if it isn't on PATH. A suite is identified by a lit.cfg or lit.site.cfg file: Python configuration modules, not shell scripts, that decide how tests get discovered and run.

$ command -v lit
/path/to/llvm/bin/lit
$ find /path/to/project -name 'lit.cfg' -o -name 'lit.site.cfg'
/path/to/project/tests/lit.cfg

If command -v comes back empty, stop and install or expose the test runner through your normal toolchain. Installing the llvm-18 runtime package does not automatically bring the separate test runner with it. Running or inspecting tests in a writable build tree needs no elevated privilege.

Checkpoint: you should have one executable path and a directory containing one of the recognised config file names. Point lit at a random directory with no suite config and you'll get a discovery error, not a useful run.

2. Run the whole suite

Pass the suite directory as input. lit searches the configured source tree recursively and may run tests concurrently, with a default worker count picked from the available CPUs.

$ lit /path/to/project/tests
Testing Time: 3.42s
  Passed: 148

The exact summary depends on test format and suite size. Each result in the standard stream has the shape PASS: test-name (progress). A run exits 0 only when there are no FAIL or XPASS results, so capture the status immediately if a script depends on it:

$ lit /path/to/project/tests
$ status=$?
$ printf 'lit exit status: %s\n' "$status"
lit exit status: 0

Don't treat a short summary as proof every test ran. Unsupported tests can be hidden by the normal display, and configuration failures use their own non-zero statuses. If the count looks off, reach for the discovery checks below.

3. Check discovery before touching test commands

Ask lit what it found before assuming anything's wrong with the tests themselves. Both of these are read-only and catch a wrong directory, a nested suite, or a config file that never loaded:

$ lit --show-suites /path/to/project/tests
$ lit --show-tests /path/to/project/tests
suite-name :: path/to/test-file.test

You can pass individual test files as well as directories. A file resolves upwards to the nearest applicable suite, then gets identified by suite name and path within that suite, which is why a failure report might show a name that isn't an absolute filesystem path.

If the suite uses a name other than lit, pass the matching prefix:

$ lit --config-prefix=project /path/to/project/tests

That makes project.cfg and project.site.cfg the names lit searches for on this invocation. Check the files actually exist before blaming the tests.

4. Rerun one failing test with full diagnostics

Once you have a failing test's path, pass that file or its containing subdirectory, and add --verbose so lit prints each command before running it and shows the complete failure output. The last printed command is the one to reproduce by hand first.

$ lit --verbose /path/to/project/tests/subdir/failing.test
FAIL: suite-name :: subdir/failing.test (1 of 1)
******************** TEST 'suite-name :: subdir/failing.test' FAILED ********************
RUN: at line 1
... command printed by lit ...
********************

Use --filter=REGEXP for a name-based subset, or --filter-out=REGEXP to exclude a noisy group. Keep the expression narrow: a broad filter can leave you with a run that looks green but tested almost nothing.

$ lit --filter='parser.*invalid' --verbose /path/to/project/tests
$ lit --filter-out='slow|gpu' /path/to/project/tests

The exit status still means something at this point too: FAIL and XPASS both make lit return 1. Treat UNRESOLVED, a timeout, or a missing executable as an investigation failure, not something to wave away.

5. Control time and parallelism

Use -j N or --workers=N to set the number of parallel workers. Leave the default alone unless the host is contended or the tests fight over a shared resource, and drop to -j 1 when you're diagnosing ordering or a resource race.

$ lit -j 1 --verbose /path/to/project/tests/subdir
$ lit --workers=4 /path/to/project/tests

There are two separate time limits, and mixing them up in automation causes confusing failures. --timeout=N caps each individual test; --max-time=N caps the approximate time for the whole run. A timeout of 0, the default, means no per-test limit at all.

$ lit --timeout=120 /path/to/project/tests
$ lit --max-time=600 /path/to/project/tests

--max-failures=N stops after a number of failures, and --max-tests=N stops after a number of selected tests. Both are handy for a quick feedback loop; neither counts as a full validation run.

6. Handle parameters and environment defaults with care

Suite configuration decides what a custom parameter actually means, so only pass -D NAME=VALUE or --param NAME=VALUE for a name the suite documents. An omitted value becomes an empty string.

$ lit -D TARGET=x86_64 --show-tests /path/to/project/tests

LIT_OPTS is read after the command-line options, so it can override whatever a build target added. That's convenient in a controlled CI environment and confusing the moment you're investigating something locally. If results don't match the command you're looking at, check and temporarily clear it:

$ printf '%s\n' "${LIT_OPTS-}"
$ env -u LIT_OPTS lit --show-suites /path/to/project/tests

Response files are another place hidden arguments hide. An input like @/path/to/options.rsp reads one argument per line and can include other response files in turn, so review an unfamiliar one before running it.

7. Learn the status names before you touch expectations

PASS is success. FAIL is a failed test. XFAIL is an expected failure. XPASS means a test expected to fail now succeeds, and it still counts as a failing result, which surprises people the first time they see it. UNSUPPORTED means the test format says the environment can't run it at all. UNRESOLVED means lit couldn't determine a result, and TIMEOUT means the test blew its individual limit.

Do not add an XFAIL purely to get a green build. Use --xfail=LIST only when the project has a documented reason to classify specific tests that way, and record it alongside the underlying defect in the same review. LIT_XFAIL gives the same semicolon-separated syntax for an indirect invocation, and --xfail-not overrides other XFAIL specs for named tests.

8. Keep safety boundaries clear

An ordinary lit run executes whatever commands the test files define. Read the suite configuration and test files before running an untrusted checkout: tests can compile code, start processes, write into the execution tree or touch anything your account can reach. Safety boundary: never run an unfamiliar suite as root; use a disposable account or an isolated build environment when you don't trust the source.

--vg runs individual tests under Valgrind memcheck, --vg-arg=ARG passes it an argument, and --vg-leak turns on leak checks. All three can make a run much slower and shift timing, so use them for diagnosis rather than as a stand-in for the normal suite result.

lit writes timing data to .lit_test_times.txt in the execution root and uses it to order later runs. That file is disposable run metadata: remove it only after confirming the exact path, and accept that the next run starts with no history.

Done means