Run a Small lit Test Suite and Diagnose Failures

You will create a minimal LLVM lit test suite, run it, select one test, and make a failure readable enough to fix. Allow about 15 minutes if Python 3 and the LLVM 20 tools are already installed. The examples use only a temporary directory and do not need root privileges.

There is one packaging wrinkle on this machine: the lit-20(1) manual is installed with the llvm-20 package, but no lit-20 command is on the normal PATH. The runnable entry point is /usr/lib/llvm-20/build/utils/lit/lit.py. Set a shell variable once and use it for the rest of the guide.

1. Check the installed entry point

Confirm the package version and the executable before building examples. The package version and lit's own version are separate pieces of information, so record both when reporting a test result.

$ dpkg-query -W -f='${Package} ${Version}\n' llvm-20 llvm-20-tools
llvm-20 1:20.1.8~++20250804090239+87f0227cb601-1~exp1~20250804210352.139
llvm-20-tools 1:20.1.8~++20250804090239+87f0227cb601-1~exp1~20250804210352.139
$ LIT20=/usr/lib/llvm-20/build/utils/lit/lit.py
$ test -f "$LIT20" && python3 "$LIT20" --version
lit 20.1.0dev

Your package revision or version string may differ. If the file check fails, stop and locate the lit entry point supplied by your distribution rather than guessing a path or installing another copy over it.

2. Create a test suite marker

lit does not treat an arbitrary directory as a test suite. It searches upwards from each input until it finds lit.cfg or lit.site.cfg. The configuration names the suite, selects a test format, and tells lit where the source and execution roots are.

Create a disposable suite under /tmp. The configuration below uses the shell-test format, where commands are written in comments beginning with RUN:.

$ mkdir -p /tmp/lit20-guide/suite
$ cd /tmp/lit20-guide/suite
$ touch lit.cfg pass.sh
$ chmod +x pass.sh

Edit lit.cfg so it contains:

import lit.formats

config.name = "lit20-guide"
config.test_format = lit.formats.ShTest(not lit_config.useValgrind)
config.suffixes = [".sh"]
config.test_source_root = "/tmp/lit20-guide/suite"
config.test_exec_root = "/tmp/lit20-guide/suite"

Put this single line in pass.sh:

# RUN: true

The absolute paths make this smoke test predictable. In a real project, derive them from the build layout instead of copying these temporary paths.

3. Discover and run the test

Ask lit what it discovered before executing anything. This catches a missing configuration file, an incorrect suffix, or a path outside the suite.

$ python3 "$LIT20" --show-suites /tmp/lit20-guide/suite
-- Test Suites --
  lit20-guide - 1 tests
    Source Root: /tmp/lit20-guide/suite
    Exec Root  : /tmp/lit20-guide/suite
$ python3 "$LIT20" --show-tests /tmp/lit20-guide/suite
-- Available Tests --
  lit20-guide :: pass.sh

Run the suite in succinct mode:

$ python3 "$LIT20" -q /tmp/lit20-guide/suite

Total Discovered Tests: 1
$ printf 'exit status: %s\n' "$?"
exit status: 0

The default worker count is chosen from the available CPUs, so even a one-test run may use the normal concurrency machinery. Use -j 1 when you want a single worker while investigating order-dependent behaviour.

Checkpoint: the suite is recognised

Do not move on until --show-tests lists pass.sh and the run exits with status 0. If discovery reports no tests, inspect the spelling of config.suffixes, the location of lit.cfg, and whether the input path is the suite directory.

4. Select a subset without editing the suite

For a larger suite, pass a test file or use a regular-expression filter. A filter matches test names, including the suite name and relative path. This command runs only the test selected by its name:

$ python3 "$LIT20" --filter='pass\.sh' /tmp/lit20-guide/suite

-- Testing: 1 tests, 1 workers --
PASS: lit20-guide :: pass.sh (1 of 1)

Testing Time: 0.22s

Total Discovered Tests: 1
  Passed: 1 (100.00%)

In a script, quote the regular expression. An unquoted expression can be altered by shell expansion, and a broad expression can select far more tests than intended. --filter-out=REGEXP removes matching tests instead.

If filtering removes everything, lit normally treats that as an error. Add --allow-empty-runs only when an empty selection is an expected result of your workflow. It hides a useful signal during ordinary debugging.

5. Make a failure diagnostic

Change the test command to a deliberate failure:

# RUN: false

Run it verbosely:

$ python3 "$LIT20" -v /tmp/lit20-guide/suite
FAIL: lit20-guide :: pass.sh (1 of 1)
******************** TEST 'lit20-guide :: pass.sh' FAILED ********************
Test 'lit20-guide :: pass.sh' failed as a result of exit code 1.
********************
Total Discovered Tests: 1

The exact timing and progress text vary. The useful facts are the test name, the failing command, and the non-zero exit status. With -v, lit prints commands before execution and adds a RUN: at line N marker, which connects a failure to its source line. The normal exit status for a run containing FAIL or XPASS is 1.

Restore # RUN: true before continuing. If the test command created files or changed a shared build directory, remove only those known temporary outputs after checking the path. Do not use a broad recursive deletion command in a real build tree.

6. Control time, order and reports

--timeout=N limits each individual test. It defaults to 0, meaning no per-test limit. --max-time=N limits the approximate duration of the whole run, while --max-tests=N limits its test count. These are different limits, so choose the one matching the failure you are containing.

The default smart order prioritises tests that failed previously and then favours longer tests. Use --order=lexical for repeatable path order or --order=random to expose order dependencies. Timing data is stored in .lit_test_times.txt under the execution root; treat it as generated state, not a test result.

For CI or another tool, write a machine-readable report with an explicitly chosen destination:

$ python3 "$LIT20" --xunit-xml-output=/tmp/lit20-guide/results.xml /tmp/lit20-guide/suite
$ test -s /tmp/lit20-guide/results.xml && echo report-written
report-written

Do not overwrite a report that another job is still reading. Give concurrent runs separate output paths.

7. Remove the temporary example

This guide has changed only files below /tmp/lit20-guide. Once you have checked the output, remove that exact temporary directory if it is no longer useful. That removal is irreversible, so verify the path first:

$ test "/tmp/lit20-guide" = /tmp/lit20-guide && test -d /tmp/lit20-guide
$ rm -rf -- /tmp/lit20-guide

Do not adapt the final command to a source or build directory. If you want to keep the suite, omit the removal and replace its absolute paths with paths appropriate to the project.