Compare LLVM Code Generation with llvm-remarkutil-18

llvm-remarkutil-18 turns Clang's optimisation remarks into CSV instruction counts and a size comparison between two builds. The examples use Ubuntu package llvm-18, version 1:18.1.3-1ubuntu1. That matters because the installed command uses LLVM 18's option spelling and output format.

Allow 15 to 20 minutes. You need a shell, clang-18, and a small C or C++ source file. The workflow reads and creates report files only: it does not install software, rewrite source, or replace a binary.

1. Confirm the installed tool

Check the executable and package version before interpreting a report:

$ command -v llvm-remarkutil-18
/usr/bin/llvm-remarkutil-18
$ llvm-remarkutil-18 --version
Ubuntu LLVM version 18.1.3
  Optimized build.

The utility has six useful subcommands: yaml2bitstream, bitstream2yaml, instruction-count, annotation-count, count, and size-diff. Start with the subcommand help when adapting an example:

$ llvm-remarkutil-18 size-diff --help

2. Generate YAML remarks for each build

llvm-remarkutil-18 analyses remark files; it does not normally create them from source. Ask Clang to save optimisation remarks while compiling the same source with two configurations:

$ clang-18 -O0 -g -c /path/to/sample.c \
    -o /tmp/sample-o0.o \
    -fsave-optimization-record \
    -foptimization-record-file=/tmp/sample-o0.yaml
$ clang-18 -O2 -g -c /path/to/sample.c \
    -o /tmp/sample-o2.o \
    -fsave-optimization-record \
    -foptimization-record-file=/tmp/sample-o2.yaml

Use one fixed source file and change only the compiler setting you want to study. The -g option gives remarks source locations, which helps with reports that include debug locations. Confirm both inputs exist and are non-empty:

$ test -s /tmp/sample-o0.yaml && test -s /tmp/sample-o2.yaml && echo 'remark files ready'
remark files ready

Keep these files as evidence for the comparison. If compilation fails, fix that failure first. Do not create an empty placeholder report, because a later command may appear to work while saying nothing useful.

3. Count instructions by function

Use instruction-count with the required parser selection. It writes CSV with one row per function:

$ llvm-remarkutil-18 instruction-count \
    /tmp/sample-o2.yaml \
    --parser=yaml \
    -o /tmp/sample-o2-instructions.csv
$ sed -n '1,5p' /tmp/sample-o2-instructions.csv
Function,InstructionCount
add,2
caller,2

The exact function names and counts depend on your source and target. The useful invariant is the header and one count per function represented by the compiler's instruction-count remarks. The feature needs asm-printer remarks, so an arbitrary optimisation record may have no rows at all.

Add --use-debug-loc when source location is more useful than a function-only report:

$ llvm-remarkutil-18 instruction-count \
    /tmp/sample-o2.yaml --parser=yaml --use-debug-loc \
    -o /tmp/sample-o2-instructions-with-loc.csv

4. Count remarks for a quick inventory

count can group all parsed remarks by function or source. For a single total, use --group-by=total:

$ llvm-remarkutil-18 count \
    --parser=yaml --group-by=total /tmp/sample-o2.yaml
Total,Count

With a real report, the second line carries the total. To see where remarks came from, swap total for function or source. Source grouping needs debug locations. Filters such as --pass-name, --remark-name, and --remark-type narrow the input before counting; add them one at a time so an empty result stays explainable.

5. Compare two reports with size-diff

Give size-diff the older or baseline report first and the comparison report second. It compares instruction counts and stack byte usage for functions present in both reports:

$ llvm-remarkutil-18 size-diff \
    /tmp/sample-o0.yaml /tmp/sample-o2.yaml \
    --parser=yaml --report_style=human
== < caller, -8 instrs, -24 stack B
== < add, -6 instrs, -8 stack B

### Summary ###
Total change:
  instruction count: -14 (-77.78%)
  stack byte usage: -32 (-100.00%)

Here == means the function is in both reports and < means the second file has fewer instructions. Positive or negative results are measurements, not recommendations: check the same target, calling convention, source, and relevant compiler flags before pinning a change on one optimisation.

In the installed LLVM 18 build, the tested option is spelled --report_style with an underscore. The local manpage shows a hyphenated spelling in prose, so trust the spelling printed by llvm-remarkutil-18 size-diff --help on this machine over the prose.

6. Save machine-readable comparison data

Use JSON when another script will consume the result. Add --pretty only for human inspection:

$ llvm-remarkutil-18 size-diff \
    /tmp/sample-o0.yaml /tmp/sample-o2.yaml \
    --parser=yaml --report_style=json --pretty \
    -o /tmp/sample-size-diff.json
$ sed -n '1,24p' /tmp/sample-size-diff.json
{
  "Files": {
    "A": "/tmp/sample-o0.yaml",
    "B": "/tmp/sample-o2.yaml"
  },
  "InBoth": [

JSON records the two input paths and per-function values in InstCount and StackSize. It does not calculate the per-function differences for you: for each array, subtract the value for A from the value for B. The report's summary is still the easiest place to read aggregate changes.

7. Convert between YAML and bitstream

Use the conversion commands when a consumer needs the other remark format:

$ llvm-remarkutil-18 yaml2bitstream \
    /tmp/sample-o2.yaml -o /tmp/sample-o2.bc
$ llvm-remarkutil-18 bitstream2yaml \
    /tmp/sample-o2.bc -o /tmp/sample-o2-roundtrip.yaml
$ test -s /tmp/sample-o2-roundtrip.yaml && echo 'round-trip report ready'
round-trip report ready

These commands reserialise remarks. They do not optimise code, validate that two reports are comparable, or repair a malformed input. Keep the original YAML until the converted file has been read by the next tool.

Common traps and recovery

• Parser omitted: instruction-count, annotation-count, count, and size-diff require --parser=yaml or --parser=bitstream. Add it explicitly in scripts.
• Empty CSV: confirm the input contains the required asm-printer or annotation remarks. Regenerate with the relevant Clang remark settings rather than treating an empty report as zero.
• Unrelated inputs: compare builds of the same source and target. Otherwise the numerical difference mixes code changes with compiler changes.
• Overwritten output: if a report path was wrong, stop and restore it from your backup or version control. Use a new temporary path for the next run; redirection and -o give you no undo.