Build ICU Confusable Data with gencfu

gencfu compiles a UTF-8 Unicode confusables rules file into a binary .cfu file that ICU's spoof checker can load. This walkthrough uses the installed gencfu from ICU 74.2, checks the result without touching any system data, and keeps everything generated inside a directory you control.

Give it about fifteen minutes. You need the icu-devtools package, a UTF-8 input file in the UAX #39 confusables format, and write access to an output directory. None of this needs root. Do not replace a system ICU data file until the application owner has told you its search path and given you a rollback copy.

1. Confirm the installed tool

Check the executable and package version before you copy any example. This is a plain read-only check:

$ command -v gencfu
/usr/bin/gencfu
$ dpkg-query -W -f='${Package} ${Version}\n' icu-devtools
icu-devtools 74.2-1ubuntu3.1

The local manpage, labelled "ICU 74.2 Manual", documents -r for the rules file, -o for the output, -d for its destination directory, and -i for ICU data. The installed help also lists -q, which suppresses warnings and progress. Options can shift between packaged builds, so keep this checkpoint next to any deployment script.

One packaging wrinkle catches people out here: this install advertises -V and --version in both the manpage and the help text, but running either one exits with an argument error. Treat the package query above as the reproducible version check for this host rather than that option.

2. Prepare a rules file

Use the Unicode source data your application actually needs. The input is plain UTF-8 text, with or without a byte-order mark. The normal filenames are confusables.txt and confusablesWholeScript.txt: source data, not already-built ICU binaries.

A mapping line holds hexadecimal code points separated by semicolons, then a status and an optional comment. This tiny file is enough for a smoke test, not a production confusables table:

0061 ; 0062 ; MA # LATIN SMALL LETTER A -> LATIN SMALL LETTER B

Save it as $HOME/tmp-confusables.txt, or swap in a path from your own build workspace. Keep the source under version control if it belongs to an application build, and do not hand-edit an upstream Unicode data file in place: that makes future updates hard to audit.

$ file "$HOME/tmp-confusables.txt"
/home/you/tmp-confusables.txt: Unicode text, UTF-8 text

3. Generate the .cfu file

Choose an explicit output path. This short command writes only the file you asked for and needs no elevated privileges:

$ mkdir -p ./build
$ gencfu \
    --rules "$HOME/tmp-confusables.txt" \
    --out ./build/confusables.cfu
gencfu: tool completed successfully.

Exit status 0 plus that completion message means the rules were accepted and the binary was written. .cfu is the usual extension, but gencfu accepts whatever filename you give --out. Avoid writing straight into /usr/lib, /usr/share or an application's install tree while you are testing.

$ gencfu --rules /path/to/confusables.txt \
    --out /path/to/build/confusables.cfu

For a real build, spelling out the full path like that is usually clearer than relying on the current directory. If you want the filename kept separate from its location, use --destdir alongside the output name, and check the exact behaviour in your build environment before automating it: a path that already exists may just get replaced.

4. Verify the generated data

Check the file type, size and command status. These are all read-only:

$ stat -c '%n %s bytes' ./build/confusables.cfu
build/confusables.cfu 176 bytes
$ file ./build/confusables.cfu
build/confusables.cfu: data

That byte count is just an example from the one-line smoke test and will change with your rules. A non-empty file identified as generic data fits the binary format, but it does not prove your mappings are semantically correct. Load it through the ICU spoof detector, or the component that will actually consume it, and run that component's own tests.

$ gencfu --verbose --rules "$HOME/tmp-confusables.txt" \
    --out ./build/confusables-verbose.cfu
gencfu: tool completed successfully.

Use --verbose for a build log when you need one, not as a substitute for validating the resulting table. --copyright embeds ICU's standard copyright text in the output; add it only when your distribution policy actually asks for that metadata.

5. Diagnose failures without guessing

A missing --rules or --out argument is a command-line error. A missing input file, a malformed line, or the wrong encoding is an input problem instead. Recheck the path, read permission and UTF-8 encoding first:

$ test -r "$HOME/tmp-confusables.txt" && echo 'rules readable'
rules readable
$ gencfu --rules "$HOME/tmp-confusables.txt" --out ./build/check.cfu
$ printf 'exit status: %s\n' "$?"
exit status: 0

If ICU data cannot be found, use --icudatadir to point at the directory holding what it needs, or set ICU_DATA as your deployment specifies. Most shared-library installs never need this. Do not point it at some arbitrary directory and hope that repairs a broken ICU install.

The manpage also documents a separate whole-script rules input with --wsrules. Use it only when you have the matching Unicode whole-script source and your consumer actually expects that data. Keep both ordinary confusables and whole-script inputs explicit in build commands, so whoever maintains this next can see which data set was compiled.

6. Keep the change reversible

This guide only writes a new build artefact. To undo it, run your normal build-clean command, or delete the specific file ./build/confusables.cfu once you have confirmed nothing is using it. Do not delete an existing shared ICU data file as a shortcut for recovering from a bad build: restore the previous version from the package or deployment artefact instead, and only restart a service if its documented data-loading behaviour actually requires it.