Build ICU Confusable Data with gencfu

You will finish with an ICU .cfu binary generated from a UTF-8 Unicode confusables rules file. The workflow uses the installed gencfu from ICU 74.2, checks the result without changing system data, and keeps generated files in a directory you control.

Allow about fifteen minutes. You need the icu-devtools package, a UTF-8 input file in the UAX #39 confusables format, and write access to an output directory. The examples do not need root. Do not replace a system ICU data file until the application owner has specified its search path and a rollback copy.

1. Confirm the installed tool

Check the executable and package version before copying an example. This is an ordinary read-only check:

$ command -v gencfu
/usr/bin/gencfu
$ dpkg-query -W -f='${Package} ${Version}\n' icu-devtools
icu-devtools 74.2-1ubuntu3.1

The local manpage is labelled "ICU 74.2 Manual" and documents -r for the rules file, -o for the output file, -d for its destination directory, and -i for ICU data. The installed help also lists -q, which suppresses warnings and progress. Options can vary between packaged builds, so keep this checkpoint with a deployment script.

One packaging wrinkle matters here: this installation advertises -V and --version in both the manpage and help text, but running either form exits with an argument error. Use the package query above as the reproducible version check for this host rather than treating that option as a successful probe.

2. Prepare a rules file

Use the Unicode source data intended for your application. The input is plain text in UTF-8, with or without a byte-order mark. The normal files are named confusables.txt and confusablesWholeScript.txt; they are source data, not already-built ICU binaries.

A mapping line contains hexadecimal code points separated by semicolons, followed by a status and an optional comment. This small file is enough for a smoke test, but it is not a production confusables table:

0061 ; 0062 ; MA # LATIN SMALL LETTER A -> LATIN SMALL LETTER B

Save it as $HOME/tmp-confusables.txt, or replace that placeholder with a path in your build workspace. Keep the source under version control if it is part of an application build. Do not hand-edit an upstream Unicode data file in place, because that makes later updates difficult to audit.

Checkpoint: verify that the file is present and recognised as UTF-8 text before generating anything:

$ file "$HOME/tmp-confusables.txt"
/home/you/tmp-confusables.txt: Unicode text, UTF-8 text

3. Generate the .cfu file

Choose an explicit output path. The short command below writes only the requested output file and does not require elevated privileges:

$ mkdir -p ./build
$ gencfu \
    --rules "$HOME/tmp-confusables.txt" \
    --out ./build/confusables.cfu
gencfu: tool completed successfully.

Exit status 0 and the completion message mean that this invocation accepted the rules and wrote the binary. The usual output extension is .cfu, but the tool accepts the filename you give to --out. Avoid writing directly into /usr/lib, /usr/share or an application installation tree while testing.

For a real build, the complete form is usually clearer than relying on the current directory:

$ gencfu --rules /path/to/confusables.txt \
    --out /path/to/build/confusables.cfu

If you want to keep the output filename separate from its directory, use --destdir with the output filename. Verify the exact result in your build environment before putting that form into automation, because an output path that already exists may be replaced.

4. Verify the generated data

Check the file type, size and command status. These checks are read-only:

$ stat -c '%n %s bytes' ./build/confusables.cfu
build/confusables.cfu 176 bytes
$ file ./build/confusables.cfu
build/confusables.cfu: data

The byte count is only an example from the one-line smoke test and will change with the rules. A non-empty file identified as generic data is consistent with the binary format, but it does not prove that your mappings are semantically correct. Load the file through the ICU spoof detector or the application component that will consume it, then run that component's own tests.

With the local build, --verbose still completes successfully:

$ gencfu --verbose --rules "$HOME/tmp-confusables.txt" \
    --out ./build/confusables-verbose.cfu
gencfu: tool completed successfully.

Use verbosity for a build log when you need it, not as a substitute for validating the resulting table. --copyright embeds the standard ICU copyright in the output file; add it only when your distribution policy requires that metadata.

5. Diagnose failures without guessing

A missing --rules or --out argument is a command-line error. A missing input file, malformed line or wrong encoding is an input problem. Recheck the path, read permission and UTF-8 encoding first:

$ test -r "$HOME/tmp-confusables.txt" && echo 'rules readable'
rules readable
$ gencfu --rules "$HOME/tmp-confusables.txt" --out ./build/check.cfu
$ printf 'exit status: %s\n' "$?"
exit status: 0

If ICU data cannot be found, use --icudatadir to point at the directory containing the required data, or set ICU_DATA as specified by the host deployment. Most shared-library installations do not need this option. Do not point it at an arbitrary directory and assume that it repairs a broken ICU installation.

The manpage also documents a separate whole-script rules input with --wsrules. Use it only when you have the corresponding Unicode whole-script source and your consumer expects that data. Keep ordinary confusables and whole-script inputs explicit in build commands so a future maintainer can see which data set was compiled.

6. Keep the change reversible

This guide writes a new build artefact. To undo it, remove the generated file using your normal build-clean command, or delete the specific file ./build/confusables.cfu after confirming that no process is using it. Do not delete an existing shared ICU data file as a shortcut for recovering from a bad build. Restore the previous version from the package or deployment artefact instead, and restart a service only if its documented data-loading behaviour requires it.