Package Data Files with gencmn

gencmn bundles a set of local files into one ICU common data file, such as demo.dat, and tells you exactly which inputs made it in. Handy when an ICU-based program wants related data as one memory-mappable package instead of a pile of separate files.

Give it about ten minutes for a first build. You need the gencmn executable from icu-devtools, a writable working directory, and the input files that belong in the package. Ordinary user permissions cover it: you only need anything more if the chosen destination is itself protected, and a private build directory is easier to inspect and recover than a shared one.

Before you start

This guide was checked against ICU 74.2 from Ubuntu package icu-devtools 74.2-1ubuntu3.1. Check your own install before trusting the default data name, because that name bakes in the ICU release number and your machine's endianness:

command -v gencmn
gencmn --help
gencmn --version

The local 74.2 build rejects --version and just prints its usage text; there is no separate version switch. The installed manpage identifies the tool as version 74.2 and shows a default name of icudt74l on this little-endian machine. gencmn --help is the authoritative check if another ICU release is installed on your box.

1. Create a dedicated build directory

Keep the list, source files and generated output together while testing. It stops you accidentally packaging an absolute path, or overwriting a system ICU data file:

mkdir -p "$HOME/icu-package-demo/input" "$HOME/icu-package-demo/out"
cd "$HOME/icu-package-demo"
printf 'first payload for ICU\n' > input/alpha.bin
printf 'second payload for ICU\n' > input/beta.bin

Both files here run longer than 20 bytes, which matters: ICU 74.2 rejects any file of 20 bytes or less when building a binary common data file. Do not test a "successful" build with a tiny empty fixture, because it will not be one.

2. Write a relative input list

Put one relative pathname per line, resolved from wherever gencmn runs. A comment line can start with #:

cat > files.lst <<'EOF'
# Files to package
input/alpha.bin
input/beta.bin
EOF
sed -n '1,10p' files.lst

Expect to see the two paths, preceded by the comment. Here is the easy trap: ICU 74.2 rejects an absolute entry such as /tmp/alpha.bin with an "absolute path encountered" error, so change into the project directory first and list paths below it.

3. Generate and inspect the package

Use 0 as maxsize when every listed file should qualify, name the output explicitly, and turn on verbose output while you are still learning what this produces:

gencmn --verbose \
  --destdir out \
  --name demo \
  --type dat \
  0 files.lst

Expect output close to this:

generating demo.dat (common data file with table of contents)
adding ./input/alpha.bin (22 bytes)
adding ./input/beta.bin (23 bytes)

Exact byte counts depend on your payloads. The generated file lands at out/demo.dat. Check it exists and holds something:

test -s out/demo.dat && stat -c '%n %s bytes' out/demo.dat

What you get is an ICU common data package, not a general-purpose archive: ICU uses its own table of contents to find the packaged entries. Do not try to unpack it with tar, or treat it like a normal filesystem directory.

4. Control large inputs with a size limit

A positive maxsize is a per-file limit in bytes, not a cap on the final package. Files bigger than the limit just get left out; a file exactly at the limit is still accepted:

gencmn --verbose \
  --destdir out \
  --name limited \
  --type dat \
  24 files.lst

With a 32-byte alpha.bin and a 24-byte beta.bin, the useful part of the output looks like this:

./input/alpha.bin ignored (size 32 > 24)
adding ./input/beta.bin (24 bytes)

Treat this as a deliberate filter, not as silent error handling. A clean exit does not mean every file you asked for got included, so read the verbose output, or compare the list against what you expected.

5. Feed the list through standard input

The list filename is optional. Leave it out and gencmn reads standard input instead, so a carefully built file list can be piped straight in:

printf '%s\n' input/alpha.bin input/beta.bin | \
  gencmn --verbose --destdir out --name piped --type dat 0

This produces out/piped.dat. Quote or otherwise constrain any generated pathnames before piping them into a packaging command like this. The list format is whitespace-sensitive: gencmn takes the first space-delimited item on a line, so unescaped filenames containing spaces will not survive.

6. Choose the destination on purpose

--destdir picks the output directory. Leave it out and ICU falls back to ICU_DATA; if that is unset too, this 74.2 install defaults to /usr/share/icu/74.2/. Some ICU tools expect a trailing slash on ICU_DATA when it is set:

ICU_DATA="$HOME/icu-package-demo/out/" \
  gencmn --verbose --name envdemo --type dat 0 files.lst
test -s "$HOME/icu-package-demo/out/envdemo.dat"

Common failure points

• Absolute path error. Replace an entry such as /home/me/data/foo with a path relative to the working directory.
• Unable to get length. Check that each file exists, is readable, and is larger than 20 bytes for a binary package on this ICU release.
• No files listed. Look for blank input, stray comments, or a typo in the list filename: that is not the same as a genuinely empty package.
• Unexpected output name. Remember --name supplies the base name and --type supplies the extension; leave both out and you get an ICU release-and-endianness name with a .dat type.
• Missing data later. Rerun with --verbose and confirm every intended file was added. A positive size limit can quietly drop inputs without failing the whole command.