Build and Test a Perl Encoding with enc2xs

enc2xs turns a Unicode Character Mapping file into a real Perl Encode extension, built with MakeMaker and checked by its own generated test. The workflow uses enc2xs from Perl 5.38.2, supplied here by the Ubuntu perl package version 5.38.2-3.2ubuntu0.6.

Allow about twenty minutes for a first run. You need Perl, enc2xs, a C compiler and make. Work in a disposable directory while learning; the example creates files there and never installs anything system-wide.

Checkpoint: stop once the generated test passes. Installing the module, or changing Encode's local demand-loading configuration, are separate, privileged or persistent actions covered later.

1. Confirm the local tool

Check the executable and its option summary before preparing a map. These are ordinary, read-only commands that need no elevated privileges:

$ command -v enc2xs
/usr/bin/enc2xs
$ dpkg-query -W -f='${Package} ${Version}\n' perl
perl 5.38.2-3.2ubuntu0.6
$ enc2xs -v 2>&1 | sed -n '1,12p'
/usr/bin/enc2xs version 2.24 calling Getopt::Std::getopts (version 1.13 [paranoid]),
running under Perl version 5.38.2.

ARGV:
opt:  q => undef
  v => 1

The important generation form is enc2xs -M ModuleName mapfiles.... The installed help lists -M, -o, -f and -n as options with arguments, and -C, -S, -Q, -q, -O and -v as boolean options. This guide only needs -M.

2. Create a small UCM file

A UCM file has a header, a CHARMAP section and an END CHARMAP marker. The map below is deliberately tiny: it maps the Unicode characters A, b and ? to their one-byte representations. Real encodings need a complete, carefully reviewed map, so start from a close existing map rather than guessing.

Tip: the |0 flag marks a round-trip-safe mapping. Keep duplicate mappings ordered so the |0 entry comes first, then mark the alternative with |1 or |3 as appropriate. Get a duplicate wrong and one byte sequence can decode fine but fail to encode back to itself.

$ workdir=$(mktemp -d /tmp/enc2xs-work.XXXXXX)
$ cd "$workdir"
$ printf '%s\n' \
  '<code_set_name> "Demo-ASCII"' \
  '<code_set_alias> "demo"' \
  '<mb_cur_min> 1' \
  '<mb_cur_max> 1' \
  '<subchar> \x3F' \
  'CHARMAP' \
  '<U0041> \x41 |0' \
  '<U0062> \x62 |0' \
  '<U003F> \x3F |0' \
  'END CHARMAP' > demo.ucm
$ sed -n '1,20p' demo.ucm
<code_set_name> "Demo-ASCII"
<code_set_alias> "demo"
<mb_cur_min> 1
<mb_cur_max> 1
<subchar> \x3F
CHARMAP
<U0041> \x41 |0
<U0062> \x62 |0
<U003F> \x3F |0
END CHARMAP

Do not treat subchar as another character mapping: it is the byte sequence used when a character cannot be represented, which in this example is the question mark byte. The map format uses hexadecimal Unicode code points such as U0041 and byte escapes such as \x41.

Checkpoint: confirm demo.ucm is the file you actually mean to process, and that you are still inside the temporary directory. enc2xs writes generated files into the current directory.

3. Generate the module skeleton

Run enc2xs with a Perl module name. Here the module name is Demo, so the generated package is Encode::Demo:

$ enc2xs -M Demo demo.ucm
/usr/share/perl/5.38/Encode
Generating Makefile.PL...
Generating Demo.pm...
Generating t/Demo.t...
Generating README...
Generating Changes...
$ find . -maxdepth 2 -type f -printf '%P\n' | sort
Changes
Demo.pm
Makefile.PL
README
demo.ucm
t/Demo.t

The generated files are a MakeMaker input, a Perl submodule, documentation placeholders, a change log and a test. enc2xs does not finish the module's documentation for you: edit the generated POD and add tests before sharing a real extension.

If the command fails, read the first error rather than repeatedly flipping flags. Common causes are a misspelled map path, malformed UCM syntax, or a directory you cannot write to. Recreate the scratch directory and try again; that leaves any existing project untouched.

4. Prepare and build without installing

Generate a local Makefile, then compile the extension:

$ perl Makefile.PL
enc2xs is /usr/bin/enc2xs
encode.h is at /usr/share/perl/5.38/Encode
Generating a Unix-style Makefile
Writing Makefile for Encode::Demo
Writing MYMETA.yml and MYMETA.json
$ make
$ test -f blib/arch/auto/Encode/Demo/Demo.so && echo 'built module exists'
built module exists

Your compiler's details and generated command lines may differ. The checkpoint that matters is a successful make and a shared object under blib, which is a local build result, not a system installation.

Warning: do not add sudo to these commands. make install changes the Perl installation and may need elevated privileges depending on the configured prefix. Test first, and only install a reviewed module into a deliberate destination. To undo an uninstalled build, just discard the disposable directory once you have confirmed it holds nothing you need.

5. Run the generated test

Use the test target before you consider the module usable:

$ make test
t/Demo.t .. ok
All tests successful.
Files=1, Tests=2,  0 wallclock secs
Result: PASS

Timing varies. A non-zero result means the generated test or the build environment needs attention: inspect the full output, fix the UCM map or module source, then rerun make and make test. A successful compile is not proof the mapping is correct; add tests for every byte sequence that matters and for the round trips your encoding promises.

6. Keep persistent configuration separate

enc2xs -C updates Encode::ConfigLocal so an encoding can be found through Encode's demand-loading list. That is a persistent configuration change, not part of generating or testing the module. Do not run it on a production Perl installation as a casual final step: decide where the module belongs first, record the current configuration, and make a backup or use your package's normal rollback mechanism.

The UCM format has limits too. The local manual says not every ICU feature is implemented, including icu:state. Algorithmic encodings such as ISO-2022 variants need a Perl module rather than a simple table. If the format cannot describe the encoding's state machine, stop at the map stage and take the module route instead.

Done means