Make roff Encoding Predictable with preconv

One accented character in a roff file can mangle troff's output, and preconv is the fix. It converts non-ASCII source into groff Unicode escapes before troff sees it. This covers detection, forcing a known encoding, choosing a sane fallback, and the one thing preconv can never see: text pulled in later by another file. The examples use GNU preconv from groff 1.23.0, package version 1.23.0-3build2, with iconv and uchardet support.

Allow about ten minutes. You need a shell, a roff input file or a pipe, and the groff-base package. These examples only read input and write converted output; nothing alters the source file or system configuration.

1. Check the installed version and interface

Start by checking the binary you will actually run:

$ preconv --version
GNU preconv (groff) version 1.23.0 with iconv support and with uchardet support

Version matters here because encoding detection and the available conversion libraries are build and release details. The useful form is preconv [-dr] [-D fallback-encoding] [-e encoding] [file ...]. With no file, or - as the file name, it reads standard input and writes converted roff to standard output.

2. Convert a known UTF-8 file

For a normal source file, let preconv inspect it and send the result on to another command or file. This creates a temporary input with a UTF-8 accented character, then shows the converted roff escape:

$ printf 'caf\303\251\n' > /tmp/example.roff
$ preconv -r /tmp/example.roff
caf\[u00E9]

• ASCII stays ASCII. Anything in that range passes through as ordinary text.
• Everything else becomes an escape. Non-ASCII characters turn into groff Unicode escapes such as \[u00E9], which troff can interpret.
• -r means raw. It suppresses the default .lf request that identifies the input for later diagnostics.

Leave out -r and the same input starts with a line marker instead:

$ preconv /tmp/example.roff
.lf 1 /tmp/example.roff
caf\[u00E9]

That marker helps later groff diagnostics point back at the original file and line. It is output, not a change to the source, so do not strip it just because it looks unfamiliar.

3. Declare the encoding when detection is not trustworthy

Detection is convenient, but a short file or a pipe may not give it enough evidence. Use -e when you already know the input encoding and want to skip guessing:

$ printf 'caf\351\n' > /tmp/latin1.roff
$ preconv -r -e latin-1 /tmp/latin1.roff
caf\[u00E9]

That input holds byte 0xE9, the accented character in Latin-1. Declare it as UTF-8 and you would get the wrong answer. Declare it as US-ASCII on this iconv-enabled build and it discards the byte outright:

$ preconv -r -e us-ascii /tmp/latin1.roff
caf

4. Make fallback behaviour explicit

When no explicit encoding, byte-order mark or coding tag settles the question, preconv tries detection and then falls back. Choose that final fallback yourself with -D:

$ LC_ALL=C preconv -r -D latin-1 /tmp/latin1.roff
caf\[u00E9]

The documented order is: explicit -e, a Unicode byte-order mark, a recognised GNU Emacs coding tag on the first or second line of a seekable file, uchardet if available, then -D, and finally the locale. For the C, POSIX or empty locale the manual names Latin-1 as that final fallback; on this installation, --help instead reports UTF-8 as the default. The practical rule stays simple: -e for a known encoding, -D for a deliberate final fallback.

A coding tag documents the source for tools that follow the GNU Emacs convention. It has to sit in a roff comment on the first or second line, using the default control and escape characters:

\" -*- coding: latin-1 -*-
caf\351

Keep the tag with the source file. It is a hint for the reader, not a conversion result, and it only counts on a seekable file.

5. Treat pipes as a separate case

A regular file can be sought back to its opening lines; a pipe cannot. Feed input through a pipe and coding-tag and uchardet detection are both skipped, so a short or ambiguous stream can end up interpreted with the wrong locale fallback.

$ printf 'caf\351\n' | preconv -r -e latin-1
caf\[u00E9]
$ printf 'caf\351\n' | preconv -r -D latin-1
caf\[u00E9]

Prefer -e whenever the producer's encoding is actually known. Reach for -D only when treating every undecided input as one fixed fallback is genuinely safe. Adding a coding tag to data already inside a pipe will not make preconv notice it.

6. Connect it to groff without duplicating work

In ordinary use you rarely need to call preconv directly: the preprocessor is meant to be selected through groff's -k or -K options. Call it directly when you want to inspect or record the converted stream, or a pipeline needs this stage spelled out.

For a direct pipeline, keep standard output separate from diagnostics and choose the output destination on purpose:

$ preconv -r -e utf-8 document.roff > document.converted.roff
$ groff -Tutf8 document.converted.roff > document.txt

Redirection truncates an existing destination before the command runs. Before replacing a valuable generated file, write to a new name and compare it first. To recover from an unwanted conversion, go back to the original source, or remove only the derived output once you have confirmed its exact path. preconv has no in-place editing mode of its own.

Common traps

• Included content stays unconverted. preconv only sees the input it directly receives. It cannot transform text inserted later by soelim, by troff's so request, or through strings passed with troff's -d option. Put the conversion stage where the bytes actually enter the pipeline, and do not assume an included file got the same treatment.
• Custom escape characters break the assumption. preconv assumes the default backslash escape and emits special-character escapes accordingly. If your source changes that convention, check the pipeline design rather than trusting that the generated Unicode escapes will still be read as intended.