Convert Text Encodings Safely with piconv

You will convert text from one character encoding to another, using standard input, files, or a shell string. You will also check that an encoding name means what you think it means, and choose what happens when the destination cannot represent a character. The examples use the piconv shipped with Perl 5.38.2, package version 5.38.2-3.2ubuntu0.6 on this machine.

Allow about ten minutes for a one-off conversion. You need Perl's piconv command and a shell. None of the normal examples need sudo. They write to standard output, so a mistaken command does not alter the input unless you redirect it over the original file.

1. Check the command and the encoding names

Start by confirming the executable and its version. The manpage is dated 14 September 2026 and identifies this implementation as Perl v5.38.2.

$ command -v piconv
/usr/bin/piconv
$ perl -v | sed -n '1,4p'
This is perl 5, version 38, subversion 2 (v5.38.2)

Encoding names are case-insensitive, and aliases are accepted. Ask piconv to resolve an alias before putting it into a script. The result is the canonical name used by Perl's Encode module.

$ piconv -r latin1
iso-8859-1
$ piconv -r UTF-8
utf-8-strict

To browse the available canonical names, use -l. The list is long, and it deliberately does not show every alias.

$ piconv -l | sed -n '1,8p'
7bit-jis
AdobeStandardEncoding
AdobeSymbol
AdobeZdingbat
ascii
ascii-ctrl
big5-eten
big5-hkscs

2. Convert a stream without changing the source

Use -f for the input encoding and -t for the output encoding. With no file argument, piconv reads standard input and writes standard output. This example produces ISO-8859-1 bytes from UTF-8 text.

$ printf 'caf\303\251\n' | piconv -f UTF-8 -t ISO-8859-1 | od -An -tx1c
  63  61  66  e9  0a
   c   a   f 351  \n

The output is a byte stream, not a report. Use file, od, or a consumer that understands the destination encoding when you need to verify it. A conversion that looks fine in a terminal can still be wrong if the terminal is interpreting the bytes differently.

For a file, name it after the options and redirect the result to a new file:

$ piconv -f UTF-8 -t ISO-8859-1 /path/to/input.txt > /path/to/output-latin1.txt
$ file -bi /path/to/output-latin1.txt

This is deliberately a new output path. Do not redirect to the input file: the shell truncates the destination before piconv reads it, which can destroy the source. If you accidentally created an output over a useful file, stop and restore it from your backup or version control; there is no undo operation in piconv.

3. Convert a quoted string

The -s option uses its argument as the source instead of standard input or a file. Quote the value so the shell passes spaces and punctuation as one argument.

$ piconv -f UTF-8 -t ISO-8859-1 -s 'caf\303\251 and tea'
café and tea

That displayed result contains the ISO-8859-1 byte for é, not necessarily UTF-8 text. For repeatable scripts, prefer a file or a pipe when the input is more than a small literal. Keep the source encoding explicit: if -f is omitted, the current locale is used.

4. Decide what to do with unrepresentable characters

ASCII cannot encode é. Without a special option, this implementation replaces the character with a question mark and exits successfully:

$ printf 'caf\303\251\n' | piconv -f UTF-8 -t ASCII
caf?
$ printf 'caf\303\251\n' | piconv -f UTF-8 -t ASCII -p
caf\x{00e9}

If replacement would hide damaged data, use -c, which is the same as -C 1. The command reports the offending character and exits non-zero, allowing a script to stop rather than publish altered text.

$ printf 'caf\303\251\n' | piconv -f UTF-8 -t ASCII -c > /tmp/ascii.txt
"\x{00e9}" does not map to ascii at /usr/bin/piconv line 101, <STDIN> line 1.
$ echo $?
255

When ASCII output is required but you need to preserve a visible marker, choose one of the transliteration modes. -p uses a hexadecimal Perl-style code point, --htmlcref uses a decimal HTML character reference, and --xmlcref uses a hexadecimal XML character reference.

$ printf 'caf\303\251\n' | piconv -f UTF-8 -t ASCII --htmlcref
café
$ printf 'caf\303\251\n' | piconv -f UTF-8 -t ASCII --xmlcref
café

These modes change the text into an ASCII representation. They do not make the destination file genuinely contain the original character, so choose them for a format that expects those references.

5. Handle UTF-16 line endings

The default conversion scheme is from_to. The manpage also documents decode_encode and perlio. Use perlio when converting UTF-16 or another encoding whose line-feed representation does not match Perl's normal input record separator.

$ piconv -f UTF-8 -t UTF-16LE /path/to/input.txt > /path/to/output-utf16le.txt
$ piconv -S perlio -f UTF-16LE -t UTF-8 /path/to/output-utf16le.txt > /path/to/round-trip.txt
$ cmp --silent /path/to/input.txt /path/to/round-trip.txt && echo 'round trip verified'
round trip verified

If a round trip differs, inspect the bytes and line endings rather than assuming the character set is wrong. Also check whether the source has a byte-order mark and whether the encoding name you selected describes it.

6. Keep the operation scriptable

For a batch conversion, make the destination temporary, check the exit status, then replace the original only after inspection. The replacement is the state-changing step and may need a writable directory, but it does not need elevated privileges when you own the files.

set -eu
input='/path/to/input.txt'
output="${input}.new"
piconv -c -f UTF-8 -t ISO-8859-1 "$input" > "$output"
cmp --silent "$input" "$output" || true
mv -- "$output" '/path/to/output-latin1.txt'

Do not keep the || true pattern on the conversion command itself. Here it only makes the optional comparison harmless because the two encodings normally have different bytes. If piconv -c fails, the shell stops before the mv and the original remains untouched. Remove an unwanted temporary file with rm -- "$output" only after checking its exact path.

With both -f and -t omitted, piconv uses the current locale for both and acts like cat. That is useful for a quick pass-through, but it is not a reliable encoding conversion across machines with different locale settings. Put both encodings in production commands.