Convert Text Between Encodings Safely with iconv

A file that looks fine in one editor and turns to garbage in another is usually an encoding mismatch, and iconv fixes it without guesswork. Allow about ten minutes for a one-off conversion, longer if you need to identify an unknown source encoding first. Examples use the iconv from glibc 2.39 on this machine, as reported by iconv --version; other glibc releases can differ in their available conversion modules and diagnostics.

1. Check the installed command

Confirm which executable runs and record its version. This is an ordinary, unprivileged check: you do not need sudo to convert files you can already read and write.

$ command -v iconv
/usr/bin/iconv
$ iconv --version
iconv (Ubuntu GLIBC 2.39-0ubuntu8.9) 2.39

iconv reads from named files, standard input, or a single -, and writes to standard output unless you use -o. -f sets the input encoding, -t the output encoding. Leave either out and iconv derives it from the current locale, which is easy to forget in a script, so state both encodings whenever the data matters.

Checkpoint: you have confirmed the executable and know which encoding the source actually uses. A file name or an editor's guess is not an encoding declaration.

2. Convert to a new file without risking the original

Warning: shell redirection with > truncates an existing destination before iconv has done any work, so never redirect straight over a file you care about. Use a new destination while testing:

$ iconv -f ISO-8859-15 -t UTF-8 < input.txt > output.txt
$ printf 'exit status: %s\n' "$?"
exit status: 0

This reads input.txt as ISO-8859-15 and writes UTF-8. A zero exit status means the conversion completed, not that you picked the right source encoding, so inspect the result in a tool that shows its detected or specified encoding. The equivalent for a named input file is:

$ iconv -f ISO-8859-15 -t UTF-8 input.txt -o output.txt

The output file is created or replaced. For an atomic swap after checking the result, write to a temporary name in the same directory, verify it, then mv it into place, keeping the original until that check is done:

$ iconv -f ISO-8859-15 -t UTF-8 input.txt > output.txt.new
$ test -s output.txt.new && file output.txt.new
$ mv output.txt.new output.txt

If the conversion fails, leave the old destination alone and remove the incomplete output.txt.new only after inspecting the error: do not run a cleanup command blindly if that temporary file might still hold data you need.

3. Convert a pipeline with explicit boundaries

With no input file, iconv reads standard input, which is handy in a pipeline but also means the command can hang waiting for input if an earlier stage is missing. Test with something small and visible first:

$ printf 'plain text\n' | iconv -f UTF-8 -t UTF-8
plain text

iconv does not add that trailing newline itself, printf does. A dash as the input argument means the same thing as standard input:

$ printf 'plain text\n' | iconv -f UTF-8 -t UTF-8 -
plain text

Keep the encoding arguments next to iconv rather than passing an unreviewed option string from an environment variable, and quote any paths or values that might contain whitespace.

4. Find an encoding name when a conversion is rejected

iconv accepts the aliases its installed gconv modules provide. List everything this installation knows with -l:

$ iconv --list | sed -n '1,12p'
437//
500//
500V1//
850//
851//
852//
855//
856//
857//
858//
860//
861//

This list is machine-specific and may contain aliases or names with a trailing //. If iconv reports an unknown encoding, compare the requested name against this list and against whatever produced the file. Do not switch encodings before you know what the source bytes actually represent: treating arbitrary bytes as UTF-8 can stop early or produce misleading text.

5. Decide what happens to unconvertible characters

A strict conversion stops the moment a character cannot be represented in the target encoding. Converting UTF-8 text containing é to ASCII returns a non-zero status and an illegal-sequence error, which is a useful failure, not a reason to discard data automatically:

$ printf 'cafe\303\251\n' | iconv -f UTF-8 -t ASCII
iconv: illegal input sequence at position 4
$ printf 'exit status: %s\n' "$?"
exit status: 1

Use -c only when your stated policy is to discard characters that will not convert:

$ printf 'cafe\303\251\n' | iconv -c -f UTF-8 -t ASCII
cafe
$ printf 'exit status: %s\n' "$?"
exit status: 0

This is lossy and cannot be undone from the output. The //IGNORE suffix on the target encoding also discards unconvertible characters, but the installed manpage says it still prints an error after conversion, so on this machine it returns status 1 for the same input. Do not use either form unless losing those characters is acceptable and you have kept the original.

//TRANSLIT asks iconv to approximate characters instead of dropping them. On this installation it renders ß as ss and € as EUR when converting to ASCII:

$ printf 'abc ß α € àḃç\n' | iconv -f UTF-8 -t ASCII//TRANSLIT
abc ss ? EUR abc

Warning: transliteration still changes meaning or spelling in some contexts. Review names, identifiers and legal text rather than treating the result as equivalent to the original.

6. Verify and recover

Check the exit status right after iconv, then inspect the output with a suitable text or encoding-aware tool. A zero status confirms the conversion path ran, not that the source encoding you chose was semantically correct:

$ iconv -f UTF-8 -t UTF-8 input.txt > output.txt.new
$ status=$?
$ if [ "$status" -eq 0 ]; then mv output.txt.new output.txt; else printf 'conversion failed: %s\n' "$status" >&2; fi

Recovery: if the command fails, the original input is unchanged. Keep the diagnostic, fix the encoding pair or input path, and rerun into a fresh temporary destination. If a destination was accidentally truncated by direct redirection, restore it from your backup or version control: iconv has no undo operation of its own.

Done means