Identify a File's Character Encoding with encguess

A file arrives and every accented letter is a row of question marks, so you run encguess to find out what you are dealing with before you touch it. You will test one or more files, narrow the candidates when the defaults are not enough, and handle an inconclusive result without silently rewriting data. Allow about five minutes if Perl is already installed. This guide was checked with Perl 5.38.2 and package perl 5.38.2-3.2ubuntu0.6.

The route

Jump straight to the step you need, or tick off Done means at the end.

Before you start
1. Check a file with the defaults
2. Limit the suspect encodings
3. Decide what an inconclusive result means
4. Verify before converting
Option summary

Before you start

You need: a shell, read access to the files, and the encguess command.
Privileges: none for ordinary files.
Safety: encguess only reports a guess. It does not change file contents. Still, work on a copy if a later conversion tool will modify the original.

Checkpoint

Confirm the command and version before you interpret any result.

command -v encguess
perl -v | sed -n '1,4p'
dpkg-query -W -f='${Package} ${Version}\n' perl

On the checked system, the version line identifies Perl 5.38.2 and the package query reports perl 5.38.2-3.2ubuntu0.6. Package versions vary, so record the output alongside any processing decision.

1. Check a file with the defaults

Pass one or more filenames after the command. With no -s option, the utility tests ASCII, UTF-8, and UTF-16 or UTF-32 data with a byte-order mark. It prints the filename, a tab, and the surviving encoding label.

encguess /path/to/input.txt

For example, an ASCII file on the checked system produced:

/path/to/input.txt	US-ASCII

Other results seen on the checked system:

UTF-8 with a byte-order mark produced UTF-8.
Little-endian UTF-16 produced UTF-16.
Little-endian UTF-32 produced UTF-32.

The output is a candidate, not a certificate that every byte has the intended meaning.

You can check several files in one go. Let the shell expand a glob from the directory you mean, so you are not surprised by a different working directory:

encguess -- /path/to/incoming/*.txt

The -- marker is accepted by the Perl option parser and stops a filename that starts with a hyphen being read as a switch. If the glob matches nothing, the shell may pass the literal pattern, so check the directory first when that distinction matters.

2. Limit the suspect encodings

Use -s when you know where the files came from. Separate encoding names with a comma or a colon:

encguess -s euc-jp,shiftjis,7bit-jis /path/to/input.txt
encguess -s euc-jp:shiftjis:7bit-jis /path/to/input.txt

A shorter suspect list is easier to read, but it can also rule out the right answer. The names must be ones Perl's Encode library recognises, so ask the installed utility instead of guessing:

encguess -S | sed -n '1,25p'

Warning

Do not put a space-separated list after -s. The descriptive text in this installed manpage mentions a quoted space-separated form, but the command only accepts comma or colon separators. Here, encguess -s 'ascii utf8' file fails with Unknown encoding: ascii utf8. Use -s ascii,utf8 instead.

3. Decide what an inconclusive result means

Short files and single-byte encodings are the hard cases. The underlying Encode::Guess documentation warns that many single-byte encodings decode almost any byte sequence successfully.

Treat any of these as "I need more evidence":

A result of unknown.
A result that changes when you alter the suspect list.

Then inspect the producer's documentation, compare a longer sample, or ask the sender to state the encoding.

To see which files in a batch have no useful label, keep the normal output first:

encguess /path/to/incoming/*.txt

Then use -u if unidentified entries are noise in a script:

encguess -u /path/to/incoming/*.txt

-u suppresses display of unidentified types. It does not improve detection, and it does not make every printed result certain. In a script, preserve the output and exit status if you need an audit trail. A printed encoding is not permission to overwrite the source.

4. Verify before converting

Use the reported name to configure the next tool, but check that the decoded text looks sensible before you save a replacement. Inspect a small portion without changing the file:

file --brief --mime /path/to/input.txt
od -An -tx1 -N16 /path/to/input.txt

These give independent clues such as a MIME label or a byte-order mark. They do not turn a guess into proof.

Recovery

There is no undo in encguess because it never edits the input. For an important archive, keep the original and write converted output to a new path. If a conversion goes wrong, discard the derived file and go back to the untouched original.

Option summary

Option	Use
`-h`	Show usage and exit.
`-s LIST`	Test only the named suspects, separated by commas or colons.
`-S`	List encoding names accepted by `-s`.
`-u`	Hide entries reported as unidentified.

Done means

Version recorded. You noted the local Perl and package version.
Sensible suspects. You checked the file with the defaults or a justified, focused list.
Label plus filename. You read them together rather than trusting a bare label.
Odd results chased. You investigated unknown or conflicting results before converting.
Original kept. Any converted output went to a new file.