Detect a file's character encoding with chardet

You will use chardet to report a best-effort character encoding and confidence level for a file or a stream. Allow about ten minutes for a few files, including a check of the result. This guide covers the Debian python3-chardet package installed here, version 5.2.0. Its manual page describes the command as heuristic, so treat the output as evidence for the next step, not as a proof that the bytes have one uniquely correct interpretation.

1. Check the installed command

The command has no documented options in this installation. Its syntax is a command followed by zero or more file names. With no file names, it reads standard input. Confirm that your shell will run the expected binary and record the package version before putting it into a script:

$ command -v chardet
/usr/bin/chardet
$ chardet --version
chardet 5.2.0
$ dpkg-query -W -f='${Package} ${Version}\n' python3-chardet
python3-chardet 5.2.0+dfsg-1

Your path or package revision may differ. If command -v finds nothing, install the package through your normal system-management process. Do not use sudo merely to inspect or detect a readable file.

Checkpoint: start with standard input

Before scanning an archive, test the data path with text that has an obvious result. The detector reads the bytes from the pipe and labels the stream as stdin:

$ printf 'one\ntwo\n' | chardet
<stdin>: ascii with confidence 1.0

The exact label and confidence formatting are part of the installed command's output. A confidence of 1.0 is the detector's score, not a cryptographic guarantee.

2. Detect one file

Pass a readable path as a separate shell argument. This example scans a file without changing it:

$ chardet /path/to/input.txt
/path/to/input.txt: utf-8 with confidence 0.7525

The filename and score above are examples of the output shape. Your encoding and confidence depend on the actual bytes. A result such as ascii is consistent with plain seven-bit text, while a result such as utf-8 is a heuristic classification of the byte sequence. A short file containing only ordinary ASCII cannot demonstrate that it would remain valid when non-ASCII text is added.

Checkpoint: confirm that the file was read rather than silently replacing the path with a shell expansion or typo:

$ test -r /path/to/input.txt && echo readable
readable
$ chardet -- /path/to/input.txt
/path/to/input.txt: utf-8 with confidence 0.7525

The -- form is useful when a pathname begins with a hyphen. The manpage documents file operands rather than a general option interface, so do not assume that arbitrary switches are supported.

3. Compare several files in one run

Give multiple paths when you want one report per input. The program reads each file and prints its name, detected encoding and confidence on its own line:

$ chardet /path/to/legacy.txt /path/to/export.csv
/path/to/legacy.txt: iso-8859-1 with confidence 0.73
/path/to/export.csv: utf-8 with confidence 0.99

Those encodings and scores are illustrative because the files are host-specific. Do not copy them into a conversion command without checking your own output. If the same directory contains names with spaces, quote each pathname:

$ chardet "/path/to/Customer export.csv" "/path/to/old notes.txt"

Quoting protects the filename from shell word splitting. It does not make an unreadable file readable, and it does not alter the bytes being detected.

4. Use the result to choose a safe next action

chardet only reports a detection. It does not rewrite a file, convert it to UTF-8, add a declaration, or repair malformed text. Keep the original file while you test a consumer or conversion tool, particularly when the confidence is low or the input is short.

For a repeatable decision, save the report separately instead of overwriting the input. The following writes the detector's text output to a new report file:

$ chardet /path/to/input.txt > /path/to/input.chardet.txt
$ test -s /path/to/input.chardet.txt && cat /path/to/input.chardet.txt
/path/to/input.txt: utf-8 with confidence 0.7525

Redirection with > truncates an existing report before chardet runs. Choose a new report name if the old report matters. If you need to replace a report, write a temporary report beside it, inspect it, then move it into place only after the command succeeds:

$ chardet /path/to/input.txt > /path/to/input.chardet.txt.new
$ test -s /path/to/input.chardet.txt.new
$ mv /path/to/input.chardet.txt.new /path/to/input.chardet.txt

The mv command changes the report name, not the source file. If detection fails, leave the old report in place and inspect the error before retrying. Remove an unwanted temporary report only after checking its path; deletion is irreversible.

5. Handle uncertain or surprising results

A detector can produce a plausible label for bytes that are damaged, truncated, mixed-encoding, or too short to distinguish several encodings. Compare the result with the file's provenance and with a known-good sample from the same system. If the text displays incorrectly after you open it, do not conclude that the highest score was correct. Try an explicitly selected encoding in a tool that supports one, and keep the original bytes unchanged until the result is verified.

If the command cannot open a path, check the path and read permission without changing anything:

$ ls -l /path/to/input.txt
$ test -r /path/to/input.txt && echo readable
readable

If the second command prints nothing, the current user cannot read the file or the path is wrong. Fix ownership or permissions through your normal administration process only when you are authorised to do so. Elevated privileges are not needed for ordinary detection, and using sudo chardet ... can expose file contents to the root account without improving the heuristic.

To undo this guide's examples, delete only a report you deliberately created, or keep it as an audit note. No command above changes the input file, a service, system configuration or package state.