Convert Text Safely with mariadb-conv

A file that lands in the wrong character set is a nuisance, and mariadb-conv is the small MariaDB utility built to fix that. You will finish with a repeatable way to convert a text file or stream between MariaDB character sets, keep separators unchanged, and decide what happens when input cannot be converted. The examples use mariadb-conv 10.11.14 from the Ubuntu mariadb-client package.

Allow about fifteen minutes. You need a shell, a readable input file or pipe, and enough free space for a separate output file. None of this needs elevated privileges. Do not run the converter as root just because a file came from a MariaDB data directory; use the least-privileged account that can read the source.

1. Check the installed command

Confirm which executable your shell will use and record the package version. This is a read-only checkpoint:

$ command -v mariadb-conv
/usr/bin/mariadb-conv
$ dpkg-query -W -f='${Package} ${Version}\n' mariadb-client
mariadb-client 1:10.11.14-0ubuntu0.24.04.1

The installed utility identifies itself as MariaDB 10.11.14. The local manual gives the interface as mariadb-conv [OPTION...] [FILE...], with these controls:

2. Convert to a new file

Name the source and destination encodings explicitly. This example converts a Latin-1 file to UTF-8:

$ mariadb-conv --from=latin1 --to=utf8 /path/to/input.txt > /path/to/output.txt

The short options do the same job:

$ mariadb-conv -f latin1 -t utf8 /path/to/input.txt > /path/to/output.txt

Warning: keep the input and output paths different. Shell redirection opens the destination before mariadb-conv even starts, so writing > input.txt can truncate the original before conversion gets a chance to read it. The command writes converted bytes to standard output; the shell decides where they land.

Checkpoint: verify the output exists and is not empty, then inspect its detected type or bytes:

$ test -s /path/to/output.txt && echo 'output is non-empty'
output is non-empty
$ file /path/to/output.txt
/path/to/output.txt: Unicode text, UTF-8 text

The exact file wording varies. For a stronger check, compare a known non-ASCII character or feed the output to a consumer that expects UTF-8. A successful exit status means the conversion completed; it does not prove the receiving application will read the output as you intended.

3. Use standard input and a pipe

Leave off the file operand when input should come from standard input, useful when another command produces a listing or stream:

$ mariadb-conv -f latin1 -t utf8 < /path/to/input.txt > /path/to/output.txt
$ some-command | mariadb-conv -f latin1 -t utf8 > /path/to/output.txt

Check the exit status immediately if the pipeline matters. Without pipeline error handling, some shells let a later command hide an earlier failure:

$ set -o pipefail
$ some-command | mariadb-conv -f latin1 -t utf8 > /path/to/output.txt
$ printf 'pipeline status: %s\n' "$?"
pipeline status: 0

set -o pipefail is not a repair for bad input; it just makes a failure easier to notice. If the producer emits a different encoding from the one named by --from, fix the source declaration or stop and investigate.

4. Keep separators out of conversion

Use --delimiter when the stream has separators that are structural rather than text data. Every character named by the option is copied as-is; the chunks between them get converted. A dot separator is enough for this small example:

$ mariadb-conv -f latin1 -t utf8 --delimiter='.' /path/to/names.txt > /path/to/names-utf8.txt

This matters for encoded MariaDB file names followed by an extension, and for line-oriented listings. The delimiter itself is never converted, so list every structural separator you need, dot and newline included, and quote the value so the shell does not touch the characters in it.

Checkpoint: compare the separators in the two files. Surrounding text can change byte length after conversion, but separator bytes should stay in the same position relative to each chunk:

$ od -An -tx1 /path/to/names-utf8.txt
 63 61 66 c3 a9 2e 62 61 64 0a

The bytes shown are just an example. Do not reach for --delimiter to hide unknown corrupt data; identify the format and its real boundaries first.

5. Choose a policy for conversion errors

By default, the command stops the moment an input sequence is invalid for the source encoding, or a character has no representation in the target encoding. That is the safer default for data that must stay exact:

$ mariadb-conv -f utf8 -t latin1 /path/to/input.txt > /path/to/output.txt
Illegal utf8mb3 byte sequence at position 12

The position and wording vary with the bad bytes and the installed build. A non-zero exit status and an incomplete output file are both possible outcomes. Treat that output as suspect; do not let it replace a known-good file.

Add --continue only when replacement is an accepted, documented policy:

$ mariadb-conv --continue -f utf8 -t latin1 /path/to/input.txt > /path/to/output.txt
$ printf 'converter status: %s\n' "$?"
converter status: 0

Warning: with this option, the utility replaces bad input sequences or unconvertible characters with ? and keeps going. That can keep a display-only listing moving, but it is lossy. Keep the original input, tell the consumer about the replacement policy, and never use this mode for identifiers, audit records, passwords, or anything that must round-trip exactly.

6. Replace an existing output only after checking it

If the destination already matters, write a temporary sibling and move it into place only once it checks out. The mv step changes state and can replace the old file, so look at the paths first:

$ ls -l /path/to/input.txt /path/to/output.txt
$ mariadb-conv -f latin1 -t utf8 /path/to/input.txt > /path/to/output.txt.new
$ test -s /path/to/output.txt.new
$ file /path/to/output.txt.new
$ mv -- /path/to/output.txt.new /path/to/output.txt

Recovery: there is no automatic undo for that final move. Make a backup first if the old output would be hard to recreate:

$ cp --preserve=all -- /path/to/output.txt /path/to/output.txt.bak

If conversion or checking fails, leave the original output alone and remove only the new temporary file. Delete the backup later, deliberately, once the replacement has proven itself. None of this needs sudo unless the chosen directory itself denies your account access.

Done means