Use zstdmt for Fast, Reversible Multi-threaded Compression

zstdmt compresses a file across every core you let it touch, which is great for throughput and a real problem on a shared host. By the end you will have compressed a file, checked the result, and restored the original data without ever deleting the source.

You need the zstd package, a shell, and enough free space for the compressed output. Allow about five minutes for the first run; choosing sensible thread and memory limits only takes longer when the files or host are large.

1. Check which zstdmt you are using

zstdmt is the multi-threaded entry point for Zstandard. The installed manpage describes it as equivalent to zstd -T0: use the detected number of physical CPU cores for compression. Check both the executable and its version before trusting behaviour from a different installation.

$ command -v zstdmt
/path/to/zstdmt
$ zstdmt --version
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***

The exact version and path on your machine may differ. The local manpage used for this guide is from zstd 1.5.5. If a shell alias, container image or alternate package supplies the command, read that installation's manual as well.

2. Compress one file and keep the source

Run zstdmt with the input file. Compression is the default operation, level 3 is the documented default, and the result is written beside the source with .zst appended. The source is kept unless you explicitly ask for removal.

$ zstdmt /path/to/report.log
/path/to/report.log   : 18.42%   ( 1024 KiB -> 188 KiB)
$ ls -lh /path/to/report.log /path/to/report.log.zst

The progress and ratio vary with the data: text often compresses well, already-compressed media may not. If the destination already exists, zstdmt normally refuses to overwrite it, and that refusal is useful: stop and decide whether the old output is disposable before reaching for -f.

To choose a different normal compression level, use a number from 1 to 19. Higher levels generally trade time and memory for a smaller result, but the useful point depends on your data and workload.

$ zstdmt -6 /path/to/report.log
/path/to/report.log   : 17.91%   ( 1024 KiB -> 183 KiB)

3. Set the worker count deliberately

zstdmt already selects a thread count through -T0. Make it explicit when sharing a host or running from a job scheduler: -T4 asks for four compression worker threads. The command still needs I/O and memory, so four workers is not a promise of four times the throughput.

$ zstdmt -T4 -6 -o /path/to/report.log.zst /path/to/report.log
/path/to/report.log   : 17.91%   ( 1024 KiB -> 183 KiB)

-o gives the output an exact name. It does not make overwriting safe. For a repeatable script, pick a new temporary destination, check its result, then move it into place as a separate, reviewed step. Keep -f out of unattended commands unless replacement is genuinely intended.

For a one-off setting, use -T rather than changing the environment. The manpage also documents ZSTD_NBTHREADS for the compression thread count and ZSTD_CLEVEL for the compression level; command-line options override those variables. Treat inherited environment variables as part of the job's input when diagnosing an unexpected result.

4. Verify the compressed file before relying on it

Use test mode to decompress and checksum the data while discarding the decompressed bytes. It creates and removes no files, so it's a safe checkpoint before copying or deleting anything.

$ zstdmt --test /path/to/report.log.zst
/path/to/report.log.zst : 1024 KiB...     OK

For a quiet script, add -q and inspect the exit status. A zero status means the integrity check succeeded; a non-zero status means the file should not be treated as a valid replacement. Note that a successful test can't prove you copied the file to the intended host or directory, so check the path too.

$ zstdmt -q --test /path/to/report.log.zst
$ printf 'test status: %s\n' "$?"
test status: 0

5. Restore the original into a chosen destination

Decompression with -d removes the .zst suffix when it derives a filename. Use -o when the destination matters, such as restoring into a staging directory. The source archive is kept by default.

$ zstdmt -d -o /tmp/report-restored.log /path/to/report.log.zst
/path/to/report.log.zst: 1024 KiB...     report-restored.log
$ cmp -- /path/to/report.log /tmp/report-restored.log
$ echo "restored bytes match"
restored bytes match

cmp prints nothing when the files match. If it reports a difference, preserve both files and investigate; do not retry with -f over the top of the evidence.

6. Use streams when a file is not the right interface

When input is redirected or piped, zstdmt reads standard input and writes standard output. That's useful for a pipeline, but output to a terminal is refused unless forced, so redirect to a named file rather than letting binary data land in your terminal.

$ tar -cf - /path/to/project | zstdmt -T4 -o project.tar.zst
$ zstdmt -dc project.tar.zst | tar -tf - | head
path/to/project/
path/to/project/README.txt

The -c option writes processed data to standard output and keeps the input unchanged. In pipelines, check each command's status rather than assuming a later command proves compression succeeded: a truncated stream can otherwise look like an ordinary pipeline failure.

Common traps

• Do not start with --rm. It removes source files after successful compression or decompression. With -o, zstdmt can prompt for confirmation; with -f, that confirmation can be suppressed. Run the non-destructive command first, test the result, and confirm the destination. If you used --rm and the source is gone, restore it from a verified copy or backup; there is no zstdmt undo command.
• Watch memory as well as CPU. -T0 can use many physical cores. More workers can increase throughput, but also increase resource pressure and compete with other services. Long-distance mode, enabled with --long, increases the window and memory use for compression and decompression, so use it only when long matches justify that cost.
• Do not confuse testing with extraction. --test validates checksums and discards decompressed data; it does not create a restored file. For that, use -d and choose a destination.