Clean, Inspect and Split Mail with formail

Formail will happily eat your only copy of a mailbox if you point a redirect at the wrong file. This guide gives you repeatable commands for cleaning mbox mail, reading or changing headers, splitting a mailbox into messages and spotting repeated Message-ID values, without that mistake. The examples use formail from procmail 3.24 (Ubuntu package 3.24-1ubuntu2). Allow about fifteen minutes. You need a shell and a readable mail file, and nothing here needs elevated privileges.

Safety boundary: Formail writes transformed mail to standard output. A shell redirection can truncate an existing file before formail has produced anything useful. Write to a new or temporary file, then replace the old one only after checking the result.

1. Check the installed command

Confirm which executable will run and record its version. The version option prints the program's version and exits.

$ command -v formail
/usr/bin/formail
$ formail -v
formail version 3.24

Checkpoint: the command resolves to the executable you expect and reports 3.24. The exact first line can vary slightly with the package build. This guide covers the installed procmail implementation, not a newer mail tool with a similar purpose.

2. Normalise an mbox stream

With no options, formail reads mail on standard input and forces it into mailbox format. It escapes every body line that looks like a separator (one beginning with From ) by adding >. That stops a line in a message body being mistaken for the start of the next message.

$ formail < old-mailbox > normalised-mailbox

For a safe replacement, keep the original and write beside it first:

$ formail < old-mailbox > normalised-mailbox.new
$ test -s normalised-mailbox.new
$ mv normalised-mailbox.new normalised-mailbox

• Test before you move. The mv command overwrites the destination name, so run it only after the new file checks out.
• Bad conversion? Leave normalised-mailbox untouched and inspect or remove the .new file by hand.
• No root needed. There is no reason to run formail as root.

3. Extract headers

Use -x with a field name to print its value. Continuation lines stay separate. Add -c when a single line suits a line-oriented tool better.

$ formail -x Subject: < message.eml
 Weekly report
$ formail -c -x Subject: < message.eml
Weekly report

The leading space in the first result is normal header content. Use -X when you want the field name included:

$ formail -X Subject: < message.eml
Subject: Weekly report

These options work on the header, and by default the body is dropped when you extract fields. Add -k if the body must pass through too.

Checkpoint: send the result to a pager or a new file, then confirm the original message still exists unchanged.

4. Edit fields with care

Header editing is handy for preparing an archive or stripping routing details, but it can also leave a message invalid. Test the output before handing it to a delivery system.

For example, remove every Received: field and keep the rest of the message:

$ formail -I Received: < message.eml > message-without-routing.eml
$ formail -x Received: < message-without-routing.eml

Checkpoint: an empty result means no matching field remains. The field name may be partial, so a value such as Received: can select fields beginning with that name.

Warning: treat the output as a privacy-sensitive copy. Removing routing headers also removes useful forensic and delivery context.

To replace a field and keep the old value under an Old- name, use -i:

$ formail -i 'Reply-To: [email protected]' < message.eml > prepared.eml
$ formail -X Reply-To: -X Old-Reply-To: < prepared.eml

• -A appends even when a similar field exists.
• -a appends only when it does not.
• -I Field: removes matching fields.
• Input is safe. These operations change only the output stream, not the input file.

5. Split a mailbox into messages

Put -s last. It detects message boundaries and sends each message to a fresh run of the program that follows. With no program, formail joins the split messages back together, which is useful when you only want its boundary detection or other options.

$ formail -ds sh -c 'printf "message %s: " "$FILENO"; formail -x Subject:' < mailbox
message 000: First message
message 001: Second message

The FILENO environment variable identifies the message currently being output. It starts at 000 by default.

Tip: do not assume every input will split. Strict mailbox separators need the expected blank-line structure. For digests, articles or other non-standard formats, add -d. That disables recognition of Content-Length: while splitting.

For a digest whose first item is a wrapper, the manual's typical pattern is:

$ formail +1 -ds < digest > messages.mbox

+1 skips the first message while splitting. A negative option such as -10 limits output to ten messages. Use a new destination, check its message count or subjects, and keep the source until you have reviewed the split.

6. Detect duplicate messages

Use -D with an approximate cache size and a cache path to remember Message-ID values:

$ formail -D 100000 /tmp/formail-message-ids < message.eml
$ status=$?
$ printf 'duplicate check status: %s\n' "$status"
duplicate check status: 1

Without splitting, status zero means a duplicate was found. Status one means this message was not already in the cache. When splitting, duplicate messages are simply not output.

The cache file is state, so pick a path with suitable permissions. Keep it between runs if you want detection across separate invocations.

Warning: with -r, duplicate detection uses the envelope sender address instead of Message-ID. Do not use that mode casually, because automatic replies are security-sensitive and can reply to a mailing list. If the sender should come from the message's header rather than its envelope, -t changes that choice. Review the generated headers before delivery.

7. Watch for surprising options

• Quiet by default. The normal behaviour suppresses write errors, duplicate notices and mismatched Content-Length: diagnostics. Use -q- when you need those messages.
• Escaping off with -b. It disables escaping of bogus mailbox headers. Use it only when the receiving format deliberately does not need mbox escaping.
• Berkeley mode with -Y. It selects traditional Berkeley mailbox handling and ignores Content-Length: fields.
• Parallel splits with -n. Split commands can run in parallel, optionally with a process limit. Parallel output makes logs harder to read and increases the load on the child program.
• The foo@bar sender. If an input has no recognisable sender, formail substitutes foo@bar. Treat that as a parsing failure to investigate, not a real address.

Recovery: if a command fails, rerun it with a new output path and capture the exit status straight away. Do not add sudo as a generic fix. Fix permission errors on the specific input, output or cache path, and keep the original mail available.

Done means

• Version confirmed. You checked the installed formail binary and the procmail 3.24 version.
• Input preserved. Mailbox conversion went to a new file, and the source stayed until you reviewed the result.
• Headers extracted deliberately. You used -x or -X and chose how to handle continuation lines.
• Edits verified. You checked which fields should be present or absent after editing.
• Splits controlled. You used -s last, with -d, +skip or a limit only where the input needed it.
• Duplicate checks contained. The cache path is controlled, and automatic replies are reviewed before delivery.