Encrypt and Verify Files with OpenSSL CMS

You will finish with a small, repeatable workflow for signing, verifying, encrypting and decrypting a file with openssl cms. The commands use the OpenSSL 3.6.1 installed on this machine and keep the test certificate separate from any real key material.

Allow about twenty minutes. You need OpenSSL, a shell, a readable input file and a directory where you can create temporary files. No command in this guide needs sudo. You need a certificate and its matching private key for signing or decryption, and a recipient certificate for encryption.

1. Check the installed command

Confirm the version and read the operation list before choosing a command. These are ordinary, read-only checks:

$ openssl version
OpenSSL 3.6.1 27 Jan 2026
$ openssl cms -help

The CMS command handles S/MIME and CMS structures. Its main operations are -sign, -verify, -encrypt and -decrypt. Input and output default to S/MIME format, but PEM and DER are also available. Choosing the format explicitly makes scripts easier to inspect.

2. Create isolated test material

Use a new temporary directory and a short-lived self-signed RSA certificate. Replace the subject only if you have a reason to identify the test certificate differently:

$ work=$(mktemp -d /tmp/cms-test.XXXXXX)
$ cd "$work"
$ printf 'CMS guide test\n' > message.txt
$ openssl req -x509 -newkey rsa:2048 -nodes \
    -keyout key.pem -out cert.pem -days 1 \
    -subj '/CN=CMS guide test'

The private key is key.pem and the public certificate is cert.pem. The -nodes option leaves this test key unencrypted, which avoids a password prompt in a disposable directory. Do not copy that choice into a long-lived production key workflow without considering how the key will be protected.

Check that the certificate and key exist, and inspect only the certificate's public identity:

$ ls -l cert.pem key.pem message.txt
$ openssl x509 -in cert.pem -noout -subject -dates
subject=CN = CMS guide test

File modes and dates vary. The key should not be world-readable. If this directory contains anything valuable, stop and correct its permissions before continuing.

3. Sign a file

Sign the message and write an opaque PEM CMS object:

$ openssl cms -sign -in message.txt -signer cert.pem \
    -inkey key.pem -nodetach -outform PEM -out signed.pem

-signer supplies the signing certificate and -inkey supplies its matching private key. -nodetach embeds the content in the signed object. Without it, the normal signed form is detached and verification needs the original content separately. PEM is text-safe and convenient for inspection; DER is binary.

Check that the output begins as a CMS object without dumping the private key:

$ sed -n '1,2p' signed.pem
-----BEGIN CMS-----
MI...

The base64 body is variable. Do not edit it by hand.

4. Verify the signature and recover content

Verify the signed message and write its content to a new file:

$ openssl cms -verify -in signed.pem -inform PEM \
    -noverify -out verified.txt
CMS Verification successful
$ cmp -- message.txt verified.txt
$ printf 'content matches\n'
content matches

-noverify is used here because the test certificate is self-signed and is not in a trusted CA store. It does not mean the signature bytes are ignored: CMS still checks that the content matches the signature. For real data, omit -noverify and provide the appropriate trust configuration, such as -CAfile or -CApath. A successful cryptographic check is not the same as a trusted identity unless the certificate chain and purpose are also validated.

If you sign with the default detached form, verify with the original content explicitly:

$ openssl cms -sign -in message.txt -signer cert.pem \
    -inkey key.pem -out detached.pem -outform PEM
$ openssl cms -verify -in detached.pem -inform PEM \
    -content message.txt -noverify -out verified-detached.txt

5. Encrypt for a recipient

Encrypt the file for the certificate owner. Keep the output separate from the input:

$ openssl cms -encrypt -in message.txt -recip cert.pem \
    -aes-256-cbc -outform PEM -out encrypted.pem

The recipient certificate is public and can be shared. The private key is not included in encrypted.pem; only the matching private key can decrypt the content. The CMS command uses a content-encryption key and protects that key for the recipient. The cipher is explicit here so a future reader does not have to infer the default from a different OpenSSL build.

Do not treat successful encryption as proof that the intended certificate was used. Inspect the certificate identity before encrypting sensitive data, and remember that the CMS command does not perform a revocation check for a recipient certificate.

6. Decrypt and compare

Supply the recipient certificate and matching private key, then compare the result with the original:

$ openssl cms -decrypt -in encrypted.pem -inform PEM \
    -recip cert.pem -inkey key.pem -out decrypted.txt
$ cmp -- message.txt decrypted.txt
$ printf 'decrypted content matches\n'
decrypted content matches

The output file is created or overwritten by -out. Before running a command that writes a real destination, check its path and make a backup. A safer replacement pattern is to decrypt to a new file, inspect it, then rename it during a controlled change window. Do not remove the original until the result has been checked.

7. Diagnose the usual failures

A certificate and private key that do not match usually cause signing or decryption to fail. Check the key and certificate pair with a harmless public-key comparison:

$ openssl x509 -in cert.pem -pubkey -noout \
    | openssl pkey -pubin -outform DER \
    | sha256sum
$ openssl pkey -in key.pem -pubout -outform DER \
    | sha256sum

The two hashes should match. If they do not, stop and locate the correct key rather than trying random files.

Format errors are another common trap. A PEM file needs -inform PEM when the operation does not infer it correctly; a binary CMS file needs -inform DER. A detached signature needs -content. If a message came from email, S/MIME line endings and MIME headers may matter, so preserve the original message and test on a copy.

For a readable structural view, print a CMS object without extracting its content:

$ openssl cms -in encrypted.pem -inform PEM \
    -cmsout -print | sed -n '1,20p'

Seeing an enveloped-data structure confirms that the file parses, not that the recipient is trusted or that decryption will succeed.