Sign, Verify and Encrypt Safely with openssl pkeyutl

You will finish with working commands for signing and verifying a file, encrypting a small message with RSA-OAEP, and deriving the same shared secret on two sides of an X25519 exchange. The examples use the installed OpenSSL 3.6.1 command. The local manpage is labelled 3.0.13, so the exact options available on an older host may differ.

Allow about fifteen minutes. You need a shell and OpenSSL. The examples use temporary files under /tmp and do not need elevated privileges. The private keys created here are for testing only. Do not reuse them for an account, service or production data.

1. Check the installed version and command

Start by checking the binary that will actually run:

$ command -v openssl
/home/linuxbrew/.linuxbrew/bin/openssl
$ openssl version
OpenSSL 3.6.1 27 Jan 2026

Then inspect the operation names supported by this installation:

$ openssl pkeyutl -help
Usage: pkeyutl [options]

Checkpoint: make sure pkeyutl is present before copying any later command. The command is a low-level public-key operation tool. It does not automatically choose a complete application protocol for you.

2. Create a disposable RSA key pair

Use a private key for signing and decryption, and derive a public key for verification and encryption:

$ work=$(mktemp -d /tmp/pkeyutl-guide.XXXXXX)
$ openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 \
    -out "$work/rsa-key.pem"
$ openssl pkey -in "$work/rsa-key.pem" -pubout \
    -out "$work/rsa-pub.pem"
$ printf '%s' 'pkeyutl guide test' > "$work/message.txt"

This creates an unencrypted PEM private key. Anyone who can read that file can use it, so keep the directory private and remove it after the exercise. If you are working with an existing encrypted key, use -passin rather than putting a secret directly in a shell command where it may be exposed through history or process inspection.

3. Sign the data and verify the signature

For a normal file, use -rawin and name the digest explicitly. This tells pkeyutl to hash the input before signing. The command's older interface can also accept a precomputed digest, but that is a common source of accidental double hashing or incorrect input lengths.

$ openssl pkeyutl -sign -rawin -digest sha256 \
    -in "$work/message.txt" -inkey "$work/rsa-key.pem" \
    -out "$work/signature.bin"
$ openssl pkeyutl -verify -rawin -digest sha256 \
    -in "$work/message.txt" -sigfile "$work/signature.bin" \
    -inkey "$work/rsa-pub.pem" -pubin
Signature Verified Successfully

-sign requires the private key. -verify reads the signature from -sigfile and can use a public key with -pubin. A successful verification returns status 0:

$ printf 'verification status: %s\n' "$?"
verification status: 0

Checkpoint: change the message and run the verify command again. It should fail. That is a useful smoke test because it proves the signature is tied to the file, rather than merely proving that the key files can be read.

4. Encrypt a small message with RSA-OAEP

Encrypt with the public key and decrypt with the private key. Use OAEP and set its digests explicitly so both sides have the same parameters:

$ openssl pkeyutl -encrypt -in "$work/message.txt" \
    -inkey "$work/rsa-pub.pem" -pubin -out "$work/cipher.bin" \
    -pkeyopt rsa_padding_mode:oaep \
    -pkeyopt rsa_oaep_md:sha256 \
    -pkeyopt rsa_mgf1_md:sha256
$ openssl pkeyutl -decrypt -in "$work/cipher.bin" \
    -inkey "$work/rsa-key.pem" -out "$work/plain.txt" \
    -pkeyopt rsa_padding_mode:oaep \
    -pkeyopt rsa_oaep_md:sha256 \
    -pkeyopt rsa_mgf1_md:sha256
$ cmp "$work/message.txt" "$work/plain.txt"
$ printf '%s\n' 'decrypted bytes match the input'

cmp prints nothing when the files match, so the final message is the visible checkpoint. RSA encryption is for short inputs and key transport, not for arbitrary large files. For larger data, use authenticated symmetric encryption and protect or exchange the symmetric key with an appropriate protocol.

Do not substitute RSA PKCS#1 v1.5 encryption because a command happened to accept it. The manpage describes OAEP as the RSA mode for encryption and decryption. Decryption errors can also reveal sensitive information if an application exposes them carelessly, so avoid building a network protocol around raw command output without a reviewed design.

5. Derive a shared secret with X25519

Key agreement is different from encryption. Each side keeps a private key, publishes the matching public key, and derives the same binary secret from its own private key plus the other side's public key:

$ openssl genpkey -algorithm X25519 -out "$work/alice.pem"
$ openssl genpkey -algorithm X25519 -out "$work/bob.pem"
$ openssl pkey -in "$work/alice.pem" -pubout -out "$work/alice-pub.pem"
$ openssl pkey -in "$work/bob.pem" -pubout -out "$work/bob-pub.pem"
$ openssl pkeyutl -derive -inkey "$work/alice.pem" \
    -peerkey "$work/bob-pub.pem" -out "$work/alice-secret.bin"
$ openssl pkeyutl -derive -inkey "$work/bob.pem" \
    -peerkey "$work/alice-pub.pem" -out "$work/bob-secret.bin"
$ cmp "$work/alice-secret.bin" "$work/bob-secret.bin"
$ wc -c < "$work/alice-secret.bin"
32

The matching cmp result and 32-byte length are the checkpoints. The derived bytes are key material, not a human-readable password and not a complete authenticated session. A real protocol still needs authentication, context binding and a safe key derivation or encryption step before using the secret.

6. Avoid the defaults that cause most failures

• Input is not normally hashed. Without -rawin, pkeyutl generally consumes the bytes supplied to it directly. The -digest option describes or selects digest handling for signing operations; it does not make every command a general-purpose file hashing utility.
• Keys have roles. A private key is needed for signing, decryption and the local side of derivation. A public key is suitable for verification or encryption. Use -pubin when the input file contains a public key.
• Parameters must match. OAEP digest and MGF1 digest settings used for encryption must be repeated for decryption. A mismatch commonly appears as a decryption failure even when both key files are correct.
• Binary output is easy to misread. Signatures, ciphertext and derived secrets are binary. Keep them in files or request -hexdump for inspection. Do not paste binary output into a terminal or treat a hex dump as the original bytes.
• Algorithms restrict operations. X25519 supports derivation, while Ed25519 and Ed448 support signing and verification. Options valid for RSA are not automatically valid for every key type.

7. Remove the test material

These examples change only the temporary directory. When you have finished, inspect the exact path before removing it:

$ printf 'review this path before removal: %s\n' "$work"
$ find "$work" -maxdepth 1 -type f -print

After confirming that it contains only the disposable files from this exercise, remove that directory with your normal local cleanup procedure. Do not run a broad recursive removal command against a variable you have not inspected. If the files are needed for another test, keep the directory private and delete the private keys when you no longer need them.