Use openssl rsautl Safely for Small RSA Tests

rsautl turns up in plenty of old scripts, but OpenSSL deprecated it in version 3.0, so this guide draws that line first. Once that is clear, you will finish with a small, repeatable RSA round trip: encrypt data with a public key, decrypt it with the private key, and recover signed data for comparison. The examples use the installed OpenSSL 3.6.1 command.

Allow about fifteen minutes. You need OpenSSL, an RSA private key and its matching public key. This guide writes temporary test artefacts in a working directory, does not require sudo, and does not alter a service or system configuration. Do not use a real production private key for a first test.

1. Check the installed command

Start by checking the binary and version. These are ordinary read-only commands:

$ command -v openssl
/usr/bin/openssl
$ openssl version
OpenSSL 3.6.1 27 Jan 2026

Your path and version may differ. Confirm the local option spelling before copying anyone's script:

$ openssl rsautl -help
Usage: rsautl [options]

 -sign                    Sign with private key
 -verify                  Verify with public key
 -encrypt                 Encrypt with public key
 -decrypt                 Decrypt with private key

On OpenSSL 3.x the command also prints a deprecation warning on every run. That is expected, not a sign something is broken. For new code, read the equivalent openssl pkeyutl workflow and use that instead. Reach for rsautl only when reproducing an existing command, testing an old protocol, or inspecting compatible RSA data.

2. Prepare keys and a small input

Already have suitable test keys? Use their paths and skip key generation. Otherwise create a disposable 2048-bit RSA key and derive its public half:

$ openssl genpkey -algorithm RSA \
    -pkeyopt rsa_keygen_bits:2048 \
    -out rsa-test-private.pem
$ openssl pkey -in rsa-test-private.pem -pubout -out rsa-test-public.pem
$ printf 'rsautl local test\n' > message.txt
$ ls -l rsa-test-private.pem rsa-test-public.pem message.txt

Keep the private key readable only by its intended owner if it contains anything valuable:

$ chmod 600 rsa-test-private.pem

Never hand the private key to another person or drop it into a public repository. The public key is the one you distribute for encryption or verification.

3. Encrypt with the public key

Encryption uses -encrypt, the public key and -pubin. Write to a new output file so a failed command cannot truncate an existing ciphertext:

$ openssl rsautl -encrypt -pubin \
    -inkey rsa-test-public.pem \
    -in message.txt -out message.bin
The command rsautl was deprecated in version 3.0. Use 'pkeyutl' instead.

That warning is diagnostic output, not ciphertext. Check the result is non-empty:

$ test -s message.bin && echo 'ciphertext created'
ciphertext created
$ wc -c message.txt message.bin
  18 message.txt
 256 message.bin

Exact plaintext and ciphertext sizes depend on your input and key. RSA operates on small blocks; it is not a general replacement for a file encryption tool, and the input must fit the selected key and padding limits.

4. Decrypt and compare the result

Decryption needs the private key. The command writes recovered plaintext to a new file:

$ openssl rsautl -decrypt \
    -inkey rsa-test-private.pem \
    -in message.bin -out recovered.txt
The command rsautl was deprecated in version 3.0. Use 'pkeyutl' instead.
$ cmp --silent message.txt recovered.txt && echo 'decrypt round trip: OK'
decrypt round trip: OK

cmp earns its place here because it compares bytes, including a final newline. Do not judge success from a readable-looking terminal display. A difference from cmp means checking that the public and private keys actually belong together, and that you have not mixed files from two test runs.

There is no persistent undo for this operation: the command only reads the key and ciphertext and writes the named output. If recovered.txt is sensitive, protect it or remove it after inspection using your normal file-retention process.

5. Sign and recover a small message

rsautl -sign uses the private key and produces a raw RSA signature-style result. Here -verify recovers the input data from that result: it is not the usual detached signature verification interface that compares a message and signature separately.

$ openssl rsautl -sign \
    -inkey rsa-test-private.pem \
    -in message.txt -out recovered-signature.bin
The command rsautl was deprecated in version 3.0. Use 'pkeyutl' instead.
$ openssl rsautl -verify \
    -inkey rsa-test-public.pem -pubin \
    -in recovered-signature.bin -out signed-message.txt
The command rsautl was deprecated in version 3.0. Use 'pkeyutl' instead.
$ cmp --silent message.txt signed-message.txt && echo 'sign/verify recovery: OK'
sign/verify recovery: OK

Mark the public key with -pubin because it is a public-key PEM file. If your input is a certificate containing an RSA public key, use -certin instead. A private key is the default interpretation for -inkey, so a missing -pubin is a common source of confusing failures.

6. Choose padding deliberately

Without a padding option, the installed command uses PKCS#1 v1.5 padding. The other documented choices are -oaep, -x931 and -raw. For signing, only -pkcs and -raw are supported. Both sides of an encryption round trip must agree on padding:

$ openssl rsautl -encrypt -oaep -pubin \
    -inkey rsa-test-public.pem \
    -in message.txt -out oaep.bin
$ openssl rsautl -decrypt -oaep \
    -inkey rsa-test-private.pem \
    -in oaep.bin -out oaep-recovered.txt
$ cmp --silent message.txt oaep-recovered.txt && echo 'OAEP round trip: OK'
OAEP round trip: OK

Use OAEP when reproducing a legacy rsautl encryption protocol that specifies it. Do not casually change padding in an existing protocol; the peer must make the same choice. Avoid -raw unless you are examining a deliberately constructed RSA block or implementing a precisely specified compatibility test.

There is a subtle safety trap around PKCS#1 v1.5 decryption. The manual says a padding-check failure may not be returned as an error, deliberately, as protection against Bleichenbacher attacks. A successful exit status from a v1.5 decrypt is not proof that untrusted ciphertext was valid. To inspect a block, use -raw only in a controlled diagnostic and check the returned bytes yourself.

7. Inspect output without changing it

For a controlled test, -hexdump shows output bytes instead of writing normal binary data. Useful for understanding a test vector, not for validating an untrusted message:

$ openssl rsautl -verify -raw -hexdump \
    -inkey rsa-test-public.pem -pubin \
    -in recovered-signature.bin
0000 - 00 01 ...

Offsets and bytes depend on the key and message, so do not compare them against a copied example. -asn1parse can parse the recovered output when it is an ASN.1 structure, particularly alongside -verify. It does not make arbitrary bytes safe, and it does not prove a certificate is trustworthy.

8. Clean up and move new work to pkeyutl

Once the test is complete, keep only artefacts you have a reason to keep. The private key, plaintext and recovered files can all contain sensitive data. Remove disposable files with your normal approved cleanup procedure, and handle any copy of a real private key according to its key-management policy.

For a new automation task, stop here and translate the requirement to openssl pkeyutl. Its interface is not a mechanical one-for-one rename, so read its installed manual and test the exact padding and key format with the other endpoint. Record the OpenSSL version and the chosen padding in your protocol notes.