Use openssl rsautl Safely for Small RSA Tests

You will finish with a small, repeatable RSA round trip: encrypt data with a public key, decrypt it with the private key, and recover signed data for comparison. The examples use the installed OpenSSL 3.6.1 command. They also make the important boundary clear: rsautl is deprecated in OpenSSL 3.0 and should not be the basis of a new integration.

Allow about fifteen minutes. You need OpenSSL, an RSA private key and its matching public key. This guide writes temporary test artefacts in a working directory, does not require sudo, and does not alter a service or system configuration. Do not use a real production private key for a first test.

1. Check the installed command

Start by checking the binary and version. These are ordinary read-only commands:

$ command -v openssl
/usr/bin/openssl
$ openssl version
OpenSSL 3.6.1 27 Jan 2026

Your path and version may differ. Confirm the local option spelling before copying a script:

$ openssl rsautl -help
Usage: rsautl [options]

 -sign                    Sign with private key
 -verify                  Verify with public key
 -encrypt                 Encrypt with public key
 -decrypt                 Decrypt with private key

On OpenSSL 3.x the command also prints a deprecation warning. That warning is expected. For new code, read the equivalent openssl pkeyutl workflow and use that instead. Use rsautl when you are reproducing an existing command, testing an old protocol or inspecting compatible RSA data.

2. Prepare keys and a small input

If you already have suitable test keys, use their paths and skip key generation. Otherwise create a disposable 2048-bit RSA key and derive its public half:

$ openssl genpkey -algorithm RSA \
    -pkeyopt rsa_keygen_bits:2048 \
    -out rsa-test-private.pem
$ openssl pkey -in rsa-test-private.pem -pubout -out rsa-test-public.pem
$ printf 'rsautl local test\n' > message.txt
$ ls -l rsa-test-private.pem rsa-test-public.pem message.txt

Keep the private key readable only by its intended owner if it contains anything valuable:

$ chmod 600 rsa-test-private.pem

Do not send the private key to another person or put it in a public repository. The public key is the one you distribute for encryption or verification.

3. Encrypt with the public key

Encryption uses -encrypt, the public key and -pubin. Write to a new output file so that a failed command cannot truncate an existing ciphertext:

$ openssl rsautl -encrypt -pubin \
    -inkey rsa-test-public.pem \
    -in message.txt -out message.bin
The command rsautl was deprecated in version 3.0. Use 'pkeyutl' instead.

The warning is diagnostic output, not ciphertext. Check that the result is non-empty:

$ test -s message.bin && echo 'ciphertext created'
ciphertext created
$ wc -c message.txt message.bin
  18 message.txt
 256 message.bin

The exact plaintext size and ciphertext size depend on your input and key. RSA operates on small blocks. It is not a general replacement for a file encryption tool, and the input must fit the selected key and padding limits.

4. Decrypt and compare the result

Decryption needs the private key. The command writes recovered plaintext to a new file:

$ openssl rsautl -decrypt \
    -inkey rsa-test-private.pem \
    -in message.bin -out recovered.txt
The command rsautl was deprecated in version 3.0. Use 'pkeyutl' instead.
$ cmp --silent message.txt recovered.txt && echo 'decrypt round trip: OK'
decrypt round trip: OK

cmp is the useful check here because it compares bytes, including a final newline. Do not judge success from a readable-looking terminal display. If cmp reports a difference, check that the public and private keys belong together and that you did not mix files from two test runs.

There is no persistent undo for this operation: the command only reads the key and ciphertext and writes the named output. If recovered.txt is sensitive, protect it or remove it after inspection using your normal file-retention process.

5. Sign and recover a small message

rsautl -sign uses the private key and produces a raw RSA signature-style result. In this command's model, -verify recovers the input data from that result; it is not the usual detached signature verification interface that compares a message and signature separately:

$ openssl rsautl -sign \
    -inkey rsa-test-private.pem \
    -in message.txt -out recovered-signature.bin
The command rsautl was deprecated in version 3.0. Use 'pkeyutl' instead.
$ openssl rsautl -verify \
    -inkey rsa-test-public.pem -pubin \
    -in recovered-signature.bin -out signed-message.txt
The command rsautl was deprecated in version 3.0. Use 'pkeyutl' instead.
$ cmp --silent message.txt signed-message.txt && echo 'sign/verify recovery: OK'
sign/verify recovery: OK

The public key must be marked with -pubin because it is a public-key PEM file. If your input is a certificate containing an RSA public key, use -certin instead. A private key is the default interpretation for -inkey, so omitting -pubin is a common source of confusing failures.

6. Choose padding deliberately

Without a padding option, the installed command uses PKCS#1 v1.5 padding. The other documented choices are -oaep, -x931 and -raw. For signing, only -pkcs and -raw are supported. Both sides of an encryption round trip must choose the same padding:

$ openssl rsautl -encrypt -oaep -pubin \
    -inkey rsa-test-public.pem \
    -in message.txt -out oaep.bin
$ openssl rsautl -decrypt -oaep \
    -inkey rsa-test-private.pem \
    -in oaep.bin -out oaep-recovered.txt
$ cmp --silent message.txt oaep-recovered.txt && echo 'OAEP round trip: OK'
OAEP round trip: OK

Use OAEP when you must reproduce a legacy rsautl encryption protocol that specifies it. Do not casually change padding in an existing protocol: the peer must make the same choice. Avoid -raw unless you are examining a deliberately constructed RSA block or implementing a precisely specified compatibility test.

There is a subtle safety trap around PKCS#1 v1.5 decryption. The manual says that a padding-check failure may not be returned as an error because of protection against Bleichenbacher attacks. Do not treat a successful exit status from a v1.5 decrypt as proof that untrusted ciphertext was valid. If you need to inspect a block, use -raw only in a controlled diagnostic and inspect the returned bytes yourself.

7. Inspect output without changing it

For a controlled test, -hexdump shows output bytes instead of writing normal binary data. This is useful for understanding a test vector, not for validating an untrusted message:

$ openssl rsautl -verify -raw -hexdump \
    -inkey rsa-test-public.pem -pubin \
    -in recovered-signature.bin
0000 - 00 01 ...

The offsets and bytes depend on the key and message, so do not compare them with a copied example. -asn1parse can parse the recovered output when it is an ASN.1 structure, particularly alongside -verify. It does not make arbitrary bytes safe or prove that a certificate is trustworthy.

8. Clean up and move new work to pkeyutl

Once the test is complete, retain only artefacts you have a reason to keep. The private key, plaintext and recovered files may all contain sensitive data. Remove disposable files with your normal approved cleanup procedure, and securely handle any copy of a real private key according to its key-management policy.

For a new automation task, stop at this point and translate the requirement to openssl pkeyutl. Its interface is not a mechanical one-for-one rename, so read its installed manual and test the exact padding and key format with the other endpoint. Record the OpenSSL version and the chosen padding in the protocol notes.