RSA in practice: key sizes, OAEP for encryption, PSS for signatures

What RSA is used for

RSA is a public key algorithm used for signatures and, less often today, for encrypting small values such as a symmetric key. It is not used to encrypt large data directly. 2048-bit keys are the minimum generally accepted today; 3072-bit keys match roughly 128-bit security and are common for new long-lived keys. Elliptic curve algorithms like Ed25519 and X25519 give similar security with much smaller keys.

Padding matters

Raw RSA is deterministic and malleable, so it is always used with padding. For encryption, use OAEP. The older PKCS#1 v1.5 encryption padding has a long history of padding oracle attacks. For signatures, PSS is the modern scheme; PKCS#1 v1.5 signatures are still widely used in certificates and are not broken, but PSS is preferred for new designs.

Lab: OAEP and PSS in Python

Generate a 3072-bit key, encrypt a 32-byte random key with OAEP-SHA256, decrypt it, and sign a message with PSS, then verify. Try encrypting 500 bytes and observe the error: RSA-OAEP can only encrypt a payload smaller than the key size minus padding overhead, which is why hybrid encryption exists.

Snippet

import os
from cryptography.hazmat.primitives.asymmetric import rsa, padding
from cryptography.hazmat.primitives import hashes

priv = rsa.generate_private_key(public_exponent=65537, key_size=3072)
pub = priv.public_key()
oaep = padding.OAEP(mgf=padding.MGF1(hashes.SHA256()), algorithm=hashes.SHA256(), label=None)

secret = os.urandom(32)
ct = pub.encrypt(secret, oaep)
assert priv.decrypt(ct, oaep) == secret

pss = padding.PSS(mgf=padding.MGF1(hashes.SHA256()), salt_length=padding.PSS.MAX_LENGTH)
sig = priv.sign(b'release 1.4.2', pss, hashes.SHA256())
pub.verify(sig, b'release 1.4.2', pss, hashes.SHA256())
print('ok')

try:
    pub.encrypt(os.urandom(500), oaep)
except ValueError as e:
    print('too large:', e)

Hybrid encryption

Real systems encrypt data with a random symmetric key using AES-GCM, then protect that key with the recipient’s public key, through RSA-OAEP or, more commonly now, an elliptic curve key agreement such as ECDH. TLS, age, and GPG all follow this pattern.

Diagram

RSA choices

Safe defaults for new work.

Key size
2048 minimum; 3072 for long-lived keys.
Encryption padding
OAEP with SHA-256; avoid PKCS#1 v1.5 encryption.
Signature padding
PSS preferred; v1.5 still common in certificates.
Large data
Hybrid: AES-GCM for data, RSA or ECDH for the key.

Check yourself

1. Which padding should be used for RSA encryption in new designs?

2. Which RSA signature scheme is preferred for new designs?

3. Why does RSA-OAEP fail on a 500-byte message with a 3072-bit key?