Why Your Password Hash Has a Pepper Problem Nobody Mentions

Adding a pepper to password hashes takes five minutes. Changing the pepper later is where it goes wrong. Most write-ups stop at "keep a secret outside the database", and never ask what happens the day that secret leaks or has to be replaced.

The short answer: with the usual HMAC design, you cannot rotate it without every user logging in again. This post shows why, and a small Go scheme that fixes it.

What a pepper is for (and what it is not)

A pepper is a secret key mixed into the hash that lives outside the database. Salts are stored next to the hash and stop precomputed tables. A pepper is different: an attacker who only has a database dump cannot even start guessing without it.

That covers a fairly specific set of breaches:

SQL injection that reads tables but not your config
A stolen or misplaced database backup
A read-only replica exposed by mistake

It does not help if the attacker gets code execution on the application server, because the app has to know the pepper to verify logins. A pepper is a second lock on a different door, not a replacement for a good hash such as Argon2id.

The usual design and its trap

The common advice is to compute the slow hash, then wrap it with HMAC-SHA256 using the pepper as the key. Stored value: HMAC(pepper, argon2(password, salt)). It is sound, and it is stdlib-only for the HMAC part.

Now try to rotate. To move from pepper A to pepper B you need the inner Argon2 output. HMAC is one-way, so all you hold is the wrapped value. The inner hash is gone. The only time you know it is when the user types their password.

So the honest options are:

Wait for every user to log in, then re-wrap. Dormant accounts stay on the old pepper for ever.
Force password resets for everyone. Unpleasant.
Never rotate, and hope the pepper never leaks.

Option three is what most systems quietly pick. A secret you cannot change is a liability, not a control.

Wrap the wrapper

The fix is almost embarrassingly small. Do not go back to the inner hash; apply the new pepper on top of the stored value, and record which peppers were applied, in order. Verification replays the chain.

Hang on, why does that work? Because verification recomputes the Argon2 output from the password anyway. It can then apply pepper A, then pepper B, and compare. Nothing needs to be unwrapped. Rotation becomes a batch job over the table that needs no passwords.

package pepper

import (
	"crypto/hmac"
	"crypto/sha256"
	"crypto/subtle"

	"golang.org/x/crypto/argon2"
)

// Record is what goes in the database.
type Record struct {
	Salt    []byte
	Tag     []byte
	Peppers []int // key IDs, applied in this order
}

func mac(key, msg []byte) []byte {
	m := hmac.New(sha256.New, key)
	m.Write(msg)
	return m.Sum(nil)
}

func slow(pw string, salt []byte) []byte {
	return argon2.IDKey([]byte(pw), salt, 3, 64*1024, 2, 32)
}

func replay(h []byte, ids []int, keys map[int][]byte) ([]byte, bool) {
	for _, id := range ids {
		k, ok := keys[id]
		if !ok {
			return nil, false
		}
		h = mac(k, h)
	}
	return h, true
}

func Hash(pw string, salt []byte, id int, keys map[int][]byte) Record {
	ids := []int{id}
	tag, _ := replay(slow(pw, salt), ids, keys)
	return Record{Salt: salt, Tag: tag, Peppers: ids}
}

func Verify(pw string, r Record, keys map[int][]byte) bool {
	got, ok := replay(slow(pw, r.Salt), r.Peppers, keys)
	return ok && subtle.ConstantTimeCompare(got, r.Tag) == 1
}

// Rotate needs no password: it wraps the stored tag in a new pepper.
func Rotate(r Record, id int, keys map[int][]byte) Record {
	r.Tag = mac(keys[id], r.Tag)
	r.Peppers = append(append([]int(nil), r.Peppers...), id)
	return r
}

The Argon2 parameters here are placeholders; choose your own properly. I have kept the code to the pepper logic and left out salt generation and storage encoding.

Quick detour: why the IDs matter

The Peppers slice looks like bookkeeping, but it is doing real work. Without it, verification has to guess how many times a given row has been wrapped. With it, you can run a rotation halfway, crash, restart, and every row still verifies, because each row says exactly what was applied to it.

It also lets you answer a useful question after an incident: "which accounts still depend on pepper 1?" That is a single query, and it tells you when the old key can finally be deleted.

The chain grows, so collapse it on login

Each rotation adds one HMAC to every verification. That is microseconds next to Argon2, so a chain of three or four is harmless. Still, you do not want it growing for ever.

The clean answer is the same trick used for upgrading hash parameters at login (there is a post on rehashing at login, if you want the details). When a user logs in successfully, you have the password, so rebuild the record with a fresh salt and just the current pepper. Active users collapse to a chain of one. Dormant users stay wrapped, but still verify.

What this does and does not buy you

You can rotate on a schedule or after a suspected leak, without touching user passwords.
If pepper A leaks, wrapping with B does not undo the leak for an attacker who has both the dump and A, because A is still in the chain. It only stops future dumps from being useful without B, so the real protection comes from the chain containing at least one key that is still secret.
Keys still need a home that is not the database: a secrets manager, a file readable only by the service, or an HSM if you have one.

The alternative some teams prefer is encrypting the stored hash with an AEAD key instead of HMAC-ing it. That is also rotatable offline (decrypt, re-encrypt), and arguably more honest about what you are doing. It costs you nonce management, which is its own way to go wrong. The wrapping chain needs only HMAC, and I find that easier to get right.

One last caveat: a pepper is defence in depth for a narrow failure. Spend your effort on the hash parameters and on not leaking the database first; add the pepper when you have a plan for changing it.