Track Key Quotas per UID with /proc/key-users

When a daemon suddenly refuses to add a new key, /proc/key-users tells you whether it hit its keyring quota before you touch anything else. Alongside /proc/keys it gives you the kernel's key-management view, and you can relate what you see to a user ID's key and payload quotas. The commands are read-only and do not create, revoke or alter keys. Allow about 10 minutes for a snapshot, longer if you are investigating a service that uses credentials in the kernel keyring.

This guide uses Linux 6.8.0-139-generic and the installed manpages package version 6.7-2. The local proc_key-users(5) page is a short pointer to the same interface documented by proc_keys(5); it says both files have existed since Linux 2.6.10 and directs you to keyrings(7) for their meaning.

1. Confirm the two proc files exist

Start by checking the paths and their permissions. Do this as the account that is seeing the problem, because the contents of /proc/keys are filtered for the reading thread.

$ ls -l /proc/keys /proc/key-users
-r--r--r-- 1 root root 0 Sep 26 12:57 /proc/key-users
-r--r--r-- 1 root root 0 Sep 26 12:57 /proc/keys

The zero length shown by ls is normal for a procfs view. It does not mean the file has no records. Read the files directly:

$ sed -n '1,8p' /proc/keys
$ sed -n '1,12p' /proc/key-users

2. Read the visible keys

Each line in /proc/keys describes one key that the reading thread can view. The list is not necessarily a complete inventory: a key must grant view permission to the reader, and an active Linux Security Module can filter it further. Being listed also does not mean the process possesses the key.

$ sed -n '1,6p' /proc/keys
027558b5 I--Q---     7 perm 3f030000  1000  1000 keyring   _ses: 1
09c9bc02 I--Q---     4 perm 3f030000  1000   987 keyring   _ses: 1
20f4aa03 I--Q---     4 perm 1f3f0000  1000 65534 keyring   _uid.1000: empty
2146c11c IR-Q---   185 expd 3f030000  1000  1000 keyring   _ses: empty
3d597a9a I--Q---     1 perm 1f3f0000  1000 65534 keyring   _uid_ses.1000: 1
3ece5cdd I--Q---    30 perm 3f030000  1000  1000 keyring   _ses: 1

The columns, in order: key serial number in hexadecimal, state flags, payload or link count, expiry state, permission mask, owner UID, owner GID, key type and description. The exact records vary with logged-in users, services and kernel activity, so treat the sample as a shape to recognise rather than output to match.

• Flags. I means instantiated, R revoked, D dead.
• Expiry word. Normally perm for a valid key or expd when it is expired.
• Keyring type. Reports the number of keys linked to that keyring, or empty.
• User type. A key of type user reports its payload size in bytes instead.

The hexadecimal permission mask is four six-bit sets for possessor, user, group and other. Within each set, the documented bits are view, read, write, search, link and setattr, represented by 0x01 through 0x20. Avoid treating the mask as an ordinary Unix mode such as 0644: the key permission model has an additional possessor category.

3. Map numeric owners to accounts

Use the UID and GID columns to identify ownership, but keep the numeric values in your notes. Name-service lookups can fail or change, while the kernel record still refers to the numeric identity.

$ getent passwd 1000
alice:x:1000:1000::/home/alice:/bin/bash
$ getent group 1000

A GID of -1 has a documented special meaning: the key has no group ID, which can occur for keys created by the kernel. A description such as _ses, _uid.1000 or _uid_ses.1000 identifies a keyring, not a file in your home directory. Do not try to edit it with a text editor.

4. Check the per-UID totals and limits

/proc/key-users contains one line for each UID with at least one key. Its fields are easier to compare as ratios:

$ sed -n '1,12p' /proc/key-users
    0:   238 237/237 207/1000000 6740/25000000
 1000:     6 6/6 6/200 60/20000

Read the line as uid: usage nkeys/nikeys qnkeys/maxkeys qnbytes/maxbytes.

• usage. A kernel-internal usage count, the first number after the colon.
• nkeys/nikeys. The total and instantiated-key count.
• qnkeys/maxkeys. The quota count and maximum number of keys.
• qnbytes/maxbytes. Payload bytes in use and the maximum allowed.

In the example, UID 1000 owns six keys, all six instantiated, and the quota counters show six of a possible 200 keys and 60 of 20,000 payload bytes. The limits are not a count of visible lines in /proc/keys: visibility is permission-filtered, while these totals describe keys owned by the UID.

5. Compare a service view without changing state

Run the same reads as the service account when diagnosing a daemon. Use an existing, harmless identity switch approved by your normal operating procedure. If you need root to enter that account's environment, the command is elevated and should be treated as sensitive because it can expose key metadata to your terminal.

$ sudo -u SERVICE_USER sh -c 'printf "%s\n" "visible keys:"; sed -n "1,8p" /proc/keys; printf "%s\n" "UID totals:"; sed -n "1,20p" /proc/key-users'
visible keys:
...
UID totals:
...

Replace SERVICE_USER with a real account name. Do not paste secrets into the command line, and do not assume running the inspection as root reveals every key: LSM policy and namespace details can still affect what is visible. Compare the UID, key type and description, then check the service's own logs for the operation that failed.

Common traps

Do not change /proc/sys/kernel/keys limits as a first response. Those controls affect the whole system or a class of users and may only hide the underlying leak or failed cleanup. If a quota is genuinely the issue, record the current values, identify the process creating keys, and use the relevant key-management tooling or service configuration under a planned change. There is no undo step for this guide because every command above only reads state.

Done means

• Files confirmed. You confirmed that /proc/keys and /proc/key-users are available on the running kernel.
• Correct account used. You inspected keys as the affected account and treated the list as permission-filtered.
• Columns understood. You can distinguish a key ID, owner UID and GID, type, description and permission mask.
• Quotas read correctly. You read the per-UID key and payload quotas without confusing them with visible-key counts.
• Nothing changed. You kept key metadata private and made no changes to keys, services or kernel limits.