Inspect Linux Kernel Keys Safely with /proc/keys and /proc/key-users

You will inspect the kernel key-management views exposed by /proc/keys and /proc/key-users, then relate what you see to a user ID's key and payload quotas. The commands are read-only and do not create, revoke or alter keys. Allow about 10 minutes if you only need a snapshot, or longer if you are investigating a service that uses credentials in the kernel keyring.

This guide uses Linux 6.8.0-139-generic and the installed manpages package version 6.7-2. The local proc_key-users(5) page is a short pointer to the same interface documented by proc_keys(5); it says both files have existed since Linux 2.6.10 and directs you to keyrings(7) for their meaning.

1. Confirm the two proc files exist

Start by checking the paths and their permissions. Do this as the account that is seeing the problem, because the contents of /proc/keys are filtered for the reading thread.

$ ls -l /proc/keys /proc/key-users
-r--r--r-- 1 root root 0 Sep 26 12:57 /proc/key-users
-r--r--r-- 1 root root 0 Sep 26 12:57 /proc/keys

The zero length shown by ls is normal for a procfs view. It does not mean that the file has no records. Read the files directly:

$ sed -n '1,8p' /proc/keys
$ sed -n '1,12p' /proc/key-users

Checkpoint: if either path is missing, stop here and check that procfs is mounted and that the running kernel provides key retention support. Do not create replacement files under /proc.

2. Read the visible keys

Each line in /proc/keys describes one key that the reading thread can view. The list is not necessarily a complete inventory. A key must grant view permission to the reader, and an active Linux Security Module can filter it further. Being listed also does not mean that the process possesses the key.

$ sed -n '1,6p' /proc/keys
027558b5 I--Q---     7 perm 3f030000  1004  1004 keyring   _ses: 1
09c9bc02 I--Q---     4 perm 3f030000  1004   987 keyring   _ses: 1
20f4aa03 I--Q---     4 perm 1f3f0000  1004 65534 keyring   _uid.1004: empty
2146c11c IR-Q---   185 expd 3f030000  1004  1004 keyring   _ses: empty
3d597a9a I--Q---     1 perm 1f3f0000  1004 65534 keyring   _uid_ses.1004: 1
3ece5cdd I--Q---    30 perm 3f030000  1004  1004 keyring   _ses: 1

The columns are, in order, the key serial number in hexadecimal, state flags, payload or link count, expiry state, permission mask, owner UID, owner GID, key type and description. The exact records vary with logged-in users, services and kernel activity, so treat the sample as a shape to recognise rather than output to match.

For the flags, I means instantiated, R revoked and D dead. The word after the count is normally perm for a valid key or expd when it is expired. A keyring type describes the number of keys linked to that keyring, or reports empty. A key of type user reports its payload size in bytes instead.

The hexadecimal permission mask is four six-bit sets for possessor, user, group and other. Within each set, the documented bits are view, read, write, search, link and setattr, represented by 0x01 through 0x20. Avoid treating the mask as an ordinary Unix mode such as 0644; the key permission model has an additional possessor category.

3. Map numeric owners to accounts

Use the UID and GID columns to identify ownership, but keep the numeric values in your notes. Name-service lookups can fail or change, while the kernel record still refers to the numeric identity.

$ getent passwd 1004
andy:x:1004:1004::/home/andy:/bin/bash
$ getent group 1004

A GID of -1 has a documented special meaning: the key has no group ID, which can occur for keys created by the kernel. A description such as _ses, _uid.1004 or _uid_ses.1004 identifies a keyring, not a file in your home directory. Do not try to edit it with a text editor.

4. Check the per-UID totals and limits

/proc/key-users contains one line for each UID with at least one key. Its fields are easier to compare as ratios:

$ sed -n '1,12p' /proc/key-users
    0:   238 237/237 207/1000000 6740/25000000
 1004:     6 6/6 6/200 60/20000

Read the line as uid: usage nkeys/nikeys qnkeys/maxkeys qnbytes/maxbytes. The first number after the colon is a kernel-internal usage count. nkeys/nikeys is the total and instantiated-key count. qnkeys/maxkeys is the quota count and maximum number of keys. qnbytes/maxbytes is payload bytes in use and the maximum allowed.

In the example, UID 1004 owns six keys, all six are instantiated, and the quota counters show six of a possible 200 keys and 60 of 20,000 payload bytes. The limits are not a count of visible lines in /proc/keys: visibility is permission-filtered, while these totals describe keys owned by the UID.

Checkpoint: if a program reports that it cannot add a key, compare its UID's quota line before changing anything. A nearly full quota is evidence to investigate, not a reason to delete an unfamiliar key.

5. Compare a service view without changing state

Run the same reads as the service account when diagnosing a daemon. Use an existing, harmless identity switch approved by your normal operating procedure. If you need root to enter that account's environment, the command is elevated and should be treated as sensitive because it can expose key metadata to your terminal.

$ sudo -u SERVICE_USER sh -c 'printf "%s\n" "visible keys:"; sed -n "1,8p" /proc/keys; printf "%s\n" "UID totals:"; sed -n "1,20p" /proc/key-users'
visible keys:
...
UID totals:
...

Replace SERVICE_USER with a real account name. Do not paste secrets into the command line, and do not assume that running the inspection as root reveals every key: LSM policy and namespace details can still affect what is visible. Compare the UID, key type and description, then check the service's own logs for the operation that failed.

6. Keep inspection separate from repair

These files are observation points, not administrative interfaces. The displayed payload size is metadata; /proc/keys does not provide the key payload for you to copy out. Keyrings can hold authentication and encryption-related data, so avoid sending complete output to an issue tracker or chat channel without reviewing descriptions and ownership first.

Do not change /proc/sys/kernel/keys limits as a first response. Those controls affect the whole system or a class of users and may only hide the underlying leak or failed cleanup. If a quota is genuinely the issue, record the current values, identify the process creating keys, and use the relevant key-management tooling or service configuration under a planned change. There is no undo step for this guide because every command above only reads state.