Articles / Security

831 words

  • linux
  • proc
  • environment-variables
  • secrets
  • systemd
  • go

Your Encryption Key in an Environment Variable Is Readable via /proc

Putting a key in an environment variable feels tidy: it is not on the command line, not in a config file, not in git. Then you run one command and the kernel hands it straight back to anyone who may read it.

$ APP_KEY=correct-horse-battery sleep 300 &
$ tr '\0' '\n' < /proc/$!/environ | grep APP_KEY
APP_KEY=correct-horse-battery

That is the whole trick. The rest of this post is about who can do it, why unsetenv does not save you, and what to do instead.

What /proc/PID/environ actually is

Every process gets a copy of its environment placed on its stack by execve. The kernel remembers where that block starts and ends, and /proc/PID/environ is simply a read of that memory range. Entries are NUL-separated, which is why the example pipes through tr.

The file is mode 0400 and owned by the process owner, but the real check is a ptrace-style access check. In practice: the same user, and root, can read it. Other unprivileged users cannot.

So this is not a remote hole. It matters when something at the same privilege level goes wrong: a compromised web app running as the same user, a sloppy plugin, a debugging tool someone runs as that account, or a backup job that sweeps up /proc.

Quick detour: Yama does not stop this

You might expect kernel.yama.ptrace_scope=1 to block it, since that setting stops a process from attaching to its siblings. Hang on, it does not. Yama only hooks attach-style ptrace access, and reading environ uses the weaker read-style check. So a hardened box that blocks gdb -p from a sibling will still happily show you the environment.

What does help is the target being non-dumpable. A process that calls prctl(PR_SET_DUMPABLE, 0) gets its /proc/PID files owned by root, so a same-user reader is refused. That is a mitigation, not a design.

Why unsetting the variable does not clear it

The obvious fix is to read the key at start-up and delete it. Try it:

package main

import (
	"fmt"
	"os"
	"time"
)

func main() {
	key := os.Getenv("APP_KEY")
	if err := os.Unsetenv("APP_KEY"); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	fmt.Println("pid", os.Getpid(), "key length", len(key))
	time.Sleep(time.Minute)
}

Run it with APP_KEY=correct-horse-battery, then read /proc/PID/environ from another shell. The key is still there.

Go keeps its own copy of the environment, and Unsetenv edits that copy. The original block the kernel set up at exec time is never touched. C's unsetenv behaves the same way: it removes a pointer from the environ array but leaves the string bytes where they were.

You could overwrite that region yourself, but that means finding it and writing to memory the runtime does not expect you to own. Not worth it.

The other places the variable travels

The /proc file is only one copy. An environment variable is inherited by every child process, so any helper you spawn gets the key too unless you build its environment explicitly.

  • Child processes inherit it by default, and each has its own /proc/PID/environ.
  • systemctl show prints Environment= values for a unit, and unit files are often world-readable.
  • docker inspect shows every -e variable to anyone with access to the Docker socket.
  • Crash dumps and core files contain the stack, environment included.
  • Error reporters and "debug info" endpoints love to dump os.Environ().

Compare that with /proc/PID/cmdline, which is the same story but worse: it is world-readable by default, so a key passed as --key=... is visible to every user on the machine. The environment is better than argv, but "better than argv" is a low bar.

What to do instead

Pass a reference to the secret, not the secret. The variable holds a path; the key lives in a file with tight permissions.

  1. Store the key in a file owned by the service user, mode 0400.
  2. Give the process the path (not the key) through a variable or flag.
  3. Read the file once at start-up and keep the bytes in memory only as long as needed.

On a systemd host there is a neater route. LoadCredential= copies the file into a private, memory-backed directory for that service and exports the directory path:

[Service]
ExecStart=/usr/local/bin/myapp
LoadCredential=appkey:/etc/myapp/appkey
User=myapp
dir := os.Getenv("CREDENTIALS_DIRECTORY")
if dir == "" {
	return nil, errors.New("CREDENTIALS_DIRECTORY not set")
}
key, err := os.ReadFile(filepath.Join(dir, "appkey"))
if err != nil {
	return nil, fmt.Errorf("reading key: %w", err)
}

The environment now contains a directory name, which is harmless. The credential files are readable only by the service itself and root, and they do not appear in systemctl show. If the key must come from a prompt or another program, a pipe on stdin works too, because pipe contents are never exposed under /proc.

Check your own services

Look at what a running service exposes, as the same user it runs as:

pid=$(pidof myapp)
tr '\0' '\n' < /proc/$pid/environ | cut -d= -f1

Printing only the names (the cut) keeps the values out of your terminal scrollback, which, given the state of most shell histories, is a courtesy worth extending to yourself. If any name looks like KEY, TOKEN or PASSWORD, you have your answer.

One honest caveat: files are not magic either. A same-user attacker can read the key file too if the process can. What you gain is a smaller number of accidental copies, and a secret that does not get pushed into every child process, unit listing and container inspection by default.

← All articles