Articles / Linux

832 words

  • linux
  • systemd
  • environment-variables
  • sysadmin
  • debugging
  • shell

Why Your systemd Service Never Sees Your Shell's Variables

You export API_URL in your shell, run the program by hand and it works. Then you start it as a systemd service and it behaves as if the variable never existed. That is because it never did, as far as the service is concerned.

The reason is simple: the service is not a child of your shell. It is started by systemd (PID 1, or your per-user manager), and it inherits whatever environment systemd has, plus whatever the unit file adds. Your export only ever touched your shell and its children.

Where the service actually comes from

When you type sudo systemctl restart myapp, the systemctl command only sends a request over D-Bus. The manager then forks the process itself. Nothing from the environment of systemctl, sudo or your terminal travels along with that request.

That also explains a few related surprises:

  • Anything set in ~/.bashrc, ~/.profile or /etc/profile.d/ is not read, because no login or interactive shell is involved.
  • PATH is a short default, not the one you are used to.
  • HOME, USER and similar are only set if the unit says User=.
  • sudo -E makes no difference, since sudo is not in the chain.

Look at what the service really has

Do not guess; check. Ask systemd what it configured, then ask the kernel what the process actually received.

systemctl show myapp -p Environment -p EnvironmentFiles

pid=$(systemctl show myapp -p MainPID --value)
sudo tr '\0' '\n' < /proc/$pid/environ

The second command is the ground truth. /proc/PID/environ holds the initial environment as a NUL-separated list, hence the tr. If your variable is not there, the program cannot see it, whatever the unit file seems to say.

Quick detour: the manager has its own environment too

Hang on, this bit is interesting. systemd keeps a global environment block for the manager, and everything it starts inherits from it. You can see it with systemctl show-environment.

You can add to it with systemctl set-environment NAME=value, or copy variables from your shell with systemctl import-environment NAME. That does work, but it is global and lost on reboot, so it is a debugging tool rather than a configuration method. For user services, systemctl --user import-environment is what desktop sessions use to hand over things like DISPLAY.

Setting variables properly in the unit

There are three sensible routes, in rough order of how often you want them.

  1. Environment= for a few non-secret values.
  2. EnvironmentFile= for a longer list, or values that differ per host.
  3. LoadCredential= for secrets (more on that below).

Edit with a drop-in rather than touching the packaged unit:

sudo systemctl edit myapp
[Service]
Environment=API_URL=https://api.example.org
Environment="GREETING=hello world"
EnvironmentFile=-/etc/myapp/myapp.env

The leading - on the file path means "do not fail if it is missing". Leave it off if the file is mandatory and you want a loud failure at start.

The file format is not shell

An EnvironmentFile looks like shell but is not parsed by one. Plain KEY=value lines are the safe choice.

# /etc/myapp/myapp.env
API_URL=https://api.example.org
LOG_LEVEL=debug
GREETING="hello world"

No export keyword, no $(command), no $OTHER expansion inside values. Lines starting with # or ; are comments. If you tried to reuse a file that you normally source, check each line against these rules.

Pitfalls that waste an afternoon

  • Forgetting to reload. After editing a unit or drop-in, run sudo systemctl daemon-reload, then restart. systemctl edit does the reload for you; hand edits do not.
  • Editing the env file but not restarting. The file is read when the service starts, not continuously.
  • Percent signs. systemd expands specifiers like %h and %n in unit files, so a literal % in a value needs writing as %%.
  • Using $VAR in ExecStart= and expecting a shell. systemd does substitute ${VAR} and $VAR from the unit's own environment, but there is no globbing, piping or quoting beyond systemd's own rules. Wrap in /bin/sh -c if you really need a shell.
  • Relying on PassEnvironment=. It copies variables from the manager's own environment (system instance only), not from your shell. It is rarely what you want.

Secrets deserve better than Environment=

Values set in the unit are visible to anyone who can run systemctl show, and the process environment is readable through /proc/PID/environ by the same user and root. Environment files with mode 0600 help with the first problem only.

For anything sensitive, LoadCredential= hands the service a file under $CREDENTIALS_DIRECTORY that only that service can read, and the program reads the file instead of an environment variable.

[Service]
LoadCredential=dbpass:/etc/myapp/dbpass
# in the program: read $CREDENTIALS_DIRECTORY/dbpass

That does mean changing the program to read a file. If you cannot, a 0600 environment file owned by root is the pragmatic fallback, with the exposure through /proc understood and accepted.

A one-off test without editing anything

To check a value reaches a process without touching the unit, systemd-run can set it for a transient service:

sudo systemd-run --wait --pipe -p Environment=FOO=bar env

If FOO=bar appears in the output alongside a very short list of other variables, you have just seen exactly what a service gets, which is the whole point. Compare that with env in your terminal and the gap you were chasing is obvious.

← All articles