Make systemd-journald Persistent and Keep Its Disk Use Predictable

You will configure a systemd host to retain its journal across reboots, check the result without guessing from service status, and choose how persistent kernel crash records from pstore are archived. Allow about fifteen minutes. You need a shell and sudo access for the configuration steps. Reading the journal is normally unprivileged, although your account may need membership of adm or systemd-journal to see every user's messages.

The commands below describe systemd 255, installed here as package version 255.4-1ubuntu8.17. Other releases may add settings, but the storage locations and controls used here are documented by this installed version.

1. Check the daemon and the current storage use

Start with read-only checks. This separates a logging-storage problem from a daemon that is not running:

$ systemctl is-active systemd-journald.service
active
$ journalctl --disk-usage
Archived and active journals take up 81.3M in the file system.

Your size will differ. The first command checks the service, while the second measures journal files that journald is managing. It does not tell you whether those files are in volatile /run or persistent /var, so check the directories directly:

$ find /run/log/journal /var/log/journal -maxdepth 2 -type f -name '*.journal*' -print 2>/dev/null
/run/log/journal/MACHINE_ID/system.journal
/var/log/journal/MACHINE_ID/system.journal

Replace the example machine ID with the directory printed on your host. An absent /var/log/journal normally means the default configuration falls back to volatile storage. Files under /run are lost at reboot.

2. Enable persistent journal storage

With the default journald configuration, create the persistent directory and let systemd-tmpfiles apply its ownership and mode:

$ sudo mkdir -p /var/log/journal
$ sudo systemd-tmpfiles --create --prefix /var/log/journal

Creating this directory changes system state but does not delete existing logs. Journald initially writes to volatile storage and the boot transaction normally flushes it to /var. Request a flush after the directory exists:

$ sudo journalctl --flush
$ find /var/log/journal -maxdepth 2 -type f -name '*.journal*' -print
/var/log/journal/MACHINE_ID/system.journal

journalctl --flush waits for the operation to complete. The equivalent daemon signal is SIGUSR1, but the command is easier to audit and less prone to targeting the wrong process.

To make the policy explicit rather than dependent on directory presence, use a journald drop-in. The exact setting is read from /etc/systemd/journald.conf.d/ by the installed journald configuration:

$ sudo install -d -m 0755 /etc/systemd/journald.conf.d
$ sudo sh -c 'printf "%s\n" "[Journal]" "Storage=persistent" > /etc/systemd/journald.conf.d/20-persistent.conf'
$ systemd-analyze cat-config systemd/journald.conf | sed -n '/20-persistent.conf/,$p'

Before using the last command, inspect the displayed file and confirm that the setting is exactly Storage=persistent. If you need to undo this explicit choice, remove only the drop-in and flush again after deciding whether the persistent directory should remain:

$ sudo rm /etc/systemd/journald.conf.d/20-persistent.conf
$ sudo systemctl restart systemd-journald.service

The removal is reversible by recreating the file. Do not remove /var/log/journal as part of a routine rollback: that would discard retained logs and is not needed to return to the default policy.

3. Reload safely and verify retained entries

Ask journald to reread its configuration, then restart it in one operation if the drop-in changed. A restart preserves service-manager stream connections; stopping journald separately is discouraged because services writing to those streams can receive broken-pipe errors:

$ sudo systemctl restart systemd-journald.service
$ systemctl is-active systemd-journald.service
active
$ journalctl -b 0 -n 5 --no-pager -o short-iso

The final command shows the newest five records from the current boot. To prove persistence after a reboot, record the current boot ID and later query the previous boot:

$ journalctl -b 0 -o verbose -n 1 --no-pager | sed -n '1,8p'
$ journalctl --list-boots

A previous boot in --list-boots is evidence that journal files survived. If only the current boot appears after a reboot, inspect Storage=, the directory permissions, and whether systemd-journal-flush.service completed. Do not infer persistence from systemctl is-active alone.

4. Keep journal growth within a deliberate limit

Journald removes older archived files according to its size and time limits. Put local limits in a drop-in rather than editing a vendor file. For example, this caps the system journal at 2 GiB and permits entries to remain for 30 days:

[Journal]
SystemMaxUse=2G
MaxRetentionSec=30day

Save that content as /etc/systemd/journald.conf.d/30-retention.conf, then restart journald and inspect the resulting usage:

$ sudo systemctl restart systemd-journald.service
$ journalctl --disk-usage

These are ceilings, not promises that the directory will immediately shrink to exactly 2 GiB. Rotate first when you need an archived boundary, then vacuum with a separate, explicit command. Vacuuming deletes old journal data and cannot be undone, so take a copy or confirm your retention requirement before running it.

5. Choose what happens to pstore crash records

If the kernel exposes pstore, systemd-pstore processes files left there after a crash. The companion pstore.conf and pstore.conf.d manuals document three Storage= values: external archives files under /var/lib/systemd/pstore/ and logs them in the journal, journal logs their contents only, and none skips processing. The default is external.

Its default Unlink=yes removes a pstore file after processing. That is deliberate: pstore is a small crash-recovery area and leaving old files there can prevent the next crash record from being saved. Treat changing this as a recovery trade-off, not as ordinary log retention.

To keep a copy in pstore while investigating a crash, use a numbered drop-in:

$ sudo install -d -m 0755 /etc/systemd/pstore.conf.d
$ sudo sh -c 'printf "%s\n" "[PStore]" "Storage=external" "Unlink=no" > /etc/systemd/pstore.conf.d/20-keep-crash-records.conf'
$ systemd-analyze cat-config systemd/pstore.conf

After the investigation, remove the drop-in to restore the compiled default:

$ sudo rm /etc/systemd/pstore.conf.d/20-keep-crash-records.conf

Do not delete files from /sys/fs/pstore or /var/lib/systemd/pstore until you have copied and reviewed them. The pstore filesystem is platform-dependent, so an empty directory is not proof that journald or systemd-pstore is broken.

6. Diagnose access and namespace surprises

Journal files are normally readable by the systemd-journal group but not writable by it. If your account sees a restricted subset, use sudo journalctl for a one-off inspection or ask an administrator to grant the appropriate group access. Do not make journal files world-writable.

A service in a non-default journal namespace will not appear in ordinary journalctl output. Namespaces have separate stores and IPC endpoints. Query a known namespace explicitly:

$ journalctl --namespace=NAME -n 20 --no-pager

Replace NAME with the configured namespace identifier. Kernel and audit messages normally belong only to the default namespace, so a missing kernel record may be a namespace choice rather than a failed socket.