Find the Deleted File That Is Filling Your Disk

Your disk is full, but du cannot find the missing gigabytes. The usual culprit is a deleted file that a running process still has open.

Deleting a file removes its name, not necessarily its storage. Linux keeps the inode and data blocks alive until every open file descriptor and every other hard link has gone away. A long-running service can therefore keep a multi-gigabyte log file alive after log rotation, manual cleanup or a careless rm.

Start by checking the disagreement

Use df to ask the filesystem how much space is allocated, then use du to ask visible directory trees how much data they contain.

df -h /
sudo du -xhd1 / 2>/dev/null | sort -h

The -x option keeps du on the same filesystem. Without it, mounted filesystems under directories such as /var, /home or /mnt can make the totals misleading.

If df reports a nearly full filesystem but the visible du total is much smaller, open deleted files become a strong suspect. It is not the only explanation: reserved blocks, filesystem metadata and snapshots can also account for some space. A large unexplained gap, though, deserves an open-file check.

Ask lsof for deleted files

The quickest check is lsof, which lists processes holding file descriptors. Run it as root so you can see files opened by services belonging to other users.

sudo lsof -nP +L1

+L1 tells lsof to show files whose link count is less than one. Those are normally unlinked files. The useful columns include the process name, PID, user, file descriptor and the file's former path.

COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NLINK  NODE NAME
myservice 842 root  7w   REG  259,2  8589934592     0 12345 /var/log/myservice.log (deleted)

Here, file descriptor 7 is open for writing and still accounts for roughly 8 GiB. The pathname is only a clue now. The directory entry is gone, so du has no route to discover the data.

Check the process before touching anything

The PID and file descriptor tell you what must be fixed. Inspect the process and its open descriptor before deciding how to release the space.

sudo ps -fp 842
sudo readlink /proc/842/fd/7
sudo ls -l /proc/842/fd/7

The /proc link will usually end in (deleted). You can also inspect the process command line, service unit and open file list:

sudo tr '\0' ' ' < /proc/842/cmdline; echo
sudo systemctl status myservice
sudo systemctl cat myservice

Do not kill a random process just because it owns a large deleted file. It might be a database, a queue, a socket-backed temporary file or something that is deliberately kept open. Confirm what the descriptor represents first.

Usually, restart the owning service

For an ordinary daemon, a controlled restart is the safest way to close the descriptor and let the filesystem reclaim the blocks.

sudo systemctl restart myservice
sudo lsof -nP +L1
sudo df -h /

If the service is managed by a supervisor, container runtime or orchestration system, restart it through that manager. Otherwise it may immediately come back with the same problem, or the manager may interpret a manual kill as a failure.

After the restart, check the service logs and health endpoint. A missing log file can point to a broken rotation configuration, but it can also reveal a service that was still writing to a file it was never told to reopen.

Log rotation is a common source of the leak

Quick detour: log rotation often works by renaming the old file, creating a replacement and asking the daemon to reopen its logs. If the reopen step never happens, the daemon keeps writing to the renamed file. If the old file is then deleted, the same data becomes invisible to du while continuing to grow.

For a service using logrotate, inspect the rotation rule and its post-rotation action. A typical rule might send a signal, run a command or use a service restart. The correct action depends on the daemon, so do not copy a signal from an unrelated service and hope for the best.

sudo logrotate -d /etc/logrotate.conf
sudo journalctl -u logrotate.service --since today

Some applications write logs through journald, in which case ordinary files under /var/log may not be the issue. Check journal usage separately:

journalctl --disk-usage
sudo journalctl --vacuum-time=14d

Only remove journal data according to your retention needs. Disk pressure is unpleasant, but deleting the audit trail you later need can be worse.

If lsof is unavailable, inspect /proc directly

Minimal rescue environments do not always include lsof. Linux still exposes the relevant information through /proc, although the shell loop is less convenient and can be slow on a busy machine.

sudo find /proc/[0-9]*/fd -type l -lname '* (deleted)' -printf '%p -> %l\n' 2>/dev/null

The result contains paths such as /proc/842/fd/7. The directory name is the PID and the final component is the file descriptor. Confirm the process with ps, then decide how it should be stopped or reloaded.

Another useful view is the process's file table:

sudo ls -l /proc/842/fd

Do not assume every (deleted) entry consumes meaningful disk space. Pipes, sockets and anonymous temporary files can appear there too. The size shown by lsof, the descriptor type and the owning application are all relevant.

Truncating the descriptor is an emergency measure

You can sometimes reclaim space without restarting a process by truncating its open descriptor. This is only appropriate when you understand the file's role, normally for a plain append-only log.

sudo sh -c ': > /proc/842/fd/7'

Truncation changes the live file, not just its directory entry. It can confuse applications that expect a complete file, corrupt an active data file or remove evidence needed for investigation. It also does not fix the underlying rotation problem, so treat it as a pressure valve while arranging a proper restart or reload.

Never use this casually on databases, package stores, queues or files whose contents are part of an application protocol. A service restart is slower, but usually much easier to reason about.

Containers add one more place to look

In containerised systems, the process holding the deleted file may live in a container while the consumed blocks belong to the host filesystem. Run the check on the host, then map the PID to its container or systemd unit.

sudo lsof -nP +L1
sudo crictl ps
sudo docker ps --no-trunc

The exact tooling depends on the runtime. A container restart may release the space, but an application-level fix is still needed if the same log handling mistake will recreate it.

Make the next incident easier

Keep three checks close at hand: df for allocated filesystem space, du for reachable directory contents and lsof +L1 for unlinked files still held open. They answer different questions, which is why comparing them works.

If this happens repeatedly, alert on filesystem usage and investigate log rotation before the filesystem reaches 100 percent. Once a disk is completely full, services may fail to write sockets, journals, lock files or recovery data. The invisible file was only the symptom; the missing reopen or retention policy is the bug.