Articles / Linux

736 words

  • linux
  • flock
  • file-descriptors
  • file-locking
  • sysadmin
  • debugging

Why a Process Still Holds a flock After You Closed the File

Your script took a lock, finished, exited, and the next run still says "already running". Nothing visible is holding the file open. Yet the lock is there, and it will stay there until something you did not think about goes away.

The cause is nearly always the same: a flock belongs to an open file description, not to the process or the path. Anything that shares that description keeps the lock alive. Let's see how it gets shared, and how to find the culprit.

A reproduction in six lines of shell

Here is a typical "run only one copy" script.

#!/bin/sh
exec 9>/tmp/job.lock
flock -n 9 || { echo "already running"; exit 1; }

./start-helper-daemon &
echo "done"

The script exits straight away. But run it a second time and you get "already running", for as long as the helper lives. The helper was started with fd 9 still open, because child processes inherit every descriptor that is not marked close-on-exec. The shell does not mark it.

The lock was never "held by the script". It was held by the open file description behind fd 9, and the daemon now has a reference to it.

What flock actually attaches to

Three layers are involved, and mixing them up is the whole bug:

  • the file descriptor: a small integer in one process's table
  • the open file description: the kernel object created by open(), holding offset, flags and, for flock, the lock
  • the inode: the file itself

flock(2) locks are attached to the middle layer. A dup(), a fork() or an inherited descriptor gives you a second fd pointing at the same description. The lock is released only when the last of those is closed, or when someone calls LOCK_UN.

Finding who has it

Start with the kernel's own list. /proc/locks has one line per lock, and the inode number is in the sixth column.

$ cat /proc/locks
3: FLOCK  ADVISORY  WRITE 48211 08:01:1310742 0 EOF

$ stat -c %i /tmp/job.lock
1310742

The PID in that line is the process that took the lock, which may be long gone, or may be the parent that spawned the real holder. Do not treat it as the answer. Match on the inode and then ask who has the file open:

$ lsof /tmp/job.lock
$ fuser -v /tmp/job.lock
$ ls -l /proc/*/fd 2>/dev/null | grep job.lock

Whatever turns up, /proc/PID/fdinfo/N for that descriptor lists a lock: line on current kernels, which confirms this process really holds the lock and is not just holding the file open.

Fixing the leak

You have a few options, roughly in order of preference.

  1. Close the descriptor in the child: ./start-helper-daemon 9>&- &
  2. If you use the flock command to wrap a program, pass -o (--close) so the descriptor is closed before the command runs
  3. Do not start long-lived children from inside a locked section at all

The second is the one people miss. flock -n /tmp/job.lock -c "start-daemon" hands the lock to the daemon for its entire life unless you ask otherwise.

Quick detour: the opposite bug with fcntl locks

Hang on, there is a mirror image of this that is nastier. POSIX locks from fcntl(F_SETLK) belong to the process and the inode, not the description. Closing any descriptor for that file, even one a library opened by itself, drops all your locks on it. They are also not inherited by children.

Linux 3.15 added open file description locks (F_OFD_SETLK), which behave like flock for sharing and like fcntl for byte ranges. If you need range locks, use those. Note that /proc/locks reports their PID as -1, since no process owns them.

What this looks like in Go

Go helps you here: files opened with os.OpenFile are close-on-exec, so a child started with exec.Command will not inherit them unless you pass them in ExtraFiles.

package main

import (
	"fmt"
	"os"
	"syscall"
)

func lock(path string) (*os.File, error) {
	f, err := os.OpenFile(path, os.O_CREATE|os.O_RDWR, 0o600)
	if err != nil {
		return nil, err
	}
	if err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil {
		f.Close()
		return nil, fmt.Errorf("lock %s: %w", path, err)
	}
	return f, nil
}

func main() {
	f, err := lock("/tmp/job.lock")
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	defer f.Close()
	// ... do the work ...
}

The catch is what you inherit, not what you open. If your Go binary is launched from a shell script that leaked fd 9, Go will not close it for you, and your program sits there with a lock it never asked for.

Releasing on purpose

If you need the lock gone regardless of who else has a copy of the descriptor, call flock(fd, LOCK_UN) before closing. Because the lock lives on the shared description, an explicit unlock releases it for every duplicate at once, whereas a plain close() only drops your reference.

That is also a decent habit for daemons that fork workers: unlock explicitly when the critical section ends, rather than relying on every process in the family tree to exit in the right order.

← All articles