Articles / Systems Programming

903 words

  • linux
  • unix-sockets
  • go
  • filesystems
  • systems-programming
  • debugging

Why a Unix Socket Won't Bind Again Until You Unlink the Path

Your daemon crashes, systemd restarts it, and the new process dies instantly with bind: address already in use. Nothing is listening. No process holds the socket. Yet the bind fails, and it keeps failing until someone runs rm.

The cause is that a Unix domain socket has two lives: the kernel object, which dies with your process, and a name in the filesystem, which does not.

The name is a real directory entry

When you call bind() on an AF_UNIX socket with a path, the kernel creates a file of type socket at that path. You can see it:

$ ls -l /run/myapp.sock
srw-rw---- 1 andy andy 0 Nov 12 09:14 /run/myapp.sock

That leading s is the file type. It is an inode like any other, and closing the socket does not remove it. Only unlink() does. If the process is killed or crashes, nobody calls it.

The next bind() to that path finds an existing entry and fails with EADDRINUSE. The kernel does not check whether anything is still listening behind it. The existence of the name is enough.

Quick detour: why doesn't the kernel just clean it up?

Hang on, why is it designed like this? Because the socket is not owned by the file. The file is only a rendezvous point that clients use to find the socket, rather like a phone number written on a wall.

Several processes can hold the same socket open (see the post on passing file descriptors for how that happens). The kernel has no single "owner" whose death should delete the name. So the application is responsible for it.

Stale versus live: connect to find out

The obvious fix is to unlink() before every bind(). It works, and it is also a small bug waiting to happen. If a second copy of your service is running, you have just deleted the first copy's name. It keeps running, but nobody can reach it any more.

A better test is to try connecting:

  • ECONNREFUSED: the file exists but nothing is listening. It is stale, so remove it.
  • Connect succeeds: a live instance owns it. Refuse to start.
  • Anything else (such as EACCES): you do not know, so stop and report it.

Also check that the thing at the path really is a socket. If someone has configured the path to point at a regular file, deleting it is not your job.

A Go version

package main

import (
	"errors"
	"fmt"
	"io/fs"
	"net"
	"os"
	"syscall"
	"time"
)

func listenUnix(path string) (*net.UnixListener, error) {
	fi, err := os.Lstat(path)
	switch {
	case err == nil:
		if fi.Mode().Type() != fs.ModeSocket {
			return nil, fmt.Errorf("%s exists and is not a socket", path)
		}
		c, err := net.DialTimeout("unix", path, time.Second)
		if err == nil {
			c.Close()
			return nil, fmt.Errorf("%s is in use by a running process", path)
		}
		if !errors.Is(err, syscall.ECONNREFUSED) {
			return nil, fmt.Errorf("probing %s: %w", path, err)
		}
		if err := os.Remove(path); err != nil {
			return nil, err
		}
	case !errors.Is(err, fs.ErrNotExist):
		return nil, err
	}
	return net.ListenUnix("unix", &net.UnixAddr{Name: path, Net: "unix"})
}

Go helps with the clean-exit case. A UnixListener created by Listen or ListenUnix unlinks the path in Close by default. You can switch that off with SetUnlinkOnClose(false).

But that only runs if your code gets to Close. A panic, SIGKILL, the OOM killer, or a plain os.Exit skips it. Stale sockets are a crash problem, so the cleanup-on-close path does not solve them.

The check-then-remove race

There is a gap between the probe, the Remove and the bind. Two instances starting together could both see a stale file, both remove it, and one then binds while the other removes the winner's fresh socket.

For a single service managed by systemd this rarely matters, since it will not start two copies. If it does matter, take an flock on a separate lock file next to the socket before you probe. The lock file is never unlinked, so the lock itself has no stale-name problem.

Permissions: bind creates the file with your umask

The socket file's mode comes from the process umask, and on Linux connecting to a socket requires write permission on it. A common pattern is bind() and then chmod(). That leaves a short window where the socket has the wrong mode.

Two tidier options:

  1. Put the socket in a directory with the restrictive mode you want (for example 0700 or a group-only directory). Directory permissions apply to the path walk, so the socket's own mode matters less.
  2. Set a tight umask around the bind, accepting that umask is process-wide and not safe if other goroutines create files at that moment.

The directory approach is simpler and has no race. It is also why /run/myapp/myapp.sock is a better choice than /tmp/myapp.sock, where anyone can pre-create the name.

Ways to avoid the problem altogether

  • systemd socket activation. systemd creates and owns the socket and hands your process the descriptor. A crash does not remove the name, and clients connecting during a restart queue up instead of failing.
  • Abstract sockets (Linux only). A path starting with a NUL byte (written @name in many tools) lives in a kernel namespace, not on disk. It vanishes when the last descriptor closes. The trade-off is that there are no file permissions to restrict who connects, and it is tied to the network namespace.
  • A runtime directory. RuntimeDirectory=myapp in a unit file gives you a fresh directory, removed when the service stops.

One last limit worth knowing: the path has to fit in sun_path, which is 108 bytes on Linux and 104 on the BSDs and macOS. A deep container or test-temp directory can exceed it, and the error is a confusing one. If you hit it, use a shorter directory or change into the directory and bind a relative name.

So the rule is simple: the socket is a file, files stay until someone removes them, and "someone" has to be your startup code, written so it cannot delete a live neighbour's name.

← All articles