Articles / Systems Programming

1073 words

  • linux
  • unix-sockets
  • systemd
  • file-descriptors
  • go
  • security

Passing File Descriptors Between Processes with SCM_RIGHTS

Can one process open a file descriptor and then physically move it into a completely unrelated process, one it never forked and doesn't share memory with? It sounds like a trick question. Not a copy of the data, not a path the other process can reopen itself: the actual open file descriptor, pointing at the same open file description in the kernel.

Yes. The mechanism is older than most of the software that relies on it: SCM_RIGHTS, an ancillary data type you can send down a Unix domain socket.

Why you'd want to move an fd at all

It is the standard answer to a recurring privilege-separation problem. Something needs to bind port 443, or open /dev/kvm, or hold a raw socket, but you don't want the process that actually handles untrusted input running with the capability to do that.

The classic fix is to split it into two processes:

  • a small, trusted one that holds the privilege and does the one privileged thing;
  • a larger, unprivileged one that does everything else.

The privileged fd has to cross that boundary somehow, and it can't cross it as a number, because file descriptor numbers are only meaningful within a single process's fd table. SCM_RIGHTS is how it crosses.

What actually gets sent is a new descriptor, not a number

A Unix domain socket can carry two kinds of payload on every sendmsg/recvmsg call: the ordinary byte stream (or datagram), and a side channel of "ancillary data" (also called control messages, or cmsgs). SCM_RIGHTS is a control message type at the SOL_SOCKET level whose payload is an array of file descriptor numbers, as understood by the sender.

The kernel does not send these numbers verbatim. It walks the array, and for each one, installs a new descriptor in the receiving process's fd table that refers to the same open file description as the sender's original.

When the receiver reads the message, it gets back a (probably different) integer. That integer now behaves exactly as if the receiver had opened the file, socket, or pipe itself: same read/write offset if it's a regular file, same socket state if it's a connected TCP socket, same everything.

Hang on, isn't that just dup2? Not quite, and the difference is the whole point: dup2 only works within one process. SCM_RIGHTS is dup2 across a process boundary that the two ends don't otherwise share.

The only requirement is that both ends have an open Unix domain socket connected to each other. They don't need to be related by fork, they don't need to share a filesystem namespace for the fd's target, they just need that one socket.

A minimal working example in plain Go

Go's standard library has everything needed for this without a single third-party dependency:

  • syscall.Socketpair to get a connected pair of Unix domain sockets;
  • syscall.UnixRights/ParseUnixRights to build and parse the control message;
  • net.UnixConn's WriteMsgUnix and ReadMsgUnix, which take the ancillary data as an explicit parameter.

So nothing here is hidden behind a higher-level abstraction that would obscure what's happening.

package main

import (
	"fmt"
	"net"
	"os"
	"syscall"
)

func fdToUnixConn(fd int) (*net.UnixConn, error) {
	f := os.NewFile(uintptr(fd), "socketpair")
	defer f.Close()
	conn, err := net.FileConn(f)
	if err != nil {
		return nil, err
	}
	return conn.(*net.UnixConn), nil
}

func main() {
	fds, err := syscall.Socketpair(syscall.AF_UNIX, syscall.SOCK_STREAM, 0)
	if err != nil {
		panic(err)
	}

	sender, err := fdToUnixConn(fds[0])
	if err != nil {
		panic(err)
	}
	defer sender.Close()

	receiver, err := fdToUnixConn(fds[1])
	if err != nil {
		panic(err)
	}
	defer receiver.Close()

	// Anything with a file descriptor works here: an open file, a
	// listening TCP socket bound to port 443, a raw socket, a pipe end.
	f, err := os.Open("/etc/hostname")
	if err != nil {
		panic(err)
	}
	defer f.Close()

	rights := syscall.UnixRights(int(f.Fd()))
	if _, _, err := sender.WriteMsgUnix(nil, rights, nil); err != nil {
		panic(err)
	}

	oob := make([]byte, syscall.CmsgSpace(4))
	_, oobn, _, _, err := receiver.ReadMsgUnix(nil, oob)
	if err != nil {
		panic(err)
	}

	cmsgs, err := syscall.ParseSocketControlMessage(oob[:oobn])
	if err != nil {
		panic(err)
	}
	recvFds, err := syscall.ParseUnixRights(&cmsgs[0])
	if err != nil {
		panic(err)
	}

	received := os.NewFile(uintptr(recvFds[0]), "received")
	defer received.Close()

	data := make([]byte, 128)
	n, _ := received.Read(data)
	fmt.Printf("read via handed-off fd: %s", data[:n])
}

Nothing here required the receiver to have opened /etc/hostname, or even to have permission to open it independently. It only had permission to receive an already-open descriptor to it.

The socket's permissions become your access control

That distinction is the entire point of the pattern, and also its main security consideration. Possession of the Unix socket is effectively possession of whatever capability gets sent across it, so the socket's own permissions (filesystem mode bits, or peer credential checks via SO_PEERCRED) become the actual access control boundary.

Get those wrong and you've built a very elaborate way to hand root-owned sockets to anyone who can connect.

One practical limit worth knowing: Linux caps the number of descriptors you can pass in a single message at SCM_MAX_FD, 253 on current kernels. It rarely bites in practice, since most uses pass one or two fds at a time, but it explains why nobody tries to bulk-transfer an entire fd table this way.

systemd socket activation isn't SCM_RIGHTS at all

systemd's socket activation is the thing most people associate with fd handoff, and it's worth being precise about what it actually is, because it isn't SCM_RIGHTS. When a .socket unit is triggered:

  1. systemd opens the listening socket itself;
  2. it then forks and execs the corresponding service with that socket's fd still open, starting at fd 3;
  3. $LISTEN_FDS and $LISTEN_PID are set so the service, typically via sd_listen_fds(), knows it's there.

That's plain fd inheritance across fork/exec, the same mechanism that keeps stdin/stdout/stderr open across any process spawn. No ancillary data involved, because systemd is the direct parent of the process it's handing the fd to.

Where systemd really does use it: the fd store

The genuinely interesting use of SCM_RIGHTS in systemd shows up somewhere less obvious: the file descriptor store. It's for a service that wants its listening sockets to survive a crash or restart without a gap in which connections get refused.

Such a service can call sd_pid_notify_with_fds(), sending an FDSTORE=1 message plus the fds themselves over $NOTIFY_SOCKET, an AF_UNIX SOCK_DGRAM socket connecting the service back to the systemd manager.

Because systemd and the service are not parent and child at that point in any way that makes inheritance available, the only way to get the fd across is exactly the mechanism above: a control message on a Unix socket.

systemd holds onto the descriptor, and if the service restarts, hands it straight back on the next start. The listening socket never actually closes; only the process holding it changes. Which is a rather lovely trick when you think about it.

The same shape turns up everywhere privilege is split

The same trick shows up under different names all over privilege-separated software:

  • OpenSSH's privilege separation model;
  • D-Bus handing fds to bus clients so they don't need to open device nodes themselves;
  • Chromium's sandboxed renderer processes receiving already-opened files from a broker process that does the actual filesystem access on their behalf.

In every case the shape is identical: a trusted process decides what gets opened, an untrusted or lower-privileged one gets to use it without ever having had the ability to open it itself. The socket isn't carrying data at that point, it's carrying a capability.

← All articles