Articles / Systems Programming

730 words

  • linux
  • nfs
  • filesystems
  • o_append
  • go
  • debugging

Why O_APPEND Writes Don't Interleave, Until NFS

Two processes append lines to the same log file, and on your laptop every line comes out whole. Move the same file to an NFS mount, and lines start overwriting each other. Nothing errors. The file just has holes and mangled text.

The cause is a guarantee that O_APPEND gives you locally and that NFS cannot give you at all.

What O_APPEND promises on a local filesystem

With O_APPEND, the kernel treats "move to the end of the file" and "write the data" as one atomic step. Every write(2) lands after whatever was at the end at that moment, whatever any other process is doing.

Without the flag, you would do the two steps yourself, and there is a gap between them:

  1. Process A seeks to offset 1000 (the current end).
  2. Process B seeks to offset 1000 as well.
  3. A writes 50 bytes at 1000.
  4. B writes 50 bytes at 1000, destroying A's line.

On Linux, a local filesystem takes the inode lock for the duration of a write to a regular file. With O_APPEND the end-of-file offset is read inside that lock. So two appenders are serialised, and each write call arrives in one piece.

A quick way to see it

This Go program writes numbered lines from the process it is run in. Start several copies against the same file.

package main

import (
	"fmt"
	"log"
	"os"
	"strings"
)

func main() {
	f, err := os.OpenFile(os.Args[1], os.O_WRONLY|os.O_APPEND|os.O_CREATE, 0o644)
	if err != nil {
		log.Fatal(err)
	}
	defer f.Close()

	pad := strings.Repeat("x", 200)
	for i := 0; i < 100000; i++ {
		// One Write call is one write(2): build the whole line first.
		line := fmt.Sprintf("pid=%d n=%d %s\n", os.Getpid(), i, pad)
		if _, err := f.Write([]byte(line)); err != nil {
			log.Fatal(err)
		}
	}
}

Run four of them at once, then check every line has the same shape:

for i in 1 2 3 4; do ./appender /mnt/test/log.txt & done; wait
wc -l /mnt/test/log.txt                      # expect 400000
grep -vcE '^pid=[0-9]+ n=[0-9]+ x{200}$' /mnt/test/log.txt   # expect 0

On ext4 or XFS you get 400000 lines and zero bad ones. On NFS you usually get fewer lines than expected and a pile of bad ones.

Why NFS cannot do it

The NFS protocol's WRITE operation takes an explicit offset. There is no "append" operation to hand to the server. The open(2) man page says it plainly: on NFS the client kernel has to simulate append, and this cannot be done without a race condition.

So the client picks the offset itself, using the file size it believes is current. Two clients (two machines) each believe the file is 1000 bytes long. Both send a WRITE at offset 1000. The server applies them in order, and the second silently overwrites the first.

Quick detour: does it matter within one client?

Interestingly, two processes on the same NFS client are mostly fine, because the client's own inode lock serialises them. It is the second machine that breaks it. Attribute caching makes it worse: the client may use a file size that is seconds old.

That also explains why the bug hides in testing. Run both writers on one box and everything looks perfect; deploy to two hosts and the file falls apart.

Other ways appends go wrong, even locally

  • Buffered writers. A bufio.Writer flushes in chunks, so one logical line can span two write(2) calls with another process's data between them. Build the whole record, write it once.
  • Short writes. A full disk or a signal can make write return fewer bytes than asked. Go's os.File.Write reports that as an error, but the partial bytes are already in the file.
  • Pipes are different. Atomicity there is bounded by PIPE_BUF (4096 bytes on Linux), not by O_APPEND.

Related Go trap: WriteAt on a file opened with O_APPEND returns an error, since the kernel would ignore the offset anyway.

What to do on a shared mount

Pick one of these rather than hoping:

  1. One file per writer. Name it with hostname and PID, and merge when reading. This is the boring answer and it works everywhere.
  2. One writer process. Others send lines over a socket, or to syslog or journald, and a single process owns the file.
  3. Locking. POSIX fcntl locks do work over NFS (via lockd or NFSv4 locking), and taking or releasing one makes the client flush and revalidate its cache. It is slower, and you must lock around the whole size-check-and-write.
  4. Unique files per record, Maildir style: write a temp file, then rename it into a directory. Good for events, clumsy for logs.

If you want a real shared append-only log, a database or a message queue is a better tool than a file on a network share. The filesystem was never promising what you assumed it was.

Check your own setup with the test above before trusting any shared mount: if the count comes up short, you have your answer.

← All articles