fsync Lies: What Linux Actually Guarantees About a 'Durable' Write

Call fsync(), get zero back, and the natural assumption is that the bytes you just wrote now exist somewhere that survives a power cut. That assumption is doing a lot of unexamined work. The POSIX contract for fsync is narrower than most people think, the Linux implementation of that contract had a genuinely alarming bug for the better part of two decades, and even a perfectly behaved kernel is only as honest as the hardware underneath it. None of this is obscure trivia: it is the reason PostgreSQL, one of the most carefully engineered pieces of storage software there is, spent 2018 rewriting its assumptions about what "the write succeeded" means.

What fsync is actually contracted to do

A regular write to a file descriptor on Linux does not touch storage at all in the common case. It copies data into the page cache and marks the affected pages dirty. The kernel writes those pages back to the device later, on its own schedule, driven by memory pressure and the periodic writeback timers. This is why a process can write gigabytes to disk in milliseconds: it isn't writing to disk, it's writing to RAM that the kernel has promised to reconcile with disk eventually.

fsync(fd) is the call that forces "eventually" to mean "now". It flushes the dirty pages belonging to that file to the underlying block device and, where the device honours it, issues a cache flush so the device's own volatile write cache is emptied to persistent media too. fdatasync(fd) does almost the same thing but skips metadata updates that aren't needed to read the data back correctly, such as pure timestamp changes. If you don't care about mtime surviving a crash, fdatasync is strictly cheaper and just as safe for the data itself. Go's standard library only exposes the fsync variant, as (*os.File).Sync, so if you want fdatasync semantics on Linux you have to reach for a syscall wrapper.

Both calls are defined to block until the data has reached "stable storage" and to return an error if it didn't. That error part is where things got interesting.

The bug that made fsync lie for twenty years

In April 2018, a PostgreSQL contributor investigating a report of corruption on a system with a briefly failing storage array traced the cause back to how Linux handles write-back errors. The scenario: the kernel tries to flush a dirty page, the underlying device or block layer reports an error (an EIO, say, because a disk briefly dropped off a RAID array), and the kernel marks the page clean anyway and discards it, because there's nowhere sensible to put data the hardware just refused. PostgreSQL, following what seemed like a reasonable reading of the man page, called fsync() again after seeing an error, on the theory that it would retry the failed writeback and report success once the underlying problem cleared. Instead, the next fsync call had nothing left to complain about, because the dirty page was already gone, and it returned success. The data that triggered the original error was never written, and nothing said so a second time.

Worse, the error was reported to whichever file descriptor happened to call fsync first, not to every open descriptor for that file. A process holding a second file descriptor onto the same file, which is the normal way a database checkpoints, could call fsync and get a clean bill of health despite an error having already fired against a sibling descriptor moments earlier.

Linux 4.13 introduced errseq_t, a per-address-space error counter, so that every file descriptor open at the time of an error is guaranteed to observe it at least once on its next fsync call, regardless of which descriptor triggered the original writeback. That closed the "wrong descriptor sees the error" hole. It did not, and could not, change the fact that once a writeback error occurs, the kernel has already given up on those specific dirty pages: there is no retry, because the data that failed to write is no longer resident anywhere to retry with. PostgreSQL's actual fix, landed for the 9.4 through 12 release lines in early 2019, was to treat any fsync failure as unrecoverable and immediately crash the whole postmaster process, forcing recovery from the last WAL checkpoint rather than silently pretending the on-disk state was fine. Dan Luu's write-up of the saga is a good survey of the mailing list argument, and the PostgreSQL wiki page on the issue lays out the practical guidance that came out of it: on Linux, an fsync error means the file descriptor, and arguably the whole cached state of that file, must be considered untrustworthy. You panic, you don't retry.

The directory you forgot to fsync

Even with a kernel that reports errors faithfully, there's a second gap that catches people who've never hit it: fsyncing a file says nothing about whether the file's directory entry survives a crash. Creating a new file, or renaming one over another, changes the containing directory's own metadata, and that change lives in its own set of dirty pages that your fsync on the file descriptor never touches.

The standard pattern for an atomic config or checkpoint replacement, write to a temp file, fsync it, rename it over the target, is incomplete without a final fsync on the directory. Skip that step and a crash between the rename and the next background writeback can leave you back with the old file, or in pathological cases no file at all, even though the new file's contents were safely on disk the whole time.

func atomicWriteFile(path string, data []byte) error {
	dir := filepath.Dir(path)

	tmp, err := os.CreateTemp(dir, ".tmp-*")
	if err != nil {
		return fmt.Errorf("create temp file: %w", err)
	}
	defer os.Remove(tmp.Name()) // no-op once the rename below succeeds

	if _, err := tmp.Write(data); err != nil {
		tmp.Close()
		return fmt.Errorf("write temp file: %w", err)
	}
	if err := tmp.Sync(); err != nil {
		tmp.Close()
		return fmt.Errorf("fsync temp file: %w", err)
	}
	if err := tmp.Close(); err != nil {
		return fmt.Errorf("close temp file: %w", err)
	}
	if err := os.Rename(tmp.Name(), path); err != nil {
		return fmt.Errorf("rename into place: %w", err)
	}

	d, err := os.Open(dir)
	if err != nil {
		return fmt.Errorf("open directory: %w", err)
	}
	defer d.Close()
	if err := d.Sync(); err != nil {
		return fmt.Errorf("fsync directory: %w", err)
	}
	return nil
}

Checking the error from Close as well as Sync matters for the same reason as the errseq_t story: on some filesystems and in some error paths, an error that occurred during writeback only surfaces when the descriptor is finally closed. Treating Close as a formality that can't fail is a habit worth breaking here specifically.

Below the block layer, the hardware gets a vote too

Everything above assumes the storage device does what it's told when the kernel issues a cache flush (an ATA FLUSH CACHE or SCSI SYNCHRONIZE CACHE command, depending on the transport). Consumer SSDs almost universally have a volatile DRAM cache in front of the flash, and cheaper drives often lack the capacitor-backed power-loss protection that lets a drive finish committing that cache during a power failure. A flush command tells the drive to empty its cache to flash; whether the drive actually waits for that to complete before acknowledging, rather than just acknowledging immediately because it's faster, is a property of the specific drive firmware, not something Linux can verify from the host side.

Virtualised and cloud block storage adds another layer of the same problem. A flush issued inside a guest has to be honoured by the hypervisor's virtual disk backend and, in turn, by whatever the backend is actually built on. Historically, some virtual disk formats and some early cloud block storage implementations acknowledged flush requests without a durable write actually having completed, on the reasoning that replication elsewhere made the local flush redundant. That reasoning may or may not hold depending on the failure mode you're worried about, and it is not something you can inspect from inside a guest kernel. If you are building something that has to survive power loss, the honest answer is that fsync returning success tells you Linux did its part; whether the layers below it did theirs is a question you can only answer by asking your storage vendor directly, or by testing it, which is precisely what tools like ALICE and CrashMonkey from academic filesystem research groups were built to do: inject crashes at every point in a write sequence and check what actually survives.

None of this is an argument against fsync. It's the only tool that gives you a durability boundary at all, and a correctly used fsync on a correctly behaving stack does exactly what the man page says. It's an argument for being specific about which of those two conditions you're actually relying on, because in the specific case of Linux buffered I/O before 4.13, and in the general case of "what does this particular disk do when I tell it to flush", the honest answer for a long time was: less than the return value suggested.