Articles / Systems Programming

1447 words

  • linux
  • fsync
  • filesystems
  • durability
  • go
  • storage

What Linux Actually Promises When fsync Returns

Call fsync(), get zero back, and the natural assumption is that the bytes you just wrote now exist somewhere that survives a power cut. That assumption is doing a lot of unexamined work.

Three things undercut it:

  • the POSIX contract for fsync is narrower than most people think;
  • the Linux implementation of that contract had a genuinely alarming bug for the better part of two decades;
  • even a perfectly behaved kernel is only as honest as the hardware underneath it.

None of this is obscure trivia. It is the reason PostgreSQL, one of the most carefully engineered pieces of storage software there is, spent 2018 rewriting its assumptions about what "the write succeeded" means.

A write lands in RAM; fsync is what drags it to disk

A regular write to a file descriptor on Linux does not touch storage at all in the common case. It copies data into the page cache and marks the affected pages dirty. The kernel writes those pages back to the device later, on its own schedule, driven by memory pressure and the periodic writeback timers.

This is why a process can write gigabytes to disk in milliseconds: it isn't writing to disk, it's writing to RAM that the kernel has promised to reconcile with disk eventually.

fsync(fd) is the call that forces "eventually" to mean "now". It flushes the dirty pages belonging to that file to the underlying block device and, where the device honours it, issues a cache flush so the device's own volatile write cache is emptied to persistent media too.

fdatasync(fd) does almost the same thing but skips metadata updates that aren't needed to read the data back correctly, such as pure timestamp changes. If you don't care about mtime surviving a crash, fdatasync is strictly cheaper and just as safe for the data itself.

Quick Go aside: the standard library only exposes the fsync variant, as (*os.File).Sync, so if you want fdatasync semantics on Linux you have to reach for a syscall wrapper.

Both calls are defined to block until the data has reached "stable storage" and to return an error if it didn't. That error part is where things got interesting.

The bug that made fsync lie for twenty years

In April 2018, a PostgreSQL contributor investigating a report of corruption on a system with a briefly failing storage array traced the cause back to how Linux handles write-back errors.

The scenario went like this:

  1. The kernel tries to flush a dirty page.
  2. The underlying device or block layer reports an error (an EIO, say, because a disk briefly dropped off a RAID array).
  3. The kernel marks the page clean anyway and discards it, because there's nowhere sensible to put data the hardware just refused.

PostgreSQL, following what seemed like a reasonable reading of the man page, called fsync() again after seeing an error. The theory was that it would retry the failed writeback and report success once the underlying problem cleared.

Instead, the next fsync call had nothing left to complain about, because the dirty page was already gone, and it returned success. The data that triggered the original error was never written, and nothing said so a second time.

Worse: the error went to the wrong descriptor

The error was reported to whichever file descriptor happened to call fsync first, not to every open descriptor for that file.

A process holding a second file descriptor onto the same file, which is the normal way a database checkpoints, could call fsync and get a clean bill of health. That was despite an error having already fired against a sibling descriptor moments earlier.

What got fixed, and what can't be

Linux 4.13 introduced errseq_t, a per-address-space error counter. Every file descriptor open at the time of an error is now guaranteed to observe it at least once on its next fsync call, regardless of which descriptor triggered the original writeback. That closed the "wrong descriptor sees the error" hole.

It did not, and could not, change the fact that once a writeback error occurs, the kernel has already given up on those specific dirty pages. There is no retry, because the data that failed to write is no longer resident anywhere to retry with.

PostgreSQL's actual fix, landed for the 9.4 through 12 release lines in early 2019, was to treat any fsync failure as unrecoverable. It immediately crashes the whole postmaster process, forcing recovery from the last WAL checkpoint rather than silently pretending the on-disk state was fine.

Dan Luu's write-up of the saga is a good survey of the mailing list argument. The PostgreSQL wiki page on the issue lays out the practical guidance that came out of it: on Linux, an fsync error means the file descriptor, and arguably the whole cached state of that file, must be considered untrustworthy. You panic, you don't retry.

The directory you forgot to fsync

Even with a kernel that reports errors faithfully, there's a second gap that catches people who've never hit it: fsyncing a file says nothing about whether the file's directory entry survives a crash.

Creating a new file, or renaming one over another, changes the containing directory's own metadata. That change lives in its own set of dirty pages that your fsync on the file descriptor never touches.

The standard pattern for an atomic config or checkpoint replacement goes: write to a temp file, fsync it, rename it over the target. It is incomplete without a final fsync on the directory.

Skip that step and a crash between the rename and the next background writeback can leave you back with the old file, or in pathological cases no file at all. That's even though the new file's contents were safely on disk the whole time.

func atomicWriteFile(path string, data []byte) error {
	dir := filepath.Dir(path)

	tmp, err := os.CreateTemp(dir, ".tmp-*")
	if err != nil {
		return fmt.Errorf("create temp file: %w", err)
	}
	defer os.Remove(tmp.Name()) // no-op once the rename below succeeds

	if _, err := tmp.Write(data); err != nil {
		tmp.Close()
		return fmt.Errorf("write temp file: %w", err)
	}
	if err := tmp.Sync(); err != nil {
		tmp.Close()
		return fmt.Errorf("fsync temp file: %w", err)
	}
	if err := tmp.Close(); err != nil {
		return fmt.Errorf("close temp file: %w", err)
	}
	if err := os.Rename(tmp.Name(), path); err != nil {
		return fmt.Errorf("rename into place: %w", err)
	}

	d, err := os.Open(dir)
	if err != nil {
		return fmt.Errorf("open directory: %w", err)
	}
	defer d.Close()
	if err := d.Sync(); err != nil {
		return fmt.Errorf("fsync directory: %w", err)
	}
	return nil
}

Checking the error from Close as well as Sync matters for the same reason as the errseq_t story. On some filesystems and in some error paths, an error that occurred during writeback only surfaces when the descriptor is finally closed.

Treating Close as a formality that can't fail is a habit worth breaking here specifically.

Below the block layer, the hardware gets a vote too

Everything above assumes the storage device does what it's told when the kernel issues a cache flush (an ATA FLUSH CACHE or SCSI SYNCHRONIZE CACHE command, depending on the transport).

Consumer SSDs almost universally have a volatile DRAM cache in front of the flash. Cheaper drives often lack the capacitor-backed power-loss protection that lets a drive finish committing that cache during a power failure.

A flush command tells the drive to empty its cache to flash. Whether the drive actually waits for that to complete before acknowledging, rather than just acknowledging immediately because it's faster, is a property of the specific drive firmware, not something Linux can verify from the host side.

Virtual disks add another layer of the same problem

A flush issued inside a guest has to be honoured by the hypervisor's virtual disk backend and, in turn, by whatever the backend is actually built on.

Historically, some virtual disk formats and some early cloud block storage implementations acknowledged flush requests without a durable write actually having completed. The reasoning was that replication elsewhere made the local flush redundant.

That reasoning may or may not hold depending on the failure mode you're worried about, and it is not something you can inspect from inside a guest kernel.

So what does a successful fsync tell you?

If you are building something that has to survive power loss, the honest answer is that fsync returning success tells you Linux did its part. Whether the layers below it did theirs is a question you can only answer in one of two ways:

  • ask your storage vendor directly; or
  • test it.

Testing is precisely what tools like ALICE and CrashMonkey from academic filesystem research groups were built to do. They inject crashes at every point in a write sequence and check what actually survives, which is a delightfully brutal way to find out.

None of this is an argument against fsync. It's the only tool that gives you a durability boundary at all, and a correctly used fsync on a correctly behaving stack does exactly what the man page says.

It's an argument for being specific about which of those two conditions you're actually relying on. In the specific case of Linux buffered I/O before 4.13, and in the general case of "what does this particular disk do when I tell it to flush", the honest answer for a long time was: less than the return value suggested.

← All articles