Articles / Linux

720 words

  • linux
  • filesystems
  • rename
  • atomic-writes
  • go
  • debugging

Why rename() Is Atomic Within a Mount and mv Across Mounts Is Not

You write a config file to /tmp, mv it into /etc/myapp/, and assume readers will only ever see the old file or the new one. Sometimes that is true. Sometimes a reader sees half a file. The difference is whether /tmp and /etc live on the same mounted filesystem, and mv will not tell you which case you are in.

What rename() actually does

A file's name is just a directory entry pointing at an inode. rename(2) changes which entry points where. No file data moves at all, which is why it can be one indivisible step: the kernel updates the directory metadata under its own locking, and other processes see either the old name or the new one.

Two useful consequences follow:

  • If the destination already exists, it is replaced in the same step. There is no moment where it is missing.
  • A process that already had the old file open keeps its descriptor; it just points at the now-unlinked inode.

Why a second mount breaks the trick

Inodes belong to a filesystem. An inode number on /dev/sda2 means nothing on a tmpfs. To "rename" across them, something has to create a new inode on the target and copy the bytes, and that cannot be done in one step. So the kernel refuses and returns EXDEV, "Invalid cross-device link".

$ strace -e trace=rename,renameat2 mv /tmp/new.conf /etc/myapp/app.conf
renameat2(AT_FDCWD, "/tmp/new.conf", AT_FDCWD, "/etc/myapp/app.conf", RENAME_NOREPLACE) = -1 EXDEV (Invalid cross-device link)

The exact syscall and flags vary by coreutils version, but the EXDEV is the point.

What mv does when it gets EXDEV

mv catches that error and falls back to something different in kind: copy the file to the destination, copy over the attributes, then unlink the source. That fallback is not atomic. A reader can open the destination while it is half written, and a crash leaves a truncated file under the final name.

Nothing about the command line changes. Same mv, same arguments, quietly different guarantees. That is the bit I find slightly rude about it.

Quick detour: same filesystem is not enough

Linux documents a nuance here: rename() does not work across different mount points even when the same filesystem is mounted at both. A bind mount of a directory is a different mount point, so renaming across it gives EXDEV despite the data being on one disk.

Containers hit this a lot. Two volumes mounted into one container, or a bind-mounted config directory next to a path in the image layer, and a rename that worked on your laptop fails. Overlay filesystems can also return EXDEV in some cases, such as renaming a directory that exists in a lower layer, depending on how overlayfs is configured.

The pattern that survives: temp file in the destination directory

You cannot make the cross-mount copy atomic, but you can make the final step atomic. Copy to a temporary file in the same directory as the destination, then rename within that directory. Same directory means same mount, so the rename cannot hit EXDEV.

package main

import (
	"errors"
	"io"
	"os"
	"path/filepath"
	"syscall"
)

// Move renames src to dst, falling back to copy-then-rename when the
// two paths are on different mounts.
func Move(src, dst string) error {
	err := os.Rename(src, dst)
	if err == nil || !errors.Is(err, syscall.EXDEV) {
		return err
	}
	if err := copyAtomic(src, dst); err != nil {
		return err
	}
	return os.Remove(src)
}

func copyAtomic(src, dst string) (err error) {
	in, err := os.Open(src)
	if err != nil {
		return err
	}
	defer in.Close()

	info, err := in.Stat()
	if err != nil {
		return err
	}

	// Same directory as dst, so the final rename stays on one mount.
	tmp, err := os.CreateTemp(filepath.Dir(dst), ".move-*")
	if err != nil {
		return err
	}
	defer func() {
		if err != nil {
			tmp.Close()
			os.Remove(tmp.Name())
		}
	}()

	if _, err = io.Copy(tmp, in); err != nil {
		return err
	}
	if err = tmp.Chmod(info.Mode().Perm()); err != nil {
		return err
	}
	if err = tmp.Sync(); err != nil {
		return err
	}
	if err = tmp.Close(); err != nil {
		return err
	}
	return os.Rename(tmp.Name(), dst)
}

Three details matter here:

  1. errors.Is(err, syscall.EXDEV) works because os.Rename wraps the errno in a *os.LinkError.
  2. The temp file is created next to dst, not in /tmp. Using the default temp directory would reproduce the original problem.
  3. The source is removed only after the rename succeeds, so a failure never loses the only copy.

Atomic is not the same as durable

Atomic means other processes see old or new. It does not mean the new file survives a power cut. For that you need fsync on the file before the rename, and ideally on the containing directory after it. The code above syncs the file but skips the directory sync, which is the simplification to flag: add it if the destination must survive a crash.

The same-mount rule applies to the source side too. The unlink of src after the copy is a separate step, so a crash between the rename and the remove leaves the file in both places. That is duplication rather than corruption, which is the failure you can live with.

Spotting it in the wild

If you suspect a deployment script is crossing mounts, check with stat -c %d on both paths (it prints the device number), or findmnt -T path to see which mount owns each. Different answers mean mv is copying.

← All articles