Articles / Go

698 words

  • go
  • filepath-walkdir
  • filesystems
  • inotify
  • concurrency
  • debugging

Why Your Go Directory Walk Misses Files Created While You Read It

A file appears in a directory while filepath.WalkDir is running, and the walk never reports it. Run it again and it is there. Nothing is broken: a directory walk is not a snapshot, and the rules for what it sees are looser than most people assume.

What the walk actually does

filepath.WalkDir calls your function for a directory, then reads that directory's entries, then recurses. The read happens once per directory, at the moment the walk reaches it. Anything created in that directory afterwards is simply not in the list the walker already holds.

The entries are also sorted by name, which makes the order stable. It does not make the contents stable.

A deterministic demonstration

Races are annoying to demo, so let us cheat. The callback below creates two files the first time it sees directory a. By then the root has already been read; directory b has not.

package main

import (
	"fmt"
	"io/fs"
	"os"
	"path/filepath"
)

func write(path string) {
	if err := os.WriteFile(path, []byte("x"), 0o644); err != nil {
		panic(err)
	}
}

func main() {
	root, err := os.MkdirTemp("", "walk")
	if err != nil {
		panic(err)
	}
	defer os.RemoveAll(root)

	for _, d := range []string{"a", "b"} {
		if err := os.Mkdir(filepath.Join(root, d), 0o755); err != nil {
			panic(err)
		}
	}
	write(filepath.Join(root, "a", "one.txt"))
	write(filepath.Join(root, "b", "two.txt"))

	created := false
	err = filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
		if err != nil {
			return err
		}
		rel, _ := filepath.Rel(root, p)
		fmt.Println(rel)
		if d.Name() == "a" && !created {
			created = true
			write(filepath.Join(root, "late-top.txt"))
			write(filepath.Join(root, "b", "late-b.txt"))
		}
		return nil
	})
	if err != nil {
		panic(err)
	}
}

The output:

.
a
a/one.txt
b
b/late-b.txt
b/two.txt

late-top.txt is missing, late-b.txt is present. Same moment, same walk, different outcome, purely because of whether the parent had been read yet.

Quick detour: what the OS promises

Hang on, is Go to blame here? No. POSIX says that if entries are added to or removed from a directory after it was opened, it is unspecified whether readdir returns them. Go inherits that from the kernel interface (getdents64 on Linux).

That matters for big directories. os.ReadDir fetches entries in buffer-sized batches until end of directory, so even one directory is read over a stretch of time, not at one instant. A file created mid-read may land before or after the cursor, and that depends on the filesystem's internal ordering (hash order on ext4, for instance). So the answer is not just "missed" or "seen": it can be either, differently on different machines.

The other direction: files that vanish

The reverse also bites. A file listed during the read can be gone by the time you act on it.

  • With WalkDir, the type comes from the directory entry, so your callback runs happily on a deleted file.
  • Calling d.Info() afterwards does a fresh stat and can return an error satisfying errors.Is(err, fs.ErrNotExist).
  • With the older filepath.Walk, an lstat per entry happens up front, so the error reaches your callback instead.

Treat ErrNotExist mid-walk as normal, not fatal, unless you know nothing else touches the tree.

Renames are the nastiest case. Move a file from a directory not yet read into one already read and the walk never sees it. Move it the other way and it can be seen twice, once in each place. If you are counting, hashing or deleting based on a walk, that is a real bug source.

What to do about it

Pick based on what you need.

  1. Subscribe first, then walk. Start watching (inotify via a library such as fsnotify, or fanotify) before the walk begins, buffer events, then reconcile them against the walk results. Anything created after the watch started is in the event stream, whether or not the walk saw it. Remember that inotify queues can overflow, so handle that event by rescanning.
  2. Walk until stable. Walk, then walk again and compare. Stop when two passes agree. Cheap to write, fine for a one-off tool, and honest about being a heuristic: a file created between your last two passes is still missed.
  3. Change the protocol, not the walker. Have producers write to a temporary name and rename into a spool directory, and have the consumer claim files by renaming them out. Anything that is a complete file in the spool is safe to process, and anything missed is picked up on the next sweep. This is the design that actually removes the race.

The third is the one I would pick for anything long-lived. A directory listing is a hint about a moving target, and the sooner a design stops treating it as a fact, the fewer odd one-in-a-thousand misses you will chase.

If you only need a consistent view of a tree for a backup, the filesystem is the right layer: use a snapshot (LVM, btrfs, ZFS) and walk that instead.

← All articles