Safely Deduplicate a Directory with hardlink

Dozens of identical files quietly eating disk space is what the hardlink command from util-linux exists to fix. This guide previews the proposed changes, replaces identical files with hardlinks, and checks the result by inode. Allow about ten minutes for a small directory, longer for a large tree or slow storage. The version used here is 2.41.3.

Hardlinking changes filesystem metadata. Once two names are linked, they refer to the same inode: editing either name changes the same data. Keep a backup and do not run this against a tree another process or user can change.

1. Prepare a stable directory

Choose one or more directories that you own and can keep unchanged during the scan. The command accepts directories or files. Do not point it at a live download directory, a database, a mail spool, or a tree a service is actively modifying. The local manual warns that changing a path during the operation can produce undefined and potentially dangerous results.

Use an ordinary account for a directory you can read and write. Elevated privileges are not a normal prerequisite. If the tree contains protected files, stop and decide whether changing its ownership and permissions is actually appropriate before reaching for sudo.

Checkpoint: You have a stable target and a backup or other recovery route.

2. Preview the candidates

Start with a dry run. The verbose option makes proposed links visible, while --ignore-time permits equal content whose modification times differ; without it, differing timestamps block a match unless your other equality requirements already make them equal.

$ hardlink --dry-run --verbose --ignore-time /path/to/archive

The command does not modify files in dry-run mode. On this util-linux build, a match is reported in a line such as:

[DryRun] Linking /path/to/archive/old/report.txt to /path/to/archive/new/report.txt (-13 B)

Review the paths. If the output includes files that must remain separate, stop and narrow the scan before running a real pass.

3. Understand what counts as equal

By default, hardlink groups files by size, compares their content, and also checks file attributes. The manual lists owner, mode, timestamps and extended attributes as boundaries that can stop a link. The default content method on this installation is SHA-256, falling back to memcmp when the Linux Crypto API is unavailable.

--ignore-time is usually useful when duplicate copies were created at different times. Use --respect-name when only identical base names should be considered, or --respect-dir when matching files must occupy the same relative directory name. Combine both and the relative paths, apart from the top-level argument, must match.

--ignore-owner and --ignore-mode broaden the set of files that may be linked, and that can be unsafe: the resulting shared inode has one owner and one mode. Use them only when that shared metadata is intentional. The manual says results may be unpredictable for these options.

4. Run the real pass

Once the dry-run paths look right, repeat the command without --dry-run:

$ hardlink --verbose --ignore-time /path/to/archive

Expect a summary containing the mode, comparison method, number of files, linked files and saved bytes. The command keeps one name per group; by default it keeps the newest file, so add --keep-oldest if the oldest copy is the one you want retained. --maximize and --minimize choose by existing link count and take precedence over age.

This operation is not undone by another hardlink command. The data stays available through the surviving names, but the old independent inode is gone. To recover the previous arrangement, restore the affected files from your backup. If you need a reversible trial, stop after the dry run.

Checkpoint: The command has completed, and its summary reports the expected number of links.

5. Verify shared inodes

Check two paths reported as duplicates. Matching inode numbers and a link count of at least two confirm they now share storage:

$ stat -c '%n inode=%i links=%h size=%s' \
    /path/to/archive/old/report.txt \
    /path/to/archive/new/report.txt
/path/to/archive/old/report.txt inode=123456 links=2 size=13
/path/to/archive/new/report.txt inode=123456 links=2 size=13

For a content check, compare hashes without changing either file:

$ sha256sum /path/to/archive/old/report.txt /path/to/archive/new/report.txt

Do not infer success from saved bytes alone. Check representative paths, especially where ownership, modes, timestamps or extended attributes mattered.

6. Handle common boundaries

Empty files are not linked by default, since the minimum size is 1 byte. Set --minimum-size explicitly if you have a reason to include or exclude a size range. The maximum size defaults to unlimited. Include and exclude filters take regular expressions, so quote them to keep the shell from interpreting special characters:

$ hardlink --dry-run --verbose --ignore-time \
    --exclude '(^|/)cache(/|$)' /path/to/archive

If the target spans mounted filesystems and you need a strict boundary, the installed 2.41.3 command also provides --mount. That newer option is not present in the local util-linux 2.39.3 manual page, so check hardlink --help on the machine where a script will run, and keep scripts explicit about the version they require.

For copy-on-write clones instead of shared hardlinks, --reflink=always permits only reflinks, while --reflink=auto can fall back to hardlinks. The manual documents filesystem detection for BTRFS and XFS. Reflinks still share on-disk data, but can carry different mode and owner metadata.

Done means