Recover Dirty Device-Mapper Cache Blocks with cache_writeback

You reach for cache_writeback when a failing SSD leaves dirty cache blocks stranded between the fast device and the slow origin. The installed command is cache_writeback 0.9.0 from thin-provisioning-tools 0.9.0-2ubuntu5.1. Allow at least fifteen minutes for the checks, then as long as the cache contents and devices require.

This is a recovery operation, not a routine cache-management command. You need three things before you start: the cache metadata, the slow origin device, and the fast device holding the cached data. You also need access to those paths, normally as root. Do not run this against a live cache.

1. Confirm the installed command

Start with read-only checks. They do not open your devices for writeback:

$ command -v cache_writeback
/usr/sbin/cache_writeback
$ cache_writeback --version
0.9.0
$ cache_writeback --help
Usage: cache_writeback [options]
        --metadata-device <dev>
        --origin-device <dev>
        --fast-device <dev>
        --buffer-size-meg <size>
        --list-failed-blocks
        --no-metadata-update

That help output came from the installed binary, so trust it over a command copied from a different thin-provisioning-tools release: option spelling has been known to shift between versions.

2. Take the cache offline

• Stop every workload that touches the cache. Unmount filesystems and stop dependent services through your normal incident procedure.
• Make sure it is not live. This tool cannot run against a live cache, and writing while another process holds the mapping can make the recovery unsafe.
• Know there is no dry run. There is no enable, disable, or preview option: the first invocation that names the devices performs writeback.

Treat the origin and fast-device arguments as destinations and sources, not interchangeable labels. Record the exact paths before you touch elevated privileges:

$ META='/path/to/cache-metadata'
$ ORIGIN='/dev/mapper/origin-volume'
$ FAST='/dev/mapper/fast-cache-volume'
$ printf 'metadata: %s\norigin: %s\nfast: %s\n' "$META" "$ORIGIN" "$FAST"
$ ls -l -- "$META" "$ORIGIN" "$FAST"

Replace every placeholder with a path from your recovery record, and confirm each one is what it claims to be. A syntactically valid path can still point at the wrong volume.

3. Decide how metadata should be handled

Normal operation writes dirty blocks to the origin and updates metadata to clear the dirty flags. Use that when you are restoring the origin as the authoritative copy and finishing the recovery.

--no-metadata-update suppresses that clearing. The manual gives decommissioning the cache as the reason to use it. It does not make the write read-only, and it does not make a failed write safe to ignore.

4. Run the writeback once

With the cache offline and the paths independently verified, run it. It needs elevated access whenever the device nodes or metadata file are not accessible to your user:

# cache_writeback \
    --metadata-device "$META" \
    --origin-device "$ORIGIN" \
    --fast-device "$FAST"
# status=$?
# printf 'cache_writeback exit status: %s\n' "$status"

Do not paste this until the variables hold real paths. A successful exit is the first checkpoint, not proof of success: check the origin and any failed-block report too. The tool is deliberately offline, so a service outage for the duration is expected. There is no documented progress format, so a quiet terminal does not mean nothing is happening.

5. List blocks that did not write back

If the writeback reports failures, or your procedure requires an explicit listing, add --list-failed-blocks:

# cache_writeback \
    --metadata-device "$META" \
    --origin-device "$ORIGIN" \
    --fast-device "$FAST" \
    --list-failed-blocks

The list is host- and incident-specific, so do not invent an expected block count. Save the output with the incident record and stop before bringing services back if any blocks failed.

6. Only touch the buffer size with a reason

--buffer-size-meg sets the size of the metadata cache the command uses. The installed manual gives a default of 16 Gig and says a larger size may improve performance; it gives no universal value for a particular device size or memory budget.

Leave the default unless your recovery plan has a measured reason to change it. If you do, record the value in the incident notes and keep the rest of the command identical:

# cache_writeback \
    --metadata-device "$META" \
    --origin-device "$ORIGIN" \
    --fast-device "$FAST" \
    --buffer-size-meg 32768

That value is an example of the option's shape, not a recommendation. Check available memory and the tool's behaviour under controlled conditions before choosing a larger buffer mid-incident.

7. Bring the service back only after verification

When the command exits with no reported failed blocks, verify the recovered origin using checks appropriate to the filesystem or application, such as confirming the expected mapping and mount exist, then running the application's own integrity check. Those checks sit outside cache_writeback, so their output depends on your storage stack.

Only then should you recreate or re-enable the cache, if your plan still calls for one, and remount or restart the dependent service. There is no undo for blocks already written to the origin. If the operation was interrupted or failed, keep the cache offline, preserve the output, and follow your storage team's recovery procedure rather than repeatedly guessing at device arguments.

If you used --no-metadata-update, a successful writeback is not permission to discard the metadata: the dirty flags were deliberately left alone. Resolve that as a separate, documented decommissioning or repair step.