Mirror a Web Document Only When It Changes with lwp-mirror

lwp-mirror does one small job well: keep a single local copy of a document fresh without refetching it every time. This is useful for a small, predictable cache of a file such as a feed, policy document or release note. It is not a whole-site crawler: one invocation handles one URL and one local file.

Allow about five minutes for a first run. You need libwww-perl and a writable directory. The examples below were checked with libwww-perl 6.76 and its installed lwp-mirror(1p) manpage on this machine. No elevated privileges are needed when the destination belongs to your user.

1. Confirm the Command and Version

Check that the executable is installed before preparing a destination:

command -v lwp-mirror
lwp-mirror -v

The version check prints a line like this:

This is lwp-mirror version 6.76 (libwww-perl-6.76)

The installed command accepts two options. -v prints its version. -t sets the wait time for a response, in seconds by default; append m for minutes or h for hours, such as -t 2m. The command then needs a URL followed by a local file path:

lwp-mirror [-v] [-t timeout] <url> <local-file>

2. Choose an Isolated Destination

Create a directory whose contents you are happy for the command to update. Keep the destination as a file inside it, rather than pointing at an important configuration file or a path served directly by a web server.

mkdir -p "$HOME/tmp/lwp-mirror"
target="$HOME/tmp/lwp-mirror/example.html"

Replace the URL in the next step with the document you actually need. The parent directory must exist and the file must be writable if it already exists. If you use a shared or privileged destination, stop and decide who should own the file before proceeding: running the command as root merely makes it easier to overwrite something valuable, not safer.

3. Fetch the Document with a Bounded Wait

Run the command with a timeout appropriate to the network you are using. Thirty seconds is a reasonable starting point for a small HTTPS document:

lwp-mirror -t 30 \
  https://www.example.com/ \
  "$target"

On success, the destination is created or updated and the command exits with status zero. The exact document and its metadata depend on the server, so verify the local result rather than treating a successful exit as proof that the content is the one you intended:

test -s "$target" && sed -n '1,5p' "$target"
printf 'bytes: '
wc -c < "$target"

Use a real document URL in production. The installed program is built on the LWP library, so the protocols it can mirror are limited to those supported by that library. An HTTPS URL is appropriate when the server provides HTTPS. A redirect, authentication requirement or unsupported protocol can make the request fail; inspect the diagnostic and the exit status instead of silently substituting another URL.

4. Run It Again as a Conditional Update

Repeat the same command with the same local path:

lwp-mirror -t 30 \
  https://www.example.com/ \
  "$target"
printf 'exit status: %s\n' "$?"

If the remote document is not newer than the local copy, lwp-mirror leaves the file alone and prints a message in this form:

lwp-mirror: /home/you/tmp/lwp-mirror/example.html is up to date

Common traps

• Keep the timeout explicit. A timeout is the time lwp-mirror waits for a response before failing; it is not a guarantee that the complete transfer will fit into an overall job budget. A large file or slow connection can still overrun your scheduler.
• Do not expect a recursive mirror. This is not a site sync tool: it does not discover links, build a directory tree or crawl. Give each document its own known target, and avoid a shared filename for unrelated URLs, because a later invocation can replace the earlier content when the remote copy is newer.
• Quote paths and URLs in scripts. This matters when a local path contains spaces or a URL includes shell-significant characters. Prefer a fixed target directory and a literal filename over building a path from untrusted URL text.

A non-zero exit status means the requested operation did not complete successfully. Capture it in a script immediately if you need to branch on failure:

if lwp-mirror -t 30 \
    https://www.example.com/ \
    "$target"; then
  printf '%s\n' 'mirror check succeeded'
else
  status=$?
  printf 'mirror failed, exit status: %s\n' "$status" >&2
  exit "$status"
fi

Recovery and undo

rm -- "$target"

That removal is irreversible unless another copy exists. If an existing file was updated, deleting it is not an undo operation: recover the previous contents from backup. To prevent another update, change the scheduled job or script that calls lwp-mirror rather than relying on the file's timestamp.

Done means

• Version confirmed. lwp-mirror -v identified the installed libwww-perl version.
• Target explicit. The URL and local target are explicit, quoted and intended to be used together.
• First fetch verified. The first run created a readable, non-empty local document.
• Conditional update proven. A repeat run reported the file was up to date when the remote copy had not changed.
• Failures visible. The timeout is bounded and failures are visible to the calling script.
• Recovery known. You know which backup or version-control copy can restore the target if it is updated incorrectly.