Crawl a Website into an Encrypted Vault with vault mirror

vault mirror crawls a site on a single host and stores every page and asset in the vault, with links rewritten so the copy browses offline. This guide runs a mirror, explains how names are chosen and lists what it cannot do.

Allow about ten minutes to set up, plus the crawl time. You need a key, an initialised vault and permission to copy the site. It has no depth limit, rate limit or robots.txt handling, so only mirror sites you may copy.

1. Run a basic mirror

$ vault mirror https://example.com/

Everything lands beneath HOST, here example.com/. The crawl is single threaded, one request at a time, in breadth-first order. It follows only links on the same host as the URL, and sends a Chrome user agent string.

2. File it in a folder

$ vault mirror https://example.com/ archive/2026

With a folder the result sits beneath FOLDER/HOST, so you can keep several dated snapshots side by side.

3. Know how pages are named

Pages are named after their URL path. A path that is empty or ends in / becomes index.html. A query string is appended as _query, URL-escaped. Anything already stored under the same name is replaced, so a repeat mirror refreshes the snapshot.

4. Check the rewrite

Links in HTML (href, src, action and poster attributes) and in CSS url(...) that point into the mirror are rewritten to relative paths. Links to other hosts, and fragments, are left as they were. Fetch the front page back to inspect it:

$ vault get archive/2026/example.com/index.html -o /tmp/index.html

5. Expect failures and limits

A page that fails to download or answers with anything but 200 OK is reported on standard error and skipped. The crawl carries on. The vault is locked exclusively for the whole run, so other writers wait.