Download Large Files Safely with lwp-download

You will finish with a repeatable way to fetch a large HTTP file into a deliberate destination, watch its progress, and check that the local copy exists. The examples use lwp-download from libwww-perl package version 6.76-1ubuntu0.1, as installed on this system.

Allow about ten minutes for a small test, plus the download time for the real file. You need a shell, a URL you trust, and write permission for the destination directory. Normal downloads do not need sudo. This command is a downloader, not a checksum verifier: if the publisher provides a checksum, verify it separately after the transfer.

1. Check the installed command

Confirm which executable will run and record the package version. These are ordinary, read-only commands:

$ command -v lwp-download
/usr/bin/lwp-download
$ dpkg-query -W -f='${Package} ${Version}\n' libwww-perl
libwww-perl 6.76-1ubuntu0.1

The installed interface is intentionally small:

$ lwp-download -h
Usage:
     lwp-download [-a] [-s] <url> [<local path>]

     Options:

       -a   save the file in ASCII mode
       -s   use HTTP headers to guess output filename

Do not treat arbitrary options from another downloader as available here. The supported switches are -a and -s, followed by a URL and, optionally, a local path.

2. Choose an explicit output file

An explicit file path is the least surprising form. Create or choose a directory you own, then pass the complete destination:

$ mkdir -p "$HOME/Downloads/release-test"
$ lwp-download https://example.invalid/releases/tool-1.2.3.tar.gz \
    "$HOME/Downloads/release-test/tool-1.2.3.tar.gz"
Saving to '/home/you/Downloads/release-test/tool-1.2.3.tar.gz'...
... bytes received

Replace the example URL and path with real values. The displayed byte count and transfer rate vary, so do not use the sample numbers as a test condition. The command writes progress while it receives the response; it does not keep the whole file in memory.

Overwrite warning: when the local path is not a directory, the manpage says an existing file is overwritten. Check first if the destination matters:

$ test ! -e "$HOME/Downloads/release-test/tool-1.2.3.tar.gz" && \
    echo 'destination is unused' || echo 'destination already exists'

If the file exists and you need to keep it, choose a different name. There is no undo operation for an overwrite unless you have another copy or backup.

3. Let the URL name a file in a directory

If the final argument names an existing directory, lwp-download appends the final URL path segment. This is convenient when the server URL has a useful filename:

$ mkdir -p "$HOME/Downloads/incoming"
$ lwp-download https://example.invalid/archive/project-1.2.3.tar.xz \
    "$HOME/Downloads/incoming"
Saving to '/home/you/Downloads/incoming/project-1.2.3.tar.xz'...
... bytes received
$ test -f "$HOME/Downloads/incoming/project-1.2.3.tar.xz" && \
    echo 'downloaded file is present'
downloaded file is present

Make the directory first. A path that does not already exist is treated as a file path, not as a directory to create. If the URL path ends in a slash, the default name is index.

Quote paths even when they currently contain no spaces. It keeps shell expansion from changing the destination if you later move the command into a script or use a path containing spaces.

4. Use server-provided filenames only when you need them

The -s option asks HTTP headers and redirect URLs for a filename. A server may provide one through Content-Disposition, and the command may add an extension based on the reported Content-Type:

$ mkdir -p "$HOME/Downloads/server-named"
$ lwp-download -s https://example.invalid/download?id=42 \
    "$HOME/Downloads/server-named"
Saving to '/home/you/Downloads/server-named/report.pdf'...
... bytes received

The exact name is controlled by the response, not by your shell command. This form fails if no acceptable name can be derived. It can also prompt before overwriting a produced filename. If standard input is not a terminal, the command fails rather than asking an unattended job to approve the overwrite.

That makes -s a poor fit for a blind scheduled download unless you first control the server response and the destination contents. For scripts, prefer an explicit, unique file path and check the command's exit status.

5. Verify the transfer and investigate failures

Check both the exit status and the expected file. A successful-looking progress line is not a substitute for checking the command result:

$ lwp-download https://example.invalid/releases/tool-1.2.3.tar.gz \
    "$HOME/Downloads/release-test/tool-1.2.3.tar.gz"
$ status=$?
$ printf 'lwp-download status: %s\n' "$status"
lwp-download status: 0
$ stat -c 'file=%n size=%s bytes' \
    "$HOME/Downloads/release-test/tool-1.2.3.tar.gz"
file=/home/you/Downloads/release-test/tool-1.2.3.tar.gz size=... bytes

The size is server- and file-specific. A non-zero status means the fetch did not complete successfully; preserve the diagnostic text and investigate the URL, network path, response status or destination permissions. For example, a missing resource on the test system produced lwp-download: 404 File not found and status 1.

For a release artefact, run the publisher's verification command after this step. A downloaded file can be complete yet untrusted, corrupted in transit, or replaced at the source. Use the checksum algorithm and expected digest published by the same trusted project, and stop if they disagree.

6. Keep the ASCII option in its narrow place

-a requests ASCII, or text, mode. The manpage says this might make a difference on DOS-like systems. It is usually not useful for binary archives, images or installers, and it does not turn a binary download into a safe text conversion. Leave it out for ordinary Linux downloads:

$ lwp-download -a https://example.invalid/notes.txt \
    "$HOME/Downloads/notes.txt"
... bytes received

Use it only when you understand the text-mode behaviour required by the destination system. Do not add it to a command merely because the downloaded file has a human-readable name.