curl without surprises: save, redirect and check status

By default curl will happily save a 404 error page and call it a success. Here is how to make curl fail loudly, save safely and follow redirects on purpose. You will build a small, repeatable workflow in about fifteen minutes.

1. Check which curl you have

Find out which binary and version your shell will actually run:

$ command -v curl
/usr/bin/curl
$ curl --version | sed -n '1,2p'
curl 8.5.0 (x86_64-pc-linux-gnu) libcurl/8.5.0 OpenSSL/3.0.13 ...
Release-Date: 2023-12-06, security patched: 8.5.0-2ubuntu10.13

The feature list depends on the build. This one supports HTTP and HTTPS, among other protocols. Another machine's curl may not share the same protocol support or option behaviour just because it has the same name.

Checkpoint: Note the version before copying a command into a script. The installed manual is the authority for this host.

2. Look before you download

With no output option, curl writes the body to standard output. Fine for a bit of text, a poor default for anything large or binary.

When you only need the headers, use -I (also spelled --head):

$ curl --head https://example.com
HTTP/2 200
content-type: text/html
...

For a compact result with no body, throw the body away and ask curl to report selected transfer details:

$ curl --silent --show-error --output /dev/null \
    --write-out 'status=%{response_code} effective=%{url_effective}\n' \
    https://example.com
status=200 effective=https://example.com/

3. Save to a new path

Use --output to write the response to a named file:

$ curl --fail --silent --show-error \
    --output response.html \
    https://example.com/
$ test -s response.html && echo 'saved a non-empty response'
saved a non-empty response

--fail turns HTTP responses of 400 and above into exit status 22, instead of treating the error page as a good download. It does not validate the document's contents, and it does not give every transport problem the same code. Always check the exit status in automation.

Warning: Saving to an existing path can overwrite it, and curl has no undo. Pick a new name, or download to a temporary file in the same directory and replace the destination only after checking it.

$ curl --fail --silent --show-error \
    --output response.html.new \
    https://example.com/
$ test -s response.html.new
$ mv response.html.new response.html

Recovery: If curl fails, do not run the mv line. Remove the incomplete .new file only once you have confirmed it is the temporary output you meant to discard. If the final file was replaced by accident, restore it from your backup or filesystem snapshot.

4. Follow redirects on purpose

curl does not follow HTTP redirects by default. Add --location when the URL may redirect, such as a download link that points at the current version:

$ curl --fail --silent --show-error --location \
    --output response.html \
    https://example.com/
$ curl --silent --show-error --output /dev/null \
    --write-out 'status=%{response_code} redirects=%{num_redirects} effective=%{url_effective}\n' \
    --location https://example.com
status=200 redirects=0 effective=https://example.com/

On a redirecting URL, num_redirects counts the redirects followed and url_effective shows where you ended up.

Warning: Treat a change of host as a security boundary. Credentials, cookies, headers or uploaded data can have consequences when sent along a redirect chain. For sensitive requests, inspect the redirect first with verbose output or a header-only request, then decide whether --location is appropriate.

5. Make scripts fail clearly

A script should never wait forever. Bound the request:

$ curl --fail --silent --show-error \
    --connect-timeout 5 --max-time 20 \
    --output response.html \
    https://example.com/
$ printf 'curl exit status: %s\n' "$?"
curl exit status: 0

Do not print a success message before checking the exit status. In a script, make the check part of the same decision:

if curl --fail --silent --show-error --location \
    --connect-timeout 5 --max-time 20 \
    --output response.html.new https://example.com/; then
    test -s response.html.new && mv response.html.new response.html
    printf '%s\n' 'download completed'
else
    status=$?
    rm -f response.html.new
    printf 'curl failed with exit status %s\n' "$status" >&2
    exit "$status"
fi

On failure it removes only its own temporary path and leaves the previous response.html alone.

Exit statuses worth knowing:

Tip: The diagnostic text and the status together tell you more than either alone.

6. Debug without switching off TLS

When a transfer fails, add --verbose for request and connection detail:

$ curl --verbose --fail --output /dev/null https://example.com
* Connected to example.com (...)
> GET / HTTP/2
< HTTP/2 200
...

Addresses, headers and server details will vary.

Warning: Do not paste verbose output into a public issue without checking it for cookies, authorisation headers, usernames or other sensitive values.

Warning: Do not reach for --insecure as a routine fix for an HTTPS certificate error. It disables certificate verification and can make a man-in-the-middle attack look like a successful download.

Check these instead:

These are security-sensitive choices. curl itself needs no root for them, but installing a system-wide CA may need an administrator and should follow your distribution's process.

7. Keep request data separate from downloads

For a form or API request, --data sends data rather than just fetching a page.

Quote URLs containing shell metacharacters such as &, ? or *. curl's own URL globbing uses braces and brackets, and the shell can interpret several of the same characters:

$ curl --fail --silent --show-error \
    --output search.html \
    'https://example.com/search?q=linux&page=1'

Warning: Single quotes stop the shell treating the query as syntax. They do not make it safe to send data to that host, so review the final URL before running a command copied from an untrusted source.

Done means