Probe HTTP Endpoints Safely with lwp-request

lwp-request gives you GET, HEAD and POST as three separate command names, which is handy when you want a quick probe without remembering curl flags. The examples use it from libwww-perl 6.76, installed here as Debian package version 6.76-1ubuntu0.1.

Allow about fifteen minutes. You need a shell, a URL you are allowed to query, and permission to read any local file used as a request body. No command in this guide needs elevated privileges. Do not test an unfamiliar endpoint with POST: a request can create, update or delete remote data.

1. Check the Installed Command

Confirm which executable will run and record the local version. These are read-only checks:

$ command -v lwp-request
/usr/bin/lwp-request
$ lwp-request -v
This is lwp-request version 6.76 (libwww-perl-6.76)
$ dpkg-query -W -f='${Package} ${Version}\n' libwww-perl
libwww-perl 6.76-1ubuntu0.1

The four command names share the same options. If you omit -m, the method comes from the program name: GET sends GET, HEAD sends HEAD and POST sends POST. lwp-request itself defaults to GET. The -m option lets you choose a method explicitly.

Checkpoint: run lwp-request -h if you need the installed usage text. Do not rely on a flag remembered from another HTTP client.

2. Make a Basic GET Request

Start with a resource that is safe to retrieve. Substitute an endpoint you control or have permission to query:

$ GET https://example.test/health

The response body is written to standard output. Diagnostics go to standard error. A successful exit status means the request completed for the URL; it does not mean the server returned a 2xx status. Show the status code alongside the body when that distinction matters:

$ GET -s https://example.test/health
200 OK
healthy

The exact status line and body are server-controlled. Capture them separately when a response will feed another command:

$ GET -s https://example.test/health > response.txt
$ status=$?
$ printf 'request exit status: %s\n' "$status"
$ sed -n '1,20p' response.txt

Do not confuse the shell exit status with the HTTP status. For several URLs, the program returns a value indicating how many URLs failed, so check it before treating a batch as complete.

3. Inspect Headers with HEAD

Use HEAD when you need response metadata but not the body:

$ HEAD https://example.test/health
200 OK
Content-Type: text/plain
Content-Length: 7

HEAD requests automatically display the response status and headers. Header values vary, and some servers implement HEAD poorly, so a failed or misleading HEAD response is not proof that GET will behave identically. To see a redirect or authentication sequence, add -S; to show full headers for every response in that sequence, use -E.

Checkpoint: use HEAD -s -e URL when you want the status and headers to be explicit in a script or a troubleshooting transcript. The -e option displays response headers and implies the status display.

4. Make Requests Visible While Diagnosing Them

When a URL, proxy or redirect is confusing, print the request details:

$ GET -U -S -e https://example.test/health
GET https://example.test/health
...
200 OK
...

The exact header block depends on the server, the LWP defaults and any proxy. -u prints the method and absolute URL; -U also prints request headers. -s prints the response status; -S prints the status chain, including redirects and authorisation requests handled by the library. Use -d to suppress the response body when you only want diagnostics.

Warning: request headers can contain identifying or sensitive information. Treat captured -U output as operational data and redact it before sharing. Never paste a password into a public issue or shell history.

5. Send a POST Body Deliberately

For POST, PUT and PATCH, request content is read from standard input. The default POST content type is application/x-www-form-urlencoded. Send only a harmless test field to an endpoint designed for it:

$ printf '%s\n' 'message=hello&source=cli' \
    | POST -s -c application/x-www-form-urlencoded \
      https://example.test/test-form
200 OK
accepted

Safety warning: POST is a state-changing operation unless the API explicitly documents otherwise. Use a staging URL, a disposable record or a documented dry-run endpoint. The command has no undo operation for a remote change. If the request created data, remove it using that service's documented recovery procedure, not by guessing at a second request.

For JSON, declare the content type and provide valid JSON as standard input:

$ printf '%s\n' '{"name":"test-item"}' \
    | POST -s -c application/json https://example.test/items

Do not build JSON or form data by concatenating untrusted shell variables without understanding quoting and encoding. For a value containing spaces, ampersands or quotes, use a tool that performs the API's required encoding, or construct the body with a language library.

6. Control Timeouts, Proxies and Conditional Requests

The default timeout is three minutes. Set a shorter value for an interactive probe, or append m or h for minutes or hours:

$ GET -t 15s -s https://example.test/health
200 OK

The manual documents seconds as the default unit and the m and h suffixes. Keep a timeout in scripts so a broken endpoint cannot hold a job for the full default period.

LWP reads proxy settings from the environment. Use -p http://proxy.example.test:8080 to select a proxy for this request, or -P to stop loading proxy settings from the environment. Check the route before sending credentials or private data:

$ env | grep -E '^(http|https|no)_proxy=' || true
$ HEAD -P -s https://example.test/health

For cache validation, -i sets If-Modified-Since. Its argument can be a file name, using that file's modification timestamp, or a literal date accepted by HTTP::Date:

$ HEAD -i ./cached-response.txt -s https://example.test/resource
304 Not Modified

A 304 response is useful only when the server supports conditional requests and the local timestamp represents the version you have. It does not download a replacement body.

7. Process HTML Only When the Optional Modules Exist

The -o option can process an HTML response as text, ps, links, html or dump. The manual says HTML-Tree is required, and HTML-Format is additionally required for text and ps. Check availability before putting one of these modes in a script:

$ GET -o links https://example.test/
https://example.test/health
https://example.test/docs

Relative links are expanded to absolute URLs in links mode. If the response is not HTML, this option has no effect. Do not treat link extraction as a security scan: URLs can be untrusted input, and the command does not validate where following them would lead.

Common traps

• Do not escalate privileges to debug. Use the verbose request options, a shorter timeout and a known-good URL to separate local configuration from server behaviour. Check the URL spelling, proxy environment, DNS and TLS diagnostics before changing anything.
• Keep credentials off the command line. -C username:password supplies Basic Authentication credentials, but putting a password directly on the command line can expose it through shell history or process inspection. Prefer the program's prompt when the server requests Basic Authentication, and avoid credentials entirely in copied examples.
• Do not point it at sensitive local files. The command can also read from the local file system, useful for a controlled read-only check, but not a reason to test it against files that matter.
• Do not retry POST on a failure. A non-zero exit status or an error on standard error is evidence to investigate, not a reason to retry a state-changing POST repeatedly.

Done means

• Version confirmed. You checked the installed libwww-perl version and the alias-to-method defaults.
• GET understood. You can inspect a GET response and distinguish HTTP status from process exit status.
• HEAD and diagnostics used. You can use HEAD for headers and -S or -E for redirect diagnostics.
• POST bodies handled. You know that POST, PUT and PATCH read their bodies from standard input.
• Timeouts and proxies checked. You set explicit timeouts and checked proxy behaviour before troubleshooting a route.
• POST tested safely. You tested POST only against an authorised, reversible or disposable endpoint.