Inspect HTTP Headers and Response Content with lwp-dump

You will use lwp-dump to make a request, see the response headers and inspect the beginning of the returned content. The examples use lwp-dump from libwww-perl 6.76-1ubuntu0.1, installed here. Allow about ten minutes. You need a shell and a URL that you are authorised to request.

The route

Jump straight to the step you need, or tick off Done means at the end.

1. Confirm the installed command
2. Make a bounded GET request
3. Remove the content limit only for a deliberate reason
4. Show the request as well as the response
5. Change the user agent without changing the URL
6. Choose a method carefully
7. Handle HTML parsing and common failures

This is a diagnostic tool, not a browser. It prints a response object to standard output. Content is escaped using Perl double-quoted string syntax, which keeps binary bytes safe to display in a terminal.

1. Confirm the installed command

Check the binary and package version before relying on option details:

$ command -v lwp-dump
/usr/bin/lwp-dump
$ dpkg-query -W -f='${Package} ${Version}\n' libwww-perl
libwww-perl 6.76-1ubuntu0.1

The command accepts one URL after its options. It does not have a --help option on this installation. Running an unknown option prints usage and exits with an error, so use the installed manpage as the option reference.

Checkpoint: the version in your output should match the behaviour described here. Distribution updates can change the package version and the exact diagnostic wording.

2. Make a bounded GET request

Start with a harmless public endpoint or a URL you control. The default method is GET, and the default content limit is 512 bytes:

$ lwp-dump --max-length 80 https://example.com
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: text/html

<!doctype html><html lang="en">...
(+ 479 more bytes not shown)

The status line and headers vary with the endpoint, cache and server. The useful checks are that a response was received, the status is what you expected, and the content is clearly bounded. A successful network request is not the same as an HTTP success: a server can return 404 or 500 while the command still prints the response.

When the content is longer than the limit, lwp-dump appends a notice showing how many bytes were not displayed. It does not silently claim that the displayed prefix is the whole document.

3. Remove the content limit only for a deliberate reason

Use --max-length 0 for unlimited content:

$ lwp-dump --max-length 0 https://example.com

This can produce a large amount of terminal output and may be a poor choice for an unknown URL. Redirect it to a file when you genuinely need the complete escaped response:

$ lwp-dump --max-length 0 https://example.com > response-dump.txt
$ test -s response-dump.txt && echo 'response dump is non-empty'
response dump is non-empty

Redirection creates or truncates response-dump.txt. Check the destination name before running it. If an existing dump matters, choose a new name or copy it first. The output is a diagnostic representation, not necessarily the original byte-for-byte response body.

4. Show the request as well as the response

Add --request when you need to compare what the client sent with what the server returned:

$ lwp-dump --request --max-length 80 https://example.com
GET https://example.com
User-Agent: lwp-dump/6.76 libwww-perl/6.76

HTTP/1.1 200 OK
Content-Type: text/html

<!doctype html>...

The request section helps expose the method and user agent. Treat the output as potentially sensitive: URLs can contain query strings, and request or response headers can contain identifiers. Do not paste a dump into a ticket or log without checking it.

lwp-dump normally removes LWP-generated Client-* headers so the response shows headers supplied by the server. Add --keep-client-headers when those generated headers are specifically relevant:

$ lwp-dump --keep-client-headers --max-length 0 https://example.com

5. Change the user agent without changing the URL

Some services vary their response by user agent. Set one explicitly with --agent:

$ lwp-dump --agent 'diagnostic-check/1.0' --max-length 256 https://example.com
HTTP/1.1 200 OK
...

Use a truthful, identifiable value and follow the service's rules. Changing the user agent can change the response, but it does not make a request anonymous or bypass access controls. Keep the limit small while comparing variants.

6. Choose a method carefully

--method replaces the default GET:

$ lwp-dump --method HEAD --request https://example.com
HEAD https://example.com
...
HTTP/1.1 200 OK
Content-Type: text/html

HEAD is useful when you want headers without a response body. The server decides how accurately it implements that method, so treat the result as a server response rather than a guarantee about a later GET.

Warning: do not experiment with POST, PUT, PATCH or DELETE against a live service merely to inspect it. The manpage permits an arbitrary method, and the request can change remote state or trigger an operation. Obtain explicit authorisation, use a test endpoint, and understand the service's authentication and CSRF controls first. lwp-dump does not provide a rollback for a server-side action.

7. Handle HTML parsing and common failures

By default, the tool does not initialise headers by parsing an HTML document's head section. Add --parse-head when you need LWP's parse_head behaviour:

$ lwp-dump --parse-head --max-length 256 https://example.com

Use this only when comparing parsed HTML metadata is part of the diagnosis. It can make the observed header set differ from the default output.

If the URL cannot be opened, first check the spelling and connectivity without changing the remote resource:

$ getent hosts example.com
$ lwp-dump --max-length 80 https://example.com
$ printf 'exit status: %s\n' "$?"

Capture the status immediately. A non-zero status indicates that the command did not complete normally, but the precise message depends on the URL, TLS setup and local LWP configuration. A response with an HTTP error status is still useful output, so inspect both the command status and the HTTP status line.

Done means

You confirmed the installed libwww-perl version and used the manpage's option names.
You inspected response headers and a bounded content prefix without assuming that a response is an HTTP success.
You used --request when the outgoing method or user agent mattered.
You treated unlimited output and redirected dumps as deliberate file and logging decisions.
You used non-GET methods only with authorisation and a recovery plan for any remote state change.