Highlight Source Files Reliably with pygmentize

You will finish with a repeatable way to turn a source file into coloured terminal output or HTML, while knowing which lexer and formatter were selected. The examples use Debian's python3-pygments package, version 2.17.2+dfsg-1, and the installed Pygments command reports version 2.17.2.

Allow about fifteen minutes. You need a shell, a readable input file and the pygmentize command. These examples only read the input and write to standard output or a named output file. No elevated privileges are needed unless you deliberately choose a directory that your account cannot write.

1. Check the installed command

Confirm the executable and version before relying on a formatter or lexer in a script:

$ command -v pygmentize
/usr/bin/pygmentize
$ pygmentize -V
Pygments version 2.17.2, (c) 2006-2023 by Georg Brandl, Matthaus Chajdas and contributors.
$ dpkg-query -W -f='${Package} ${Version}\n' python3-pygments
python3-pygments 2.17.2+dfsg-1

The copyright line can vary slightly between package builds. The useful checkpoint is that the command is the one you expect and the package version is recorded.

2. Highlight a named source file

Create or choose a small file whose extension identifies its language. Pygmentize guesses the lexer from that extension when you omit -l:

$ printf '%s\n' 'def greet(name):' '    return f"Hello, {name}"' > /tmp/example.py
$ pygmentize /tmp/example.py

The output is intended for a terminal, so it normally contains ANSI colour escape sequences. A terminal displays the highlighted source. If you redirect this default output to a file, those control sequences are still present and are usually the wrong format for a web page or a plain text attachment.

Checkpoint: ask the command which lexer it would use without highlighting anything:

$ pygmentize -N /tmp/example.py
python

-N bases its answer on the filename. An unknown extension produces text, so a plausible-looking result is not proof that the source was parsed as the language you intended.

3. Select the lexer and formatter explicitly

Use -l for the lexer and -f for the formatter when the filename is ambiguous or the output is part of a repeatable process:

$ pygmentize -l python -f terminal /tmp/example.py

For an HTML fragment, choose the HTML formatter:

$ pygmentize -l python -f html /tmp/example.py
<div class="highlight"><pre><span></span><span class="k">def</span> ...
</pre></div>

The exact spans depend on the source and Pygments version. The HTML formatter emits markup and CSS class names, but not the stylesheet itself. Generate or provide matching CSS separately if a browser must display the colours.

Use -L to inspect what is installed rather than guessing a name:

$ pygmentize -L lexers
$ pygmentize -L formatters
$ pygmentize -H formatter html

These are read-only queries. They are also a useful recovery step when a copied command fails with an unknown lexer or formatter.

4. Write HTML to a new file safely

Pass -o when another program needs a file. The output filename can also help Pygments choose a formatter, but specifying -f html makes the intention visible:

$ pygmentize -l python -f html -o /tmp/example.html /tmp/example.py
$ file /tmp/example.html
/tmp/example.html: HTML document, ASCII text
$ sed -n '1,4p' /tmp/example.html
<div class="highlight"><pre><span></span>...

Do not point -o at the input file. More generally, shell redirection and -o can overwrite an existing destination. That is a destructive change: choose a new path first, or make a backup before replacing a useful file.

$ cp --preserve=all /path/to/result.html /path/to/result.html.bak
$ pygmentize -l python -f html -o /path/to/result.html.new /path/to/example.py
$ mv /path/to/result.html.new /path/to/result.html

If the conversion fails, the old result remains in place and the .new file can be inspected or removed. Keep the backup until you have checked the replacement; deleting it with rm is irreversible.

5. Handle standard input deliberately

With no input filename, pygmentize reads standard input. This is useful in pipelines, but the extension-based lexer guess is unavailable. Select the lexer yourself:

$ printf '%s\n' 'SELECT id FROM users;' | pygmentize -l sql -f terminal

Do not combine -g with -l. They are alternative lexer choices. Use -g when you want Pygments to attempt content-based guessing, including a shebang or modeline where applicable:

$ printf '%s\n' '#!/usr/bin/env python3' 'print("ok")' | pygmentize -g -f terminal

Content guessing is a fallback, not a correctness check. It can be unreliable, particularly for short or ambiguous input. Use -C to print the guessed lexer name for standard input without producing highlighted output:

$ printf '%s\n' 'SELECT id FROM users;' | pygmentize -C
text

That result is a reminder to use -l sql when the language matters. If you are following an unbounded stream such as tail -f, the installed command also supports -s for line-at-a-time processing, but it only works with standard input and lexers without line-spanning constructs.

6. Set options without losing shell arguments

Use -O for comma-separated lexer or formatter options, and quote the whole value when the shell could split or expand it:

$ pygmentize -l python -f html -O 'linenos=1,style=emacs' /tmp/example.py

Use -P when one option value itself contains commas or equals signs, because it accepts one option per occurrence:

$ pygmentize -l python -f html -P 'cssclass=source-code' /tmp/example.py

Option names belong to the selected lexer or formatter. Check their documented help with -H lexer NAME or -H formatter NAME before putting them into automation. If an option is rejected, remove it and confirm the exact object name with -L.

7. Diagnose the common mistakes

An empty or missing output file usually means the input path, output path or permissions need checking. Read access and destination writability can be tested without changing anything:

$ test -r /path/to/example.py && echo 'input is readable'
$ test -w /path/to/output-directory && echo 'directory is writable'
$ pygmentize -l python -f html -o /tmp/example.html /path/to/example.py
$ test -s /tmp/example.html && echo 'output is non-empty'

A non-zero exit status means the command failed, but a successful status does not prove that the lexer was correct. Check the selected lexer with -N or -C, and inspect the output format before handing it to a browser or another tool. Avoid sudo as a first response: it can hide a path or ownership problem and is unnecessary for files in your working directory.