Trace Perl's Regex Plugin Interface Before You Write C

You will finish with a local map of Perl's regular expression plugin interface, the header that defines the ABI on this machine, and a short pre-flight check for a C extension. This is a developer interface, not a command that changes Perl's normal regular expression engine. Allow about 20 minutes for the inspection. You need a shell, Perl development headers and the perl-doc package.

The examples below describe the installed Perl 5.38.2 package, perl-doc 5.38.2-3.2ubuntu0.6. The interface is version-sensitive. Read the manual and headers belonging to the Perl you will actually load, rather than copying a callback table from another host.

1. Confirm the interpreter and documentation

Start by recording the interpreter that will run the extension. A plugin compiled for one Perl installation is not automatically suitable for another ABI or build configuration.

$ perl -v | sed -n '1,8p'
$ perldoc -l perlreapi
This is perl 5, version 38, subversion 2 (v5.38.2) built for x86_64-linux-gnu-thread-multi
/usr/share/perl/5.38/pod/perlreapi.pod

Your patch level and path may differ. The useful checkpoint is the version, architecture and whether the documentation resolves locally. If perldoc cannot find perlreapi, install the matching documentation package through your normal package-management process; do not substitute an unverified web copy for an ABI decision.

2. Locate the public C header

Ask Perl's configuration where its core headers live. The CORE directory is part of this installation, so use its path when inspecting or compiling against it.

$ perl -MConfig -e 'print "$Config{archlib}/CORE\n"'
/usr/lib/x86_64-linux-gnu/perl/5.38/CORE
$ perl -MConfig -e 'print "$Config{archlib}/CORE/regexp.h\n"' | xargs test -r \
  && echo regexp.h-readable
regexp.h-readable

The file size and ownership are not part of the API contract, so do not compare the sample line character for character. Check that the file exists and that it is the header selected by the same perl executable you recorded in step 1.

Checkpoint: inspect the dispatch type without editing the installed file:

$ perl -MConfig -e 'print "$Config{archlib}/CORE/regexp.h\n"' \
  | xargs grep -n -m 2 'regexp_engine\|typedef struct regexp'
57:struct regexp_engine;
224:typedef struct regexp_engine {

3. Understand the hand-off into a custom engine

Perl selects a custom engine through the compile-time hint $^H{regcomp}. That value is treated as an integer which resolves to a regexp_engine structure. The engine's comp callback receives the pattern and compilation flags, creates a REGEXP, and must set that object's engine field back to the same dispatch structure before returning.

This is why a plugin is more than a replacement for a single matching function. The Perl core keeps information about captures, offsets, stringification, optimisation and reference counting in the REGEXP object. The engine owns its private data through pprivate and intflags, while Perl owns the rest of the structure. Do not allocate a private layout by guessing from a sample or by treating every field as yours.

Threaded builds add an interpreter context argument wherever the declaration contains pTHX_. This installed interpreter is built with -thread-multi, so preserve the macro declarations and use the Perl headers rather than rewriting the signatures manually.

4. Map callbacks to responsibilities

Read the complete callback declarations in the local perlreapi(1) page and then use this compact map to plan tests. Every callback that your engine advertises has to agree with Perl's expectations for the contexts in which it can be called.

A missing optimisation callback is not a licence to return plausible data. Follow the documented contract for each entry and test the Perl operators that can reach it. The intuit flags and optimisation data are marked subject to change in the local documentation.

5. Preserve flags and object invariants

The comp callback receives a bitfield containing pattern modifiers such as /m, /s, /i and /x, plus locale information. In the usual case, preserve the applicable flags in rx->extflags. Some flags affect Perl after compilation, so implementing only the parser-facing part can produce a match that looks correct but breaks split, substitutions or match variables.

Two compatibility traps deserve an explicit note. RXf_SPLIT and RXf_SKIPWHITE were removed as active behaviour in Perl 5.18.0 even though they remain defined for source compatibility. Code that sets them should not claim that they still enable the old optimisation. The local manual also marks the character-set interface as experimental and subject to change.

When comp creates a REGEXP, initialise its reference count to 1. Keep the engine pointer valid for the object's lifetime. Treat seen_evals as part of Perl's security checks when patterns are embedded in larger patterns, and do not discard capture or stringification fields merely because a simple match does not use them.

6. Build a compatibility test matrix

Before loading a plugin into a service or long-running application, test the exact Perl executable, compiler flags and thread model used in deployment. At minimum, exercise a literal match, a failed match, captures, named captures, qr// stringification, split, substitution and interpreter cloning if the build uses threads. Include UTF-8 and locale-sensitive cases when the engine claims to support them.

Keep the first run in a disposable process. A faulty callback is native code: it can crash Perl, corrupt memory or return incorrect security-sensitive matches. Do not load an unreviewed shared object into a privileged service. If the test process crashes, remove the plugin from the test command, keep the source and core dump if your debugging policy permits, and return to the smallest failing callback. There is no persistent configuration change in the commands in this guide, so recovery is simply to stop loading the test library.

For API questions, restrict yourself to documented public interfaces. The current Perl API reference warns that items not listed in the public documentation may change without notice. That boundary matters more than whether an internal symbol happens to be visible in proto.h.