Build a PerlIO Layer Without Losing the Stream Stack

You will finish with a reliable design for a PerlIO layer: a handle stack you can inspect, a vtable with deliberate fallbacks, and a test boundary that catches broken ownership before the layer reaches real data. This is a C extension task, not a shell command. Allow about an hour for a small pass-through layer, longer if it buffers or transforms bytes.

Before you start

1. Install the development headers for the Perl you will run. This machine has Perl 5.38.2 and the perl-doc package at 5.38.2-3.2ubuntu0.6.
2. Work in a disposable checkout. A faulty layer can lose buffered data, leak memory, or crash the interpreter. Do not load experimental code into a long-running service or a process holding the only copy of important output.
3. Read the API against the headers for the target Perl. The document is source-compatible in broad outline, but its vtable and macros are tied to Perl's build.

Check the local version and header before writing C:

perl -V:version -V:archlib
perl -MConfig -E 'say "$Config{archlib}/CORE/perliol.h"; say((-f "$Config{archlib}/CORE/perliol.h") ? "present" : "missing")'

Expected output includes version='5.38.2' and present. If the header is missing, stop there and install the matching Perl development package rather than borrowing a header from another version.

Checkpoint: see the stack you are changing

A PerlIO stream is a linked stack. The bottom layer deals with the operating system, while layers above it may buffer, translate, or otherwise process the bytes. Perl's application-level PerlIO * is deliberately an extra level of indirection, so the handle table can keep its pointer while the current top layer changes.

Inspect a normal handle on this host:

perl -MPerlIO -E 'say for PerlIO::get_layers(*STDOUT)'

unix
perlio

That output is local evidence, not a universal default. The stack varies with the operating system, Perl compile-time choices and runtime configuration. When an open or binmode call names layers, Perl pushes them above the existing stack, processing the names from left to right. sysopen is lower level: on Unix-like systems it uses the unix layer around the file descriptor.

Design the instance first

Include perliol.h, then make your per-handle structure begin with a struct _PerlIO. The base must be first so a pointer to your structure can be treated as a pointer to the common layer structure. Put only per-instance state after it, such as a byte buffer, a cursor and a transformation state.

#include <perliol.h>

typedef struct {
    struct _PerlIO base;
    unsigned char *buffer;
    Size_t used;
    Size_t cursor;
} PerlIOExample;

The vtable is the class part. It names the layer, gives the size to allocate for each instance, declares attributes such as buffered or raw, and supplies callbacks for setup and I/O. The mandatory callback is Pushed. It should normally call PerlIOBase_pushed(), which converts the mode into the relevant PERLIO_F_ flags, then initialise only state that belongs to your layer.

Choose delegation instead of copying the base layer

A layer does not need to implement every callback, but every slot exists in the vtable. A null callback is not a general pass-through: some null methods inherit from the layer below, some return success, and others fail with EINVAL. The documented defaults make Open inherit, Close use PerlIOBase_close, Read use PerlIOBase_read, and Write, Seek and several buffer accessors fail. Treat that table as part of your design review.

For a transforming layer, start with the smallest useful surface. Implement Read or Write, and delegate positioning, flushing, close and file-descriptor lookup to the base helpers where their semantics still apply. A read callback typically fills its own buffer from the layer below; a write callback must pass bytes down and report the number actually accepted. Both return -1 on error.

If your layer stores an argument such as an encoding name, implement Getarg. The duplication path uses it to recover the argument originally passed to Pushed. Without it, a handle duplicated with & or copied during thread creation may not reproduce the original state.

Keep ownership and removal explicit

Open combines several paths, including ordinary opens, descriptor wrapping, system opens and reopen operations. If you provide it, normally call the next layer's open method first and push your layer only after that succeeds. If you push and then fail, pop your layer yourself. Leaving a failed layer on the stack can poison later I/O.

Popped runs when a layer is removed, sometimes without a preceding close. Free buffers and translation state there. If your layer has read ahead, return unconsumed bytes to the lower layer with Unread before discarding them. Close should flush and close lower layers through PerlIOBase_close(), then release state owned by your layer.

Do not mark a transforming layer PERLIO_K_RAW. That flag says the layer is safe to keep in a :raw stack. A buffer-only layer may be PERLIO_K_BUFFERED; a layer that supports Perl's buffer snooping needs the corresponding fast-gets contract, not merely a flag copied from an example.

Test the boundary before testing performance

Build and load the extension with the same interpreter and headers you checked above. First test a temporary file in read mode, then write mode, then a read/write handle. Verify byte counts, tell, seek, end-of-file and close errors. Repeat with an empty file and a file smaller than one buffer. Finally test a handle with :raw, duplication with open my $copy, '&', $fh, and removal with binmode or the layer API.

For a Perl-side stack check, use a harmless temporary scalar. The bundled scalar layer reads and writes the scalar itself, and the ordinary scalar form is equivalent:

perl -MPerlIO -e '
my $text = "alpha\n";
open my $fh, "+<", \$text or die $!;
print scalar PerlIO::get_layers($fh), "\n";
print scalar <$fh>;
' 

The exact layer list may vary, but the command should open successfully and print a layer description followed by alpha. This checks the interpreter's layer machinery without loading your experimental C code.

Common traps

• Assuming "pushed on top" means "only the top object is affected". A layer can inspect or change layers below it, and :raw actively removes layers that cannot handle binary data.
• Confusing bytes and characters. Most layers should perform I/O in binary mode; the crlf layer handles text line-ending translation, while utf8 and encoding affect interpretation.
• Ignoring close-on-exec. A layer that takes ownership of a descriptor must leave its close-on-exec flag correct. New descriptors should be opened with the flag set when possible.
• Returning success from an unimplemented callback. A null Flush can succeed, but a null Write fails. Confirm each required operation rather than trusting a partial smoke test.