Keep gawk state between runs with a persistent heap

You will make separate gawk processes retain script-defined variables and functions in a heap file. The examples use GNU Awk 5.2.1 from Ubuntu package gawk version 1:5.2.1-2ubuntu0.1, where persistent memory is provided by pm-gawk. Allow about fifteen minutes, including a small test and a decision about where the heap should live.

This feature is useful for a long-lived associative array or a shared set of AWK functions. It is not a database, a replacement for backups or a way to share arbitrary shell state. The heap contains program data and code, so choose its location and permissions as carefully as you would for a private application data file.

1. Check the installed gawk version

Persistent memory was first released in gawk 5.2. Check the binary before building a workflow around it:

$ gawk --version | head -n 1
GNU Awk 5.2.1, API 3.2, PMA Avon 8-g1, (GNU MPFR 4.2.1, GNU MP 6.3.0)
$ command -v gawk
/usr/bin/gawk

The exact version and feature details can differ on another machine. If the command reports an older gawk, do not assume that GAWK_PERSIST_FILE will work.

2. Create a new zero-filled heap

Choose a directory that is local to the machine and inaccessible to untrusted users. The backing file must be initially empty in the pm-gawk sense: its bytes must all be zero. The manual uses a one-gigabyte example; this guide uses a one-megabyte heap for a small demonstration.

$ mkdir -p "$HOME/.local/state/my-gawk"
$ chmod 700 "$HOME/.local/state/my-gawk"
$ truncate -s 1M "$HOME/.local/state/my-gawk/heap.pma"
$ stat -c 'size=%s bytes mode=%a' "$HOME/.local/state/my-gawk/heap.pma"
size=1048576 bytes mode=644

The directory is private, but the file created by truncate may still be readable by other users, as the output shows. Tighten it before putting sensitive values in the heap:

$ chmod 600 "$HOME/.local/state/my-gawk/heap.pma"
$ stat -c 'size=%s bytes mode=%a' "$HOME/.local/state/my-gawk/heap.pma"
size=1048576 bytes mode=600

Checkpoint: the size should be the value you chose and the mode should be private enough for the data. Do not point GAWK_PERSIST_FILE at an existing non-zero file and do not place the heap on a GNU/Linux CIFS filesystem. The installed manual warns that CIFS causes problems for the persistent memory allocator.

3. Run gawk with persistence enabled

Set GAWK_PERSIST_FILE for one command while testing. The variable is an environment setting, not an AWK option:

$ heap="$HOME/.local/state/my-gawk/heap.pma"
$ GAWK_PERSIST_FILE="$heap" gawk 'BEGIN { print ++i }'
1
$ GAWK_PERSIST_FILE="$heap" gawk 'BEGIN { print ++i }'
2

Both commands start a new process, but i is script-defined state and is recovered from the heap. Repeat the command once more if you want a simple checkpoint:

$ GAWK_PERSIST_FILE="$heap" gawk 'BEGIN { print ++i }'
3

Using the variable on each command makes the boundary visible and prevents an unrelated gawk invocation from accidentally sharing this state.

4. Persist a function as well as a variable

Define a function during one run, then call it from a later run. This verifies the part of pm-gawk that ordinary file output cannot provide:

$ GAWK_PERSIST_FILE="$heap" gawk 'function bump(x) { return x + 1 } BEGIN { print bump(41) }'
42
$ GAWK_PERSIST_FILE="$heap" gawk 'BEGIN { print bump(41) }'
42

The second command has no function definition in its AWK program. Its successful result shows that the definition came from the persistent heap. Keep the function names and variable names deliberate: a later script can observe old definitions even when its source looks self-contained.

5. Choose ambient or per-command configuration

For a short session, exporting the environment variable reduces repetition:

$ export GAWK_PERSIST_FILE="$HOME/.local/state/my-gawk/heap.pma"
$ gawk 'BEGIN { print ++i }'
4
$ gawk 'BEGIN { print ++i }'
5

Unset it when you need traditional, non-persistent gawk behaviour:

$ unset GAWK_PERSIST_FILE
$ gawk 'BEGIN { print ++i }'
1
$ gawk 'BEGIN { print ++i }'
1

Checkpoint: if an ordinary command unexpectedly prints a value carried over from an earlier run, inspect env | grep '^GAWK_PERSIST_FILE='. A shell export affects every child command until you unset it or close the shell.

6. Reset the heap deliberately

Removing the heap permanently forgets its variables and functions. This is destructive and cannot be undone unless you have a copy:

$ cp --preserve=mode "$HOME/.local/state/my-gawk/heap.pma" /path/to/a/backup/heap.pma
$ rm -- "$HOME/.local/state/my-gawk/heap.pma"

Only run the removal after checking the path. If you need a clean heap, create a new zero-filled file with the same chosen size and restore its permissions:

$ truncate -s 1M "$HOME/.local/state/my-gawk/heap.pma"
$ chmod 600 "$HOME/.local/state/my-gawk/heap.pma"
$ GAWK_PERSIST_FILE="$HOME/.local/state/my-gawk/heap.pma" gawk 'BEGIN { print ++i }'
1

Do not use sudo for this workflow unless the selected directory genuinely requires elevated access. Running the AWK program as root would give it more access to input files than the feature requires.