Test wsrep_sst_backup Safely in a MariaDB Galera Cluster

wsrep_sst_backup looks like a backup tool by name, but it is really a handshake script that Galera invokes on your behalf. You will learn what it actually does, verify its donor-side handshake in an isolated data directory, and recognise when it must be left to MariaDB rather than run by hand.

Allow about 15 minutes for the disposable test. A production incident needs more care, because an SST can affect cluster availability and data synchronisation.

Before you start

You need a shell on a MariaDB server, the mariadb-server package, and permission to read the installed script. The machine used for this guide has MariaDB server 1:10.11.14-0ubuntu0.24.04.1. Its manpage is labelled MariaDB 10.3 and says only that this is a backup helper for MariaDB Galera Cluster, so treat the installed script as the authority for the version-specific details below.

In a real cluster, the Galera provider starts the SST script and supplies its arguments. Do not invoke it against a live datadir just to see what it prints. The script creates a process marker, waits for other SST state, removes and creates handshake files, and can tell the donor it may continue.

Checkpoint: Confirm which copy you are about to inspect:

$ command -v wsrep_sst_backup
/usr/bin/wsrep_sst_backup
$ dpkg-query -W -f='${Package} ${Version}\n' mariadb-server
mariadb-server 1:10.11.14-0ubuntu0.24.04.1

1. Understand the donor handshake

The script loads wsrep_sst_common and accepts the arguments that common SST setup parses. Its important inputs here are --datadir, --address, --role, --bypass, --gtid, and --gtid-domain-id. These are protocol inputs, not a friendly interactive interface.

On the normal donor path, the helper prints flush tables and waits for a file named tables_flushed in the supplied data directory. That file must contain a colon, representing the state value supplied by the surrounding server workflow. It then prints continue, writes the state to backup_sst_complete, and prints done followed by the state.

With --bypass, the helper skips the flush wait and builds the state from the supplied GTID and GTID domain ID. The common script labels this transfer as IST rather than SST, a shortcut that only makes sense when the caller has deliberately selected the bypass protocol.

2. Run a disposable bypass test

Use a new directory under /tmp. The example has no database files and cannot touch a MariaDB datadir. Keep the placeholder values visibly fake, and do not reuse this command with a production path.

$ test_dir=$(mktemp -d /tmp/wsrep-sst-backup-test.XXXXXX)
$ /usr/bin/wsrep_sst_backup \
    --datadir "$test_dir" \
    --address 127.0.0.1:4444/backup/0/1 \
    --role donor \
    --bypass \
    --gtid 123 \
    --gtid-domain-id 7
continue
done 123 7

The command should return status 0. Its informational lines go to standard error, including messages that the backup method started and completed on the donor. The two protocol lines above go to standard output. The address is parsed to provide the host, port and SST method path; this test does not contact a peer.

Checkpoint: Verify the marker rather than trusting the terminal text:

$ printf '%s\n' "$?"
0
$ od -An -c "$test_dir/backup_sst_complete"
   1   2   3       7  \n

The marker contains the state followed by a newline. The helper does not make a logical backup copy of tables in this test, and it does not replace a proper mariadb-backup workflow.

3. Exercise the normal donor wait

This test demonstrates the other branch without a live server. Start the helper in the background, then provide the exact file the installed script waits for. The value 456:789 is only test data.

$ test_dir=$(mktemp -d /tmp/wsrep-sst-backup-normal.XXXXXX)
$ /usr/bin/wsrep_sst_backup \
    --datadir "$test_dir" \
    --address 127.0.0.1:4444/backup/0/1 \
    --role donor >"$test_dir/stdout" 2>"$test_dir/stderr" &
$ helper_pid=$!
$ sleep 1
$ printf '%s\n' '456:789' > "$test_dir/tables_flushed"
$ wait "$helper_pid"
$ cat "$test_dir/stdout"
flush tables
continue
done 456:789

Timing can vary. If the helper has not finished, do not start repeated copies: check the process and the two captured files instead. A successful run removes tables_flushed and leaves backup_sst_complete containing 456:789.

4. Handle failures without making the cluster worse

An invocation without the protocol arguments is not a useful help query. On this installation, wsrep_sst_backup --help reaches the common setup and fails because required SST state is absent; the manpage does not define a standalone help mode. Use the source and the MariaDB configuration instead of guessing flags.

If the role is joiner, this helper logs an unsupported role and exits with status 22. It is the donor-side recovery-state helper in this package, and choosing --role joiner is not a repair action.

If the normal donor path sees an sst_error file before the flush marker is complete, it removes that error marker and exits with status 255. If another SST is still running for long enough, the common setup can exit with status 114. Record the exact standard error, exit status and cluster node before escalating.

Warning: Do not delete protocol files in a live datadir or restart nodes at random. A failed SST can leave a cluster needing a controlled retry or a resynchronisation decision. Use the MariaDB error log and Galera status, then follow your cluster runbook.

If you changed only a disposable directory in this guide, leave it for inspection or remove that specific temporary directory once you have checked it. No elevated privilege is needed for the examples; production recovery may require the service account or sudo depending on your installation.

5. Keep the helper in its proper place

MariaDB documents SST methods as a scriptable interface selected through wsrep_sst_method. The cluster normally supplies the role, state and data directory. Set and test the same method across nodes through the cluster configuration, and plan a maintenance window before changing a live SST method.

The installed helper is short because it only coordinates state with the surrounding SST machinery. It does not stream a backup, authenticate a client, transfer table files, or restore a joiner. Those jobs belong to the selected SST method and the Galera server workflow. Treat a successful helper exit as one completed handshake, not proof that a node is healthy or fully synchronised.

Done means