Rejoin a Diverged PostgreSQL Cluster with pg_rewind

You will finish with a target PostgreSQL data directory rewound to follow a source cluster after their timelines diverged. The workflow uses pg_rewind from PostgreSQL 16.15, checks the prerequisites first, performs a dry run, then starts the target as a standby only when you have reviewed the copied configuration.

Allow 30 to 60 minutes for preparation and verification, plus the time needed for WAL replay. You need access to the target data directory, a source cluster that is either cleanly stopped or reachable through libpq, and an existing failover or WAL-archive procedure. This is a service-disrupting operation. Keep a fresh backup of the target and a way to recreate it before you begin.

1. Confirm the version and the two clusters

Identify the binary and read its installed version as the account that owns the PostgreSQL installation. These checks are ordinary and do not need sudo:

$ command -v pg_rewind
/usr/lib/postgresql/16/bin/pg_rewind
$ pg_rewind --version
pg_rewind (PostgreSQL) 16.15

The path and patch version can differ. Do not mix a tool from one PostgreSQL major version with a data directory from another major version without following the upgrade procedure. Set two clear variables, and verify that they name the clusters you intend to change:

$ TARGET_PGDATA=/srv/postgresql/16/target
$ SOURCE_PGDATA=/srv/postgresql/16/source
$ test -f "$TARGET_PGDATA/PG_VERSION" && cat "$TARGET_PGDATA/PG_VERSION"
16
$ test -f "$SOURCE_PGDATA/PG_VERSION" && cat "$SOURCE_PGDATA/PG_VERSION"
16

Checkpoint: both commands must identify the expected major version. A path typo here can make a later command fail, or, worse, operate on the wrong directory.

2. Check the rewind prerequisites

pg_rewind needs either data checksums enabled when the cluster was created or wal_log_hints = on in the target cluster. It also needs full_page_writes = on, which is the default but should be confirmed. These settings are not invented at rewind time, so discover them before scheduling failover work:

$ sudo -u postgres psql -d postgres -c \
  "SELECT name, setting, source FROM pg_settings WHERE name IN ('full_page_writes', 'wal_log_hints');"
$ sudo -u postgres pg_controldata "$TARGET_PGDATA" | \
  grep -i 'data page checksum version'

The psql query needs a running cluster and may require an appropriate database role. The pg_controldata check reads the target directory and commonly needs the PostgreSQL owner account. If checksums are disabled and wal_log_hints was not enabled before the divergent writes, stop here and plan a fresh base backup or a separately tested recovery method. Changing the setting now does not make old pages rewindable.

Make sure the target is shut down cleanly before the actual operation. Stop it through its service manager, then confirm that no postmaster is still using the directory. The exact unit name is host-specific:

$ sudo systemctl stop [email protected]
$ sudo systemctl is-active [email protected]
inactive

Do not copy that unit name blindly. Use the unit that owns the target cluster, and check your monitoring so an automatic restart cannot race the rewind.

3. Choose the source access method

Use --source-pgdata when the source data directory is directly accessible and the source server is cleanly stopped. Use --source-server when the source is running and accepting connections. The latter must be a normal, non-replication libpq connection.

For an online source, a dedicated login role can be granted only the functions pg_rewind uses to inspect files. Run these SQL statements as an administrator on the source, and store the password through your normal secret mechanism rather than putting it in shell history:

CREATE USER rewind_user LOGIN;
GRANT EXECUTE ON function pg_catalog.pg_ls_dir(text, boolean, boolean) TO rewind_user;
GRANT EXECUTE ON function pg_catalog.pg_stat_file(text, boolean) TO rewind_user;
GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text) TO rewind_user;
GRANT EXECUTE ON function pg_catalog.pg_read_binary_file(text, bigint, bigint, boolean) TO rewind_user;

Test the connection without changing the target:

$ psql 'host=SOURCE_HOST port=5432 dbname=postgres user=rewind_user' \
  -c 'SELECT current_database(), inet_server_addr();'

If this fails, fix connectivity, authentication or permissions before invoking pg_rewind. Do not solve a missing privilege by casually switching to a superuser in an automated script.

4. Run the dry run

First use --dry-run. It performs the checks and planning but does not modify the target directory. With a stopped source:

$ sudo -u postgres pg_rewind \
  --target-pgdata="$TARGET_PGDATA" \
  --source-pgdata="$SOURCE_PGDATA" \
  --dry-run --progress

For a live source, replace the source option:

$ sudo -u postgres pg_rewind \
  --target-pgdata="$TARGET_PGDATA" \
  --source-server='host=SOURCE_HOST port=5432 dbname=postgres user=rewind_user' \
  --dry-run --progress

The output is diagnostic and progress information, not a portable transcript. The useful result is a successful exit status and no missing-WAL or write-permission error. If old WAL is absent from the target's pg_wal, repeat the dry run with --restore-target-wal, provided the target configuration has a working restore_command and the archive contains the required history.

5. Perform the rewind deliberately

Warning: this is the irreversible state-changing step. pg_rewind overwrites files in the target directory, including relation data, WAL, and configuration files. If it fails while processing, the target may not be recoverable; the PostgreSQL documentation recommends taking a fresh backup in that case.

Keep the target stopped, take or confirm your backup, and run the same command without --dry-run. Add --restore-target-wal only when the archive configuration was tested:

$ sudo -u postgres pg_rewind \
  --target-pgdata="$TARGET_PGDATA" \
  --source-server='host=SOURCE_HOST port=5432 dbname=postgres user=rewind_user' \
  --restore-target-wal --progress
$ printf 'rewind exit status: %s\n' "$?"
rewind exit status: 0

Do not use --no-sync for production. It returns before waiting for file data to reach stable storage, so a subsequent operating system crash can corrupt the directory.

6. Configure and start the target as a standby

A successful rewind is not the end of recovery. WAL replay must complete before the directory is consistent. The simplest controlled setup uses --write-recovery-conf with --source-server:

$ sudo -u postgres pg_rewind \
  --target-pgdata="$TARGET_PGDATA" \
  --source-server='host=SOURCE_HOST port=5432 dbname=postgres user=rewind_user' \
  --write-recovery-conf

This creates standby.signal and appends connection settings to postgresql.auto.conf. Because configuration files are copied from the source during rewind, inspect the target's authentication, TLS, archive and recovery settings before starting it. Restore site-specific values from your reviewed backup if the source's versions are not correct.

$ sudo -u postgres grep -nE '^(primary_conninfo|restore_command)' \
  "$TARGET_PGDATA/postgresql.auto.conf"
$ sudo systemctl start [email protected]
$ sudo -u postgres psql -d postgres -c \
  "SELECT pg_is_in_recovery();"
 pg_is_in_recovery
-------------------
 t

Use the real service unit and connection details for your host. If the target should not become a standby, do not use --write-recovery-conf. If you used it accidentally, stop the target, preserve a copy of the generated files, remove or restore standby.signal and the appended settings according to your change record, then review the configuration before any restart.

7. Diagnose the likely failures

A missing WAL segment means the target no longer retains the history back to the divergence point. Use the WAL archive with --restore-target-wal, or abandon this attempt and build a fresh standby from a new base backup. Do not invent a missing WAL file or copy an unrelated segment into pg_wal.

A write-permission failure can also come from read-only SSL keys or certificates represented by file mappings. The tool requires files to be writable directly. Resolve those links using your deployment procedure before retrying, then restore the intended links after checking what the source copied.

After startup, check that the target is still in recovery and that replay is progressing. A non-zero result from the rewind command, a target that will not start, or a target that becomes a writable primary unexpectedly is a stop condition. Keep the old backup and service rollback path until replication has been tested.