Postgres Logical Replication Failover Runbook Builder

Create a production runbook for Postgres logical replication failover with slot checks, subscriber promotion steps, data validation, and rollback criteria.

Prompt Template

You are a senior database reliability engineer. Build a failover runbook for a Postgres logical replication setup.

Environment: [Postgres version, cloud provider or self-hosted, managed service constraints]
Topology: [publisher, subscribers, publications, subscriptions, replication slots, downstream consumers]
Data scope: [tables, schemas, sequences, large objects, partitioned tables, excluded tables]
Failure scenario: [publisher outage, subscriber lag, slot bloat, schema drift, network partition, failed migration]
RPO/RTO target: [acceptable data loss, recovery time target]
Monitoring available: [pg_stat_subscription, slot lag, WAL disk alerts, application metrics, logs]
Access level: [superuser, managed DB admin, limited role, read-only observer]
Traffic cutover method: [DNS, app config, feature flag, maintenance window, read-only mode]
Validation needs: [row counts, checksums, sequence values, application smoke tests, audit queries]
Rollback constraints: [old primary availability, bidirectional writes risk, backups, restore point]

Produce:
1. Preconditions and assumptions to verify before any failover.
2. Detection checklist and severity classification.
3. Step-by-step failover plan with commands or pseudo-commands.
4. Replication slot, WAL retention, sequence, and schema checks.
5. Data validation queries before and after cutover.
6. Application cutover and communication checklist.
7. Rollback or roll-forward decision tree.
8. Post-incident cleanup tasks and prevention work.

Call out where exact SQL depends on the managed provider or schema. Do not invent credentials or run destructive SQL without an explicit confirmation step.

Example Output

Logical Replication Failover Runbook

Preconditions

- Confirm the subscriber has received changes through the planned cutover timestamp.

- Record current slot names, confirmed_flush_lsn, restart_lsn, and subscriber replay position.

- Freeze writes or put the app in maintenance mode before promotion if dual writes are possible.

Validation Query Set

| Check | Query | Pass Criteria |

|---|---|---|

| Subscription health | SELECT subname, status FROM pg_stat_subscription; | All critical subscriptions healthy or lag understood |

| Table counts | SELECT count(*) FROM critical_table; | Matches publisher snapshot tolerance |

| Sequence ceiling | SELECT last_value FROM order_id_seq; | Greater than max(order.id) |

Cutover Decision

If slot lag is increasing and the publisher is unrecoverable inside RTO, proceed with subscriber promotion and app config cutover. If publisher may return with unreplicated writes, keep old primary isolated until reconciliation is complete.

Tips for Best Results

  • 💡List every publication and subscription; logical replication failures are often table-specific.
  • 💡Include sequence validation because copied rows do not guarantee safe future inserts.
  • 💡Ask for provider constraints, since RDS, Cloud SQL, and self-hosted Postgres expose different controls.