PostgreSQL Backup & WAL Strategy

My production PostgreSQL 13 cluster moves thousands of transactions every second and can’t afford more than a few minutes of downtime. Day-to-day operations are solid, yet I want iron-clad confidence that any disaster—hardware failure, accidental DROP, or site outage—can be reversed quickly. Speed of recovery is the single most important goal.

Current state
• Full and differential backups run with pgBackRest, alongside continuous archiving for point-in-time recovery.
• Backups land in a dedicated repo in DC-A, then the entire pgBackRest repository is replicated to DC-B with rsync.
• I have observed lag in rsync that occasionally leaves standby servers waiting for the latest WAL segments.

What I need
1. A reviewed and improved backup/recovery design that guarantees the shortest possible RTO/RPO for a high-volume workload.
2. A concrete, tested method for cross-DC WAL shipping that is demonstrably faster and more reliable than my current rsync pull—whether that is pgBackRest stanza-sync, native asynchronous replication tweaks, SSH streaming, or another approach you recommend.
3. Documented runbooks for common failure scenarios so my team can execute restores without you on call.
4. Proof of concept: a simulated “fat-finger” DROP on staging, followed by full recovery, timed and documented.

Acceptance criteria
• End-to-end restore of a 500 GB dataset completes within the recovery window we agree on.
• No WAL gap exceeds the archive_timeout you prescribe, verified under sustained write load.
• All procedures are repeatable via shell scripts or Ansible playbooks checked into Git.

Tools in play today: PostgreSQL 13.x, pgBackRest, rsync, Debian 11. I’m open to additional utilities (barman, wal-g, pg_receivewal, etc.) if they meet the above targets.

If this sounds like your sweet spot, let’s talk through your proposed architecture and testing plan so we can lock in a safer, faster path to recovery.