Disaster Recovery
- Pradeep P
- 3 days ago
- 3 min read
Layer 5 · Post 12 of 12
← Previous: Active-Active vs Active-Passive → Next: Designing a Notification System
Layer 5 — Modern infrastructure · Post 62 of 88
Disaster recovery is the plan for losing a datacenter, a region, or a whole cloud account — and still bringing the business back within an agreed time.
What you'll learn
RTO and RPO in one sentence each — and why they drive cost
Backups vs high availability vs DR, which people collapse into one word
Why an untested runbook is not a recovery plan
The idea in one minute
Disaster recovery (DR) is what you do when the failure is bigger than a node: region down, ransomware, bad deploy that wiped data, account locked.
Two numbers define the contract:
RTO (Recovery Time Objective): how long until the service is usable again.
RPO (Recovery Point Objective): how much data you may lose, measured in time (e.g. 5 minutes of writes).
-- writes --> [ primary ] --async replica / backup--> X boom | RPO = data since last copy v [ recover ] ----RTO----> users back
HA (multi-AZ, Kubernetes restarts) handles small failures automatically. DR is the plan, people, and data copies for the big ones.
Why it matters
A multi-region diagram without RPO/RTO is theater. "We replicate" could mean 1 second lag or a nightly dump. The business cares: unpaid orders, lost messages, legal hold.
Interviews: pick numbers, then pick architecture. RPO 0 and RTO 30s is sync replication and active-active (or a very hot standby) and a large bill. RPO 24h and RTO 8h is backups in another region and a restore drill.
How it works
1. Decide the disaster you mean. AZ, region, cloud provider, "human deleted the bucket," ransomware (backups must be immutable / isolated).
2. Copy data off the failure domain. Snapshots, WAL shipping, cross-region replicas, object storage replication. Test that backups restore, not only that jobs are green.
3. Stand up compute. IaC (Terraform, GitOps) in the surviving region. Pilot light vs warm vs hot matches the previous post. Secrets and DNS must already exist there, or RTO includes "we forgot KMS."
4. Fail over traffic. DNS, Anycast, or a global LB. Communicate with clients whose TTLs and connection pools ignore your runbook.
5. Fail back (optional). Returning home is a second migration. Some teams stay on B.
Backups are not HA
A backup in the same region as the database dies with the region. A replica you have never promoted is a rumor. Point-in-time recovery fixes "dropped a table," not "region gone," unless copies live elsewhere.
Chaos / game days: pull the plug on a staging region, time the clock. DR that only exists in Confluence fails the first incident.
A simple example
An ecommerce shop: RPO 15 minutes, RTO 2 hours. WAL shipped to another region's bucket every minute. DR runbook: restore Postgres in region B (~40 min in the last drill), apply GitOps, flip Route 53. They do not claim RPO 0. They drill quarterly.
A payments API: RPO seconds, RTO minutes. Warm replica, rehearsed promotion, health-checked global steering. They pay for idle capacity. That is the trade.
Common mistakes
Backups never restored. The first restore is during the outage. It fails on encryption keys, version skew, or size.
RTO that ignores DNS, TLS, and partner allowlists. The DB came up; the world still talks to the dead VIP.
One person knows the steps. DR is an operations design, not a hero.
Same-account, same-region snapshots as the only copy. Account compromise or regional S3 issues take the backups too. Offsite / other account / immutability for the ransomware case.
Treating Kubernetes as DR. It reschedules Pods. It does not resurrect a disk you did not replicate.
How this shows up in real systems
Pilot light / warm / hot standby (AWS Well-Architected language): cost vs RTO.
Backup vaults, PITR, cross-region Aurora/GCS/Blob replication.
Tabletop exercises and game days: the only evidence the plan is real.
Layer 5 ends here: you can run, see, scale, and recover. Layer 6 applies this to concrete products.
Recap
DR is RTO + RPO + a tested path off the failure domain, not a snapshot job alone.
HA ≠ DR. Restarts and multi-AZ do not replace offsite copies and failover practice.
Pick numbers first; the active-passive vs active-active choice follows.
Next: a concrete system — notifications — built from these pieces.
Layer 5 · Post 12 of 12
← Previous: Active-Active vs Active-Passive → Next: Designing a Notification System
Comments