Disaster Recovery & High Availability

Disaster recovery planning fails in a predictable way: a document is written, nobody tests it, and the first real execution discovers that the backup restore takes six hours and the runbook references a tool that was decommissioned.

RPO and RTO#

State these as numbers agreed with the business, not as aspirations. "As little downtime as possible" is not a target — it cannot be designed against or tested. "RTO one hour" tells you whether a manual restore is acceptable.

Replication trade-offs#

Asynchronous — the primary commits, replication follows. No write latency cost, RPO of seconds to minutes depending on lag. The default for most systems, and correct for most.

Synchronous — the transaction does not commit until a standby has it. RPO zero, at the cost of a network round trip on every write. Across availability zones that is single-digit milliseconds; across regions it is tens, which changes your application's performance profile substantially.

Semi-synchronous — wait for at least one standby, not all. A reasonable middle ground: bounded data loss with less latency cost than full synchronous.

Whichever you choose, monitor replication lag as a first-class metric. Lag is your actual RPO, and it is not the configured value — it is whatever the replica is currently behind by. A replica that has silently fallen an hour behind means an hour of data loss on failover, regardless of what the design document says.

Multi-region topologies#

Active-active is the one to be cautious about. Bidirectional data synchronisation means conflict resolution, and conflict resolution means deciding what happens when the same record is written in two regions within the replication window. For a read-heavy workload it is manageable; for anything with contended writes it is a genuinely hard distributed systems problem.

Warm standby is the sweet spot for most systems that need cross-region recovery: a scaled-down copy that is continuously receiving replicated data and can be scaled up in minutes.

Failover#

resource "aws_route53_health_check" "primary" {
  fqdn              = "api-eu.acme.com"
  type              = "HTTPS"
  resource_path     = "/actuator/health/readiness"
  failure_threshold = 3
  request_interval  = 30
}
 
resource "aws_route53_record" "primary" {
  zone_id        = var.zone_id
  name           = "api.acme.com"
  type           = "A"
  set_identifier = "primary"
  failover_routing_policy { type = "PRIMARY" }
  health_check_id = aws_route53_health_check.primary.id
  alias { name = aws_lb.eu.dns_name, zone_id = aws_lb.eu.zone_id, evaluate_target_health = true }
}

DNS failover has a floor set by TTL plus client caching, and clients routinely ignore TTL. A 60-second TTL does not give a 60-second RTO — some clients will keep resolving the old address for far longer. Where the RTO must be tight, a global load balancer with anycast is a better mechanism than DNS.

The health check path matters. Pointing it at / returns healthy while the database is unreachable; pointing it at readiness means the check reflects the service's actual ability to serve.

Testing the plan#

An untested plan is a hypothesis. Three levels of testing, in increasing order of confidence.

Tabletop. Walk through the runbook in a meeting. Cheap, and it reliably finds missing steps, stale tool references and unclear ownership. Do this quarterly.

Restore drill. Actually restore a backup into an isolated environment and time it. This produces your real RTO number, which is frequently several times the assumed one. Do this quarterly and record the timing.

Full failover. Fail over to the standby region for real, serve production traffic from it, and fail back. This is the only test that proves the plan, and the only one that finds the dependency nobody documented — a certificate, a queue, a third-party allowlist that only knows the primary region's addresses. Do this annually, announced, in a low-traffic window.

Chaos engineering#

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata: { name: order-service-chaos }
spec:
  appinfo:
    appns: production
    applabel: 'app=order-service'
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - { name: TOTAL_CHAOS_DURATION, value: '60' }
            - { name: CHAOS_INTERVAL, value: '10' }
            - { name: PODS_AFFECTED_PERC, value: '50' }

The method matters more than the tool. Define the steady state you expect — error rate under 0.1%, p99 under 500ms. State a hypothesis: killing half the pods will not breach it. Run the experiment in a controlled window with a clear abort condition. Then either your hypothesis held, or you have learned something specific and actionable.

Start small and in staging. Kill one pod, not a zone. Once the team is comfortable and the obvious gaps are fixed, run in production during business hours when everyone is awake — which is exactly the opposite of when a real failure will happen, and precisely the point.

What to take away#

Agree RPO and RTO as numbers with the business, then buy the cheapest architecture that meets them — which for most systems is multi-AZ, not multi-region. Monitor replication lag, because it is your real RPO. Test restores quarterly and time them, fail over annually, and use chaos experiments to convert assumptions about resilience into evidence.