High availability and disaster recovery concepts

Resilience conversations go wrong early when the terms are used loosely. This page fixes the vocabulary that the strategy and architecture pages depend on.

Business continuity and disaster recovery

Business continuity (BC) is the discipline of minimising disruption to business activity when something unexpected happens. It is preventative and it is broader than technology: it defines which business processes must keep running, at what level of service, and what the organisation will do about the ones that cannot.

Disaster recovery (DR) is the act of responding to an event that threatens continuity. It is reactive, it is largely technical, and it exists to deliver the objectives the continuity plan set.

The relationship runs one way: the business continuity plan defines the recovery objectives; the recovery objectives justify the investment in high availability and define what disaster recovery has to achieve; high availability then mitigates the events that would otherwise require disaster recovery at all.

High availability and fault tolerance

High availability means designing in enough redundancy that the chance of a failure affecting service levels is small. It accepts that some disruption may occur and optimises the trade-off between reliability and cost.

Fault tolerance means designing in the ability to absorb failures with no user-visible disruption at all. It targets zero downtime, and it is materially more expensive because absorbing a failure without a gap requires standing capacity you are not using.

Most systems should be highly available. A small number of components — usually the ones whose failure would be catastrophic rather than merely expensive — warrant fault tolerance.

Service level agreements

An SLA is a commitment about quality or availability, usually with a service credit attached when it is missed. Two things about SLAs are consistently misread:

• An SLA is not a guarantee. It is a statement of intent plus a refund policy, and the refund is almost never proportionate to the cost of the outage to you.
• Claiming against one is your job. Most providers require the customer to detect, document and submit the claim.

Compose SLAs carefully when you design: a chain of dependencies is no more available than its weakest link, and serial dependencies multiply.

Recovery objectives

Recovery Time Objective (RTO) is the time allowed between the disruption and the business process being usable again. It sets how much automation and standing capacity the recovery approach needs.

Recovery Point Objective (RPO) is the amount of data loss that is acceptable, measured in time — effectively the gap between the last good recovery point and the moment of failure. It sets the backup or replication frequency.

Both are stated by the business, per workload, and then designed to. Deriving them backwards from whatever the current architecture happens to achieve is the most common failure of this process.

What “disaster” actually covers

Real incidents are rarely the flood or fire that the word suggests. The categories worth designing against:

Infrastructure failures. Hardware — a network device, a storage array, a power feed. Software — a bad deployment, a configuration mistake, an application crash.

Load-related events. A denial-of-service attack, an unplanned traffic spike, or a downstream system exhausting a shared resource.

External infrastructure events. A cut fibre run, a facility power failure, a provider’s own outage.

Data-induced failures. Corruption, an unhandled type conversion, or a batch of malformed input that propagates before anyone notices. These are the ones replication cheerfully copies to your standby.

Credential and certificate expiry. An expired TLS certificate, a rotated key that was not rotated everywhere, an authentication dependency that stops answering. Reliably one of the top causes of self-inflicted outages.

Resource and identifier exhaustion. No capacity available for an instance type in the Availability Zone you asked for; a subnet out of addresses; a service quota reached in the middle of a scaling event.

The 2017 S3 outage as a worked example

On 28 February 2017 an operator error during maintenance took a large part of Amazon S3’s capacity in us-east-1 offline for roughly four hours. Three things about it are worth carrying into a design review:

• A single human action bypassed the safeguards. The system did what it was told, very quickly, at a scale the change-management process had not anticipated.
• Dependencies turned one failure into many. Services that used S3 failed with it, including — famously — AWS’s own status dashboard, which could not report the outage because it depended on the thing that was down.
• A single-Region design has a single-Region availability ceiling. No amount of multi-AZ redundancy inside us-east-1 helped, because the impairment was not zonal.

Werner Vogels’ formulation is the right default posture: everything fails, all the time. Assume failure, design so that failures are absorbed rather than propagated, and rehearse recovery rather than documenting it.

Practical implications

• Deploy across Availability Zones for anything with an availability requirement, with health checks and automatic failover rather than manual intervention.
• Cross-Region for anything whose RTO or RPO cannot survive a Regional impairment — replicated data, routed traffic, and a tested promotion path.
• Manage dependencies explicitly. Decouple where you can, apply circuit breakers and timeouts where you cannot, and define what the system does when a dependency is absent rather than letting it hang.
• Test. Run drills, simulate failures, verify recoveries actually restore working service rather than a green health check, and keep the documentation and the people current.