Reliability and Fault Tolerance

After this lesson, you will be able to: Reason about reliability with SLAs/SLOs/SLIs, and design fault tolerance with circuit breakers, retries with backoff, graceful degradation, and the chaos-engineering mindset.

Every system fails. The question senior engineers answer is: when a dependency goes down, does your whole product go down with it, or does it degrade gracefully and recover on its own? This lesson covers how teams measure reliability and the patterns that keep a system standing when parts of it fall over.

Prerequisites:API Design

SLI, SLO, SLA: measuring reliability

An SLI (Service Level Indicator) is a metric you actually measure, for example the percentage of requests served in under 200ms, or the percentage that succeed. An SLO (Service Level Objective) is the internal target for an SLI, for example 99.9% of requests succeed each month. An SLA (Service Level Agreement) is the contractual promise to customers, usually looser than the SLO, with penalties if you miss it. The gap between SLO and 100% is your error budget: the amount of failure you are allowed, which teams spend deliberately on risk (shipping faster) rather than chasing an impossible 100%.

Retries with exponential backoff and jitter

Naive retries make outages worse (the thundering herd). Back off and add randomness.

tsx

async function withRetry(fn, maxAttempts = 4) {
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      // Only retry transient failures (5xx, timeouts, network), NOT 4xx.
      if (!isTransient(err) || attempt === maxAttempts - 1) throw err;
      // Exponential backoff: 100ms, 200ms, 400ms... plus random jitter
      // so a thousand clients don't all retry at the same instant.
      const base = 100 * 2 ** attempt;
      const jitter = Math.random() * base;
      await sleep(base + jitter);
    }
  }
}

Circuit breakers and graceful degradation

A circuit breaker wraps a call to a flaky dependency. After too many failures it 'trips' (opens) and fails fast for a cooldown period instead of hammering a service that is already down, then sends a trial request to see if it recovered before closing again. Graceful degradation is the product-level version: when the recommendations service is down, show a static 'popular items' list instead of an error page; when search is slow, serve cached results. The goal is that one failed dependency degrades one feature, not the whole site.

Chaos engineering: break it on purpose

You do not actually know your system tolerates failure until you cause failure. Chaos engineering (pioneered by Netflix's Chaos Monkey) deliberately kills instances, injects latency, and severs dependencies in production-like environments to verify the system degrades gracefully and alerts fire. You start small (kill one non-critical pod in staging), form a hypothesis ('the queue should absorb this'), run the experiment, and fix what breaks. It turns 'we think we are resilient' into evidence.

Common mistakes only experienced engineers catch

Chasing 100% uptime instead of setting an SLO the product needs and spending the error budget. Retrying without backoff or jitter, turning a brief blip into a self-inflicted DDoS (thundering herd). Retrying non-idempotent operations, causing duplicate charges (tie this back to idempotency keys). No circuit breaker on a flaky dependency, so one slow service exhausts all your threads and takes the whole app down (cascading failure). Assuming the system is resilient without ever testing failure. If you have not killed it on purpose, you do not know.

Quick Check

Your payment provider has a brief outage and all clients retry instantly and repeatedly. What two patterns prevent this from making things worse?

Pick the best pairing.

←API Design: REST vs GraphQL vs gRPC

Back to System Design

System Design Interview Job Readiness→