# Designing for Failure — Timeouts, Retries with Backoff and Jitter, Circuit Breakers and Bulkheads

Source: https://skills101.in/learn/system-design/system-design-reliability-patterns — Skills101 free tutorial (System Design). Updated 2026-11-21.

A food delivery app calls a restaurant service, a pricing service, a payment gateway, a maps API and a notification service. If each is up 99.9% of the time, the chance that **all five** are up at a given moment is 0.999⁵ ≈ 99.5%. With dozens of dependencies, **something is always slightly broken**. Good systems are not the ones where nothing fails; they are the ones where a failure stays **small**.

This lesson covers the patterns that keep failures small. Interviewers ask "what happens when service X is slow or down?" in almost every design round, and these are the answers.

## 1. The real danger: slow, not dead

A service that is **down** fails fast: the connection is refused in milliseconds, and the caller moves on. A service that is **slow** is worse. Each request to it holds a thread, a database connection or memory while it waits. If the payment gateway starts taking 30 seconds instead of 300 ms, the order service's threads fill up waiting for it, then the order service stops answering *its* callers, and the failure spreads backwards through the system. This is a **cascading failure**.

Every pattern below either stops the waiting (timeouts, circuit breakers), limits how much can wait (bulkheads), or avoids making things worse (backoff, jitter).

## 2. Timeouts on every network call

A **timeout** is the longest you will wait for a reply before giving up. Many HTTP and database clients wait a very long time — or forever — by default.

- Set a timeout on **every** remote call: HTTP, database, cache, queue.
- Base it on the dependency's normal latency: if the p99 (99th percentile) is 400 ms, a timeout of about 1 second is reasonable; 60 seconds is not.
- Remember the **user's** budget. If the whole page must respond in 2 seconds, three sequential calls cannot each have a 2-second timeout.

## 3. Retries with exponential backoff and jitter

Many failures are **transient**: a dropped packet, a server restarting, a brief overload. Trying again often works. But retries done badly are dangerous:

- **Retry only safe things:** `GET`s, and writes protected by an idempotency key (see the API design lesson). Never blindly retry "charge ₹499".
- **Retry only retryable errors:** timeouts, 503 and 429 — not 400 or 404, which will fail the same way every time.
- **Limit the attempts:** usually 2–4.

**Exponential backoff** means waiting longer after each failure: 0.5 s, 1 s, 2 s, 4 s… This gives an overloaded service room to recover instead of hitting it again immediately.

But if 1,000 phones all failed at the same moment, they all retry at 0.5 s, then all at 1.5 s, then all at 3.5 s — synchronised waves that knock the service down again. This is the **thundering herd**. The fix is **jitter**: wait a *random* time between 0 and the backoff value, so retries spread out.

```python
import random
from collections import Counter

random.seed(7)
clients = 1000

def busiest_100ms(jitter):
    buckets = Counter()
    for c in range(clients):
        t = 0.0
        for attempt in range(4):
            cap = min(8.0, 0.5 * 2 ** attempt)            # 0.5, 1, 2, 4 seconds
            t += random.uniform(0, cap) if jitter else cap
            buckets[int(t * 10)] += 1                     # count retries per 100 ms slot
    return max(buckets.values())

print("no jitter:   busiest 100 ms gets", busiest_100ms(False), "retries")
print("full jitter: busiest 100 ms gets", busiest_100ms(True), "retries")
```

```text
no jitter:   busiest 100 ms gets 1000 retries
full jitter: busiest 100 ms gets 291 retries
```

![Retry times of 1,000 clients: without jitter all 1,000 retries land in the same 100 ms slots at 0.5, 1.5, 3.5 and 7.5 seconds; with full jitter they are spread across the whole period](https://skills101.in/learn-img/system-design/retry-jitter.svg)

Without jitter, every retry wave hits the service with all 1,000 requests in the same 100 ms. With jitter, the peak drops to a fraction of that. Most cloud SDKs (for example the AWS SDKs) use backoff with jitter by default for this reason.

**Retry amplification:** if a page calls service A, which calls B, which calls C, and every layer makes up to 3 attempts, one failing call to C can turn into 3 × 3 × 3 = 27 attempts. Retry at **one** layer, usually the one closest to the user, or use a retry budget.

## 4. Circuit breakers

If the payment gateway has failed the last 20 calls, the 21st will almost certainly fail too. Waiting for its timeout wastes time and adds load to a service that is already struggling. A **circuit breaker** — named after the electrical switch — notices repeated failures and **stops calling** for a while.

![Circuit breaker states: Closed lets calls through and counts failures; after 3 failures it trips to Open, which fails calls instantly; after a cool-down it becomes Half-open and lets one trial call through, going back to Closed on success or Open on failure](https://skills101.in/learn-img/system-design/circuit-breaker.svg)

```python
class CircuitBreaker:
    def __init__(self, threshold=3, cooldown=10):
        self.threshold, self.cooldown = threshold, cooldown
        self.state, self.failures, self.opened_at = "CLOSED", 0, None

    def call(self, now, service_ok):
        if self.state == "OPEN":
            if now - self.opened_at < self.cooldown:
                return "fail fast (not called)"
            self.state = "HALF_OPEN"                     # cool-down over: allow one trial
        if service_ok:
            self.state, self.failures = "CLOSED", 0
            return "success"
        self.failures += 1
        if self.state == "HALF_OPEN" or self.failures >= self.threshold:
            self.state, self.opened_at = "OPEN", now
        return "error"

cb = CircuitBreaker()
calls = [(0, False), (1, False), (2, False), (3, True), (8, True), (12, False), (13, True), (24, True)]
for now, ok in calls:
    result = cb.call(now, ok)
    print(f"t={now:<2} service {'up  ' if ok else 'down'} -> {result:<22} state={cb.state}")
```

```text
t=0  service down -> error                  state=CLOSED
t=1  service down -> error                  state=CLOSED
t=2  service down -> error                  state=OPEN
t=3  service up   -> fail fast (not called) state=OPEN
t=8  service up   -> fail fast (not called) state=OPEN
t=12 service down -> error                  state=OPEN
t=13 service up   -> fail fast (not called) state=OPEN
t=24 service up   -> success                state=CLOSED
```

Follow the trace:

- Three failures in a row trip the breaker to **OPEN** at t=2.
- At t=3 and t=8 the service is actually back, but the breaker does not call it — it fails fast. That is the price of protection.
- At t=12 the cool-down is over; the breaker goes **HALF_OPEN** and lets one trial call through. It fails, so the breaker opens again.
- At t=13 calls are blocked; at t=24 the trial succeeds and the breaker **closes**.

When the breaker is open, the caller should use a **fallback** instead of an error page where possible.

## 5. Bulkheads

Ships are divided into watertight compartments called **bulkheads**, so one hole does not sink the whole ship. In software, give each dependency its **own limited pool** of threads or connections.

If the order service has 200 threads and lets any dependency use all of them, a slow maps API can consume all 200, and orders stop. With bulkheads — say 20 threads for the maps API, 50 for payments, the rest for everything else — a slow maps API can exhaust only its 20. Orders keep working; only the "estimated delivery time" feature degrades.

## 6. Graceful degradation and fallbacks

Decide in advance what the product does when a dependency is unavailable:

| Dependency down | Fallback |
|---|---|
| recommendations service | show "popular courses" from a cache |
| review service | show the course page without reviews |
| exact delivery-time estimate | show "30–40 min" based on the city average |
| payment gateway | let the user try another method; never pretend the payment succeeded |

A page that loads with one section missing is far better than an error page. **Core flows** (login, checkout) deserve the most protection; nice-to-have features can switch off.

## Common mistakes

- No timeout, or a default of minutes, on calls to other services.
- Retrying immediately, without backoff or jitter, and retrying non-idempotent writes.
- Retrying at every layer, multiplying load during an outage.
- Treating "slow" the same as "up" — a 30-second response is a failure.
- No plan for what the user sees when a dependency is down.

## Quick recap

- In big systems something is always failing; slow dependencies cause cascading failures.
- Put timeouts on every remote call, sized from normal latency.
- Retry only idempotent, transient failures, with exponential backoff, jitter and few attempts.
- Circuit breakers fail fast after repeated errors and test recovery in half-open state.
- Bulkheads isolate each dependency's resources; fallbacks keep the core product working.


Practice quiz and free certificate: https://skills101.in/learn/system-design/system-design-reliability-patterns