AdvancedSystem Design · Lesson 14 of 19 · Updated ·
Designing for Failure — Timeouts, Retries with Backoff and Jitter, Circuit Breakers and Bulkheads
A food delivery app calls a restaurant service, a pricing service, a payment gateway, a maps API and a notification service. If each is up 99.9% of the time, the chance that all five are up at a given moment is 0.999⁵ ≈ 99.5%. With dozens of dependencies, something is always slightly broken. Good systems are not the ones where nothing fails; they are the ones where a failure stays small.
This lesson covers the patterns that keep failures small. Interviewers ask "what happens when service X is slow or down?" in almost every design round, and these are the answers.
1. The real danger: slow, not dead
A service that is down fails fast: the connection is refused in milliseconds, and the caller moves on. A service that is slow is worse. Each request to it holds a thread, a database connection or memory while it waits. If the payment gateway starts taking 30 seconds instead of 300 ms, the order service's threads fill up waiting for it, then the order service stops answering its callers, and the failure spreads backwards through the system. This is a cascading failure.
Every pattern below either stops the waiting (timeouts, circuit breakers), limits how much can wait (bulkheads), or avoids making things worse (backoff, jitter).
2. Timeouts on every network call
A timeout is the longest you will wait for a reply before giving up. Many HTTP and database clients wait a very long time — or forever — by default.
- Set a timeout on every remote call: HTTP, database, cache, queue.
- Base it on the dependency's normal latency: if the p99 (99th percentile) is 400 ms, a timeout of about 1 second is reasonable; 60 seconds is not.
- Remember the user's budget. If the whole page must respond in 2 seconds, three sequential calls cannot each have a 2-second timeout.
3. Retries with exponential backoff and jitter
Many failures are transient: a dropped packet, a server restarting, a brief overload. Trying again often works. But retries done badly are dangerous:
- Retry only safe things:
GETs, and writes protected by an idempotency key (see the API design lesson). Never blindly retry "charge ₹499". - Retry only retryable errors: timeouts, 503 and 429 — not 400 or 404, which will fail the same way every time.
- Limit the attempts: usually 2–4.
Exponential backoff means waiting longer after each failure: 0.5 s, 1 s, 2 s, 4 s… This gives an overloaded service room to recover instead of hitting it again immediately.
But if 1,000 phones all failed at the same moment, they all retry at 0.5 s, then all at 1.5 s, then all at 3.5 s — synchronised waves that knock the service down again. This is the thundering herd. The fix is jitter: wait a random time between 0 and the backoff value, so retries spread out.
import random
from collections import Counter
random.seed(7)
clients = 1000
def busiest_100ms(jitter):
buckets = Counter()
for c in range(clients):
t = 0.0
for attempt in range(4):
cap = min(8.0, 0.5 * 2 ** attempt) # 0.5, 1, 2, 4 seconds
t += random.uniform(0, cap) if jitter else cap
buckets[int(t * 10)] += 1 # count retries per 100 ms slot
return max(buckets.values())
print("no jitter: busiest 100 ms gets", busiest_100ms(False), "retries")
print("full jitter: busiest 100 ms gets", busiest_100ms(True), "retries")
no jitter: busiest 100 ms gets 1000 retries
full jitter: busiest 100 ms gets 291 retries
Without jitter, every retry wave hits the service with all 1,000 requests in the same 100 ms. With jitter, the peak drops to a fraction of that. Most cloud SDKs (for example the AWS SDKs) use backoff with jitter by default for this reason.
Retry amplification: if a page calls service A, which calls B, which calls C, and every layer makes up to 3 attempts, one failing call to C can turn into 3 × 3 × 3 = 27 attempts. Retry at one layer, usually the one closest to the user, or use a retry budget.
4. Circuit breakers
If the payment gateway has failed the last 20 calls, the 21st will almost certainly fail too. Waiting for its timeout wastes time and adds load to a service that is already struggling. A circuit breaker — named after the electrical switch — notices repeated failures and stops calling for a while.
class CircuitBreaker:
def __init__(self, threshold=3, cooldown=10):
self.threshold, self.cooldown = threshold, cooldown
self.state, self.failures, self.opened_at = "CLOSED", 0, None
def call(self, now, service_ok):
if self.state == "OPEN":
if now - self.opened_at < self.cooldown:
return "fail fast (not called)"
self.state = "HALF_OPEN" # cool-down over: allow one trial
if service_ok:
self.state, self.failures = "CLOSED", 0
return "success"
self.failures += 1
if self.state == "HALF_OPEN" or self.failures >= self.threshold:
self.state, self.opened_at = "OPEN", now
return "error"
cb = CircuitBreaker()
calls = [(0, False), (1, False), (2, False), (3, True), (8, True), (12, False), (13, True), (24, True)]
for now, ok in calls:
result = cb.call(now, ok)
print(f"t={now:<2} service {'up ' if ok else 'down'} -> {result:<22} state={cb.state}")
t=0 service down -> error state=CLOSED
t=1 service down -> error state=CLOSED
t=2 service down -> error state=OPEN
t=3 service up -> fail fast (not called) state=OPEN
t=8 service up -> fail fast (not called) state=OPEN
t=12 service down -> error state=OPEN
t=13 service up -> fail fast (not called) state=OPEN
t=24 service up -> success state=CLOSED
Follow the trace:
- Three failures in a row trip the breaker to OPEN at t=2.
- At t=3 and t=8 the service is actually back, but the breaker does not call it — it fails fast. That is the price of protection.
- At t=12 the cool-down is over; the breaker goes HALF_OPEN and lets one trial call through. It fails, so the breaker opens again.
- At t=13 calls are blocked; at t=24 the trial succeeds and the breaker closes.
When the breaker is open, the caller should use a fallback instead of an error page where possible.
5. Bulkheads
Ships are divided into watertight compartments called bulkheads, so one hole does not sink the whole ship. In software, give each dependency its own limited pool of threads or connections.
If the order service has 200 threads and lets any dependency use all of them, a slow maps API can consume all 200, and orders stop. With bulkheads — say 20 threads for the maps API, 50 for payments, the rest for everything else — a slow maps API can exhaust only its 20. Orders keep working; only the "estimated delivery time" feature degrades.
6. Graceful degradation and fallbacks
Decide in advance what the product does when a dependency is unavailable:
| Dependency down | Fallback |
|---|---|
| recommendations service | show "popular courses" from a cache |
| review service | show the course page without reviews |
| exact delivery-time estimate | show "30–40 min" based on the city average |
| payment gateway | let the user try another method; never pretend the payment succeeded |
A page that loads with one section missing is far better than an error page. Core flows (login, checkout) deserve the most protection; nice-to-have features can switch off.
Common mistakes
- No timeout, or a default of minutes, on calls to other services.
- Retrying immediately, without backoff or jitter, and retrying non-idempotent writes.
- Retrying at every layer, multiplying load during an outage.
- Treating "slow" the same as "up" — a 30-second response is a failure.
- No plan for what the user sees when a dependency is down.
Quick recap
- In big systems something is always failing; slow dependencies cause cascading failures.
- Put timeouts on every remote call, sized from normal latency.
- Retry only idempotent, transient failures, with exponential backoff, jitter and few attempts.
- Circuit breakers fail fast after repeated errors and test recovery in half-open state.
- Bulkheads isolate each dependency's resources; fallbacks keep the core product working.