Cascading Failure: Timeouts, Retries and Backpressure

One dependency slows down. Twenty minutes later, an unrelated part of the system is returning errors and the dependency that started it has been healthy for ten minutes. That is a cascading failure, and it is almost always amplified by the code written to make the system more reliable.

The chain#

The key insight is that the failure spreads through a shared resource, not through the call graph. Endpoints that never touch recommendations fail because they need a thread from the same pool.

Retry amplification#

@Retryable(maxAttempts = 3)                  // no backoff, no jitter, no limit on which errors
public Recommendations fetch(String userId) {
    return client.get("/recommendations/" + userId);
}

Normal:  1,000 req/s  ->  1,000 calls/s to recommendations
 
Recommendations starts failing. Each request now retries twice:
         1,000 req/s  ->  3,000 calls/s
 
A service that could not serve 1,000 is now asked for 3,000. It cannot
recover even after the original trigger is gone.

Worse, retries compound multiplicatively down a chain. Gateway retries 3 times, product retries 3 times, recommendations retries 3 times: one user action becomes 27 calls at the bottom. This is how a brief blip becomes a multi-hour outage.

Three rules:

Retry only idempotent operations. A retried payment capture can charge twice. Use an idempotency key so the server can deduplicate, and never blind-retry a non-idempotent write.

Retry only retryable errors. A 400, 401, 404 or 422 will fail identically on the second attempt. Retry timeouts, connection failures, 429s and 503s only.

Retry at one layer. If the gateway retries, the services beneath it should not.

RetryConfig config = RetryConfig.custom()
        .maxAttempts(3)
        .intervalFunction(IntervalFunction.ofExponentialRandomBackoff(
                Duration.ofMillis(100), 2.0, 0.5))     // 100ms, 200ms, 400ms — each ±50%
        .retryOnException(e -> e instanceof TimeoutException
                            || e instanceof ConnectException)
        .build();

Jitter is not optional. Without it, every client that failed at the same instant retries at the same instant, producing a synchronised wave that knocks the recovering service over again.

Timeout budgets#

Browser        timeout 5s
  Gateway      timeout 30s    <- longer than the browser will wait
    Product    timeout 30s
      Recs     timeout 30s
      Database timeout 30s
 
Total possible wait: 30s. The browser gave up at 5s.
Everything after that is work nobody will see.

Browser        5s
  Gateway      4s          (1s of headroom)
    Product    3s
      Recs     800ms       optional — degrade if it fails
      Database 2s

Each layer's timeout must be shorter than its caller's remaining budget. Passing the remaining deadline down the chain — as a header, or via gRPC's built-in deadline propagation — is the rigorous version: each service knows how long it has left and can refuse work it cannot finish in time.

HttpClient client = HttpClient.newBuilder()
        .connectTimeout(Duration.ofSeconds(2))       // TCP connect
        .build();
 
HttpRequest request = HttpRequest.newBuilder(uri)
        .timeout(Duration.ofMillis(800))             // the whole request
        .build();

Circuit breakers#

CircuitBreakerConfig config = CircuitBreakerConfig.custom()
        .slidingWindowType(COUNT_BASED)
        .slidingWindowSize(100)
        .failureRateThreshold(50)                          // open at 50% failures
        .slowCallDurationThreshold(Duration.ofSeconds(2))
        .slowCallRateThreshold(50)                         // slow counts as failed
        .waitDurationInOpenState(Duration.ofSeconds(30))
        .permittedNumberOfCallsInHalfOpenState(5)
        .build();

The slowCallRateThreshold setting matters more than the failure threshold. A dependency returning errors quickly is survivable; a dependency answering slowly is what fills your pools. Treating slow as failed is what makes a breaker effective against the scenario at the top of this page.

public Page render(String userId) {
    List<Product> recs = circuitBreaker.executeSupplier(() -> client.fetch(userId))
            // Open circuit or failure: show the page without recommendations.
            .recover(ex -> popularProducts.cached())
            .get();
    return new Page(recs);
}

An optional feature should never be able to fail the whole page. Deciding, per dependency, whether it is critical or optional is a design exercise worth doing explicitly — and it is a good thing to volunteer in a system-design round.

The four defences together#

Supplier<Recommendations> decorated = Decorators
        .ofSupplier(() -> client.fetch(userId))
        .withBulkhead(bulkhead)              // outermost: limit concurrency
        .withCircuitBreaker(circuitBreaker)  // then: is it even worth trying?
        .withRetry(retry)                    // then: retry transient failures
        .withFallback(List.of(Exception.class), ex -> cached())
        .decorate();

The order is deliberate. The bulkhead is outermost so concurrency is capped before anything else runs. The retry is inside the circuit breaker so that retries count towards the failure rate — putting it outside means retries never open the breaker, which defeats both.

Load shedding is the fourth and least implemented. When the queue is deep, reject new requests immediately with a 503 rather than accepting work that will time out anyway. CallerRunsPolicy on a bounded executor is the simplest form; a queue-depth check in a filter is the explicit one.

The recovery problem#

A failure mode people forget: the dependency comes back, and every client retries simultaneously.

// 10,000 requests queued during a 5-minute outage, all released at once
// against a service that has just restarted with cold caches and an
// empty connection pool. It falls over again immediately.

Defences: jittered backoff so retries are spread over time; the circuit breaker's half-open state, which admits only a handful of probe calls; and dropping work that is already stale rather than replaying it — a request whose client timed out four minutes ago should not be executed.

The incident, end to end#

1. Symptom. Checkout returning 503s. Checkout does not call recommendations.
2. Trace. Distributed tracing shows checkout requests spending 8 seconds in the product service.
3. Product service. Every worker thread blocked in the recommendations client. Queue at 40,000.
4. Recommendations. A deploy 25 minutes earlier introduced an unindexed query; p99 went from 200ms to 8s.
5. Why did it spread? No timeout on the recommendations client, one shared thread pool, and three retries per call tripling the load.
6. Mitigate. Feature-flag recommendations off. Product recovers in 60 seconds.
7. Fix. 800ms timeout, dedicated bulkhead, circuit breaker with a cached fallback, retries reduced to one with jitter.
8. Guardrail. A CI check that fails if any HTTP client bean is built without an explicit timeout, plus a game-day exercise that injects latency into each dependency and asserts the blast radius.

What gets asked#

"How do you stop one slow service taking down the rest?" is a system-design staple. Name all four defences, and explain why retries make things worse — the amplification arithmetic is concrete and memorable. If you can also describe the recovery thundering herd, you are describing someone who has watched a system come back up badly.