Circuit Breakers & Resilience4j

A circuit breaker exists for one reason: when a dependency is down, continuing to call it makes things worse. Every request you send occupies a thread, a connection and a timeout while returning nothing, until your own service falls over from a failure that was never yours.

The state machine#

Half-open is the part that makes this better than a manual kill switch. After the wait duration the breaker lets a small number of calls through; if they succeed it closes, if any fails it opens again for another wait period. Nobody has to notice the recovery and flip a flag.

Configuration that works#

resilience4j:
  circuitbreaker:
    configs:
      default:
        sliding-window-type: COUNT_BASED
        sliding-window-size: 50
        # The breaker will not evaluate until it has seen this many calls.
        # Leave it at the default of 100 on a low-traffic endpoint and the
        # breaker will effectively never open.
        minimum-number-of-calls: 20
        failure-rate-threshold: 50
        # A dependency that responds in 8 seconds is failing even when it
        # returns 200. Without these two lines the breaker ignores that.
        slow-call-duration-threshold: 2s
        slow-call-rate-threshold: 50
        wait-duration-in-open-state: 30s
        permitted-number-of-calls-in-half-open-state: 5
        automatic-transition-from-open-to-half-open-enabled: true
        record-exceptions:
          - java.io.IOException
          - java.util.concurrent.TimeoutException
          - org.springframework.web.client.HttpServerErrorException
        # A 4xx is the caller's fault and will fail identically on retry.
        # Counting it as a breaker failure opens the circuit for everyone
        # because one client sent bad requests.
        ignore-exceptions:
          - com.acme.payments.PaymentDeclinedException
          - org.springframework.web.client.HttpClientErrorException
    instances:
      payments:
        base-config: default
        wait-duration-in-open-state: 10s
      catalogue:
        base-config: default
        failure-rate-threshold: 70
 
  retry:
    instances:
      payments:
        max-attempts: 3
        wait-duration: 200ms
        enable-exponential-backoff: true
        exponential-backoff-multiplier: 2
        retry-exceptions:
          - java.io.IOException
          - java.util.concurrent.TimeoutException
 
  bulkhead:
    instances:
      payments:
        max-concurrent-calls: 20
        max-wait-duration: 100ms
 
  timelimiter:
    instances:
      payments:
        timeout-duration: 3s
        cancel-running-future: true

The ignore-exceptions list is the one most often left empty, and leaving it empty causes a specific production incident: a client starts sending malformed requests, your downstream correctly returns 400, the breaker counts those as failures and opens — and now every other client is cut off from a service that was working perfectly.

Applying it#

@Component
public class PaymentGateway {
 
    private static final Logger log = LoggerFactory.getLogger(PaymentGateway.class);
 
    private final RestClient client;
    private final PaymentOutbox outbox;
 
    @CircuitBreaker(name = "payments", fallbackMethod = "chargeFallback")
    @Retry(name = "payments")
    @Bulkhead(name = "payments")
    public Receipt charge(ChargeRequest request) {
        return client.post().uri("/v2/charges").body(request)
                     .retrieve().body(Receipt.class);
    }
 
    // The fallback signature is the original parameters plus the Throwable.
    // Declaring the specific exception type lets you handle "circuit is open"
    // differently from "the call failed".
    private Receipt chargeFallback(ChargeRequest request, CallNotPermittedException ex) {
        log.warn("payments circuit open — queueing {}", request.orderId());
        outbox.enqueue(request);
        return Receipt.pending(request.orderId());
    }
 
    private Receipt chargeFallback(ChargeRequest request, Throwable ex) {
        log.error("payment failed for {}", request.orderId(), ex);
        throw new PaymentUnavailableException(ex);
    }
}

Decorator order#

When several annotations apply, Resilience4j composes them in a fixed order, outermost first:

Bulkhead → TimeLimiter → RateLimiter → CircuitBreaker → Retry

Read from the inside out, this is exactly what you want. Retry sits innermost, so a transient blip is retried before anything else sees a failure. The circuit breaker wraps the retry, so a sustained outage counts the whole retried attempt as one failure and eventually stops the retries happening at all. The bulkhead is outermost, capping how much of your capacity this dependency can consume regardless of what the inner layers do.

The wrong order — breaker inside retry — means each retry attempt is evaluated separately by the breaker, so a single logical call registers three failures and the circuit opens three times faster than the configured threshold implies.

Bulkheads#

A breaker protects you from a dependency that is failing. A bulkhead protects you from one that is merely slow, which is more common and often more damaging, because slow calls do not trip a breaker but do occupy threads.

Capping payments at twenty concurrent calls means a payment slowdown degrades checkout while search, browsing and account pages carry on. Without the cap, every request thread eventually blocks on payments and the entire application stops responding — the classic cascading failure.

Two implementations exist. The semaphore bulkhead counts concurrent calls in the calling thread and is nearly free. The thread-pool bulkhead runs calls on a separate bounded pool, isolating more strongly at the cost of a context switch and the loss of thread-locals such as the MDC and the security context. Start with the semaphore.

Observability#

Resilience4j publishes Micrometer metrics automatically. Three are worth putting on a dashboard and alerting on.

resilience4j_circuitbreaker_state as a gauge tells you which breakers are open right now — the single most useful panel during an incident, because it immediately answers "which dependency is the problem". resilience4j_circuitbreaker_calls broken down by kind shows successful, failed and not-permitted calls, and the not-permitted count is how much traffic your fallback is absorbing. resilience4j_retry_calls shows how often retries are happening and whether they succeed; a rising retry rate is an early warning that arrives before the breaker opens.

Expose breaker state through the health endpoint too, but be careful: a service whose readiness fails because one non-critical breaker is open will be removed from the load balancer for a problem it can tolerate. Report breaker state as health detail, and only fail readiness for dependencies without which the service genuinely cannot serve.

What to take away#

Set minimum-number-of-calls to something your traffic will actually reach, count slow calls as failures, and ignore client errors. Layer retry inside the breaker inside the bulkhead. Write fallbacks that degrade honestly, and put breaker state on a dashboard — it is the fastest way to see whose outage you are experiencing.