Resilience patterns exist to handle the failures that are guaranteed to occur in distributed systems. Downstream services become unavailable. Networks time out. Databases become overloaded. These are not exceptional events — they’re normal operating conditions that a production service must handle correctly.
The mistake is adding resilience patterns everywhere as a precaution. A circuit breaker on a service that never fails adds overhead without value. A retry on a non-idempotent operation can cause duplicate side effects. Resilience patterns should be applied deliberately, to specific failure modes that matter for your system.
Resilience4j: The Standard Library for Java
Resilience4j is the go-to resilience library for Java services. It provides circuit breakers, retries, bulkheads, time limiters, and rate limiters as composable decorators.
Spring Boot autoconfigures Resilience4j when the starter is present:
<dependency>
<groupId>io.github.resilience4j</groupId>
<artifactId>resilience4j-spring-boot3</artifactId>
</dependency>
Circuit Breakers
A circuit breaker tracks the failure rate of a dependency and stops sending calls when failures exceed a threshold. This prevents the caller from wasting resources waiting for a dependency that’s clearly not working.
States:
- Closed: calls pass through normally
- Open: calls fail immediately, no attempt made
- Half-open: a test call is made to check if the dependency has recovered
resilience4j:
circuitbreaker:
instances:
inventory-service:
failure-rate-threshold: 50 # Open when 50% of calls fail
slow-call-rate-threshold: 80 # Also open when 80% are "slow"
slow-call-duration-threshold: 2s # Threshold for "slow"
wait-duration-in-open-state: 30s # Wait before trying again
sliding-window-size: 20 # Evaluate over last 20 calls
minimum-number-of-calls: 5 # Min calls before evaluation
permitted-calls-in-half-open: 3 # Test calls in half-open state
@Service
public class InventoryService {
@CircuitBreaker(name = "inventory-service", fallbackMethod = "getDefaultStock")
public StockLevel getStockLevel(ProductId productId) {
return inventoryClient.getStock(productId.value());
}
private StockLevel getDefaultStock(ProductId productId, Exception e) {
log.warn("Inventory service unavailable for {}, using cached/default", productId);
return stockCache.getOrDefault(productId, StockLevel.unknown());
}
}
The fallback method receives the same parameters as the original method plus the exception. It provides degraded-but-functional behavior instead of an error.
When to apply: any network call to an external service where failure should not cascade to your service. Not useful for internal calls that are effectively free.
Retries
Retries handle transient failures — network blips, temporary service unavailability, rate limit errors. They’re appropriate when:
- The operation is idempotent (safe to call multiple times)
- The failure is likely transient (network error, not a 400 Bad Request)
- Retrying won’t make the problem worse (consider exponential backoff to avoid thundering herd)
resilience4j:
retry:
instances:
payment-service:
max-attempts: 3
wait-duration: 500ms
exponential-backoff-multiplier: 2 # 500ms, 1s, 2s
retry-exceptions:
- java.net.ConnectException
- java.net.SocketTimeoutException
- org.springframework.web.client.HttpServerErrorException$ServiceUnavailable
ignore-exceptions:
- java.lang.IllegalArgumentException
- org.springframework.web.client.HttpClientErrorException # 4xx — don't retry
@Service
public class PaymentService {
@Retry(name = "payment-service")
@CircuitBreaker(name = "payment-service")
public PaymentResult processPayment(PaymentRequest request) {
return paymentGateway.charge(request);
}
}
Critical: only retry idempotent operations. Retrying a payment that wasn’t acknowledged (timeout) can result in double charging. Design payment operations with idempotency keys before adding retries.
Retry budgets: at a service level, cap total retry rate. If 30% of incoming requests are retries, you’re amplifying traffic to an already-struggling downstream service.
Bulkheads
A bulkhead isolates resources for different operations, preventing one from exhausting resources needed by others. Named after ship bulkheads that contain flooding to one compartment.
Two types:
Thread pool bulkhead: dedicate a thread pool to each dependent service.
resilience4j:
bulkhead:
instances:
inventory-service:
max-concurrent-calls: 20 # Thread pool size
max-wait-duration: 100ms # Wait before failing if pool full
If the inventory service is slow and backing up, the 20-thread pool for inventory fills up. The slowdown does not affect the thread pool for payment service or any other dependency.
Semaphore bulkhead: limit concurrent calls without a separate thread pool.
@Bulkhead(name = "inventory-service", type = Bulkhead.Type.SEMAPHORE)
public StockLevel getStockLevel(ProductId productId) {
return inventoryClient.getStock(productId.value());
}
When to apply: high-traffic services where a slow dependency could exhaust your thread pool. Particularly relevant when your service calls multiple downstream services with different reliability characteristics.
Time Limiters
Time limiters abort calls that exceed a duration limit. They work with async operations:
resilience4j:
timelimiter:
instances:
inventory-service:
timeout-duration: 2s
cancel-running-future: true
@TimeLimiter(name = "inventory-service")
@CircuitBreaker(name = "inventory-service")
public CompletableFuture<StockLevel> getStockLevelAsync(ProductId productId) {
return CompletableFuture.supplyAsync(
() -> inventoryClient.getStock(productId.value())
);
}
For synchronous calls with virtual threads, timeout is often more elegantly handled at the HTTP client level than with Resilience4j.
Rate Limiters
Protect downstream services from being overwhelmed by your service:
resilience4j:
ratelimiter:
instances:
external-api:
limit-for-period: 100 # 100 requests per refresh period
limit-refresh-period: 1s # Refresh every second
timeout-duration: 500ms # Wait up to 500ms for a permit
Also use rate limiters for incoming requests from your clients:
// Rate limiting by API key using Resilience4j
@Service
public class ApiGatewayService {
private final Map<String, RateLimiter> rateLimiters = new ConcurrentHashMap<>();
public void checkRateLimit(String apiKey) {
RateLimiter limiter = rateLimiters.computeIfAbsent(
apiKey,
key -> RateLimiter.of("api-key-" + key, RateLimiterConfig.custom()
.limitForPeriod(100)
.limitRefreshPeriod(Duration.ofSeconds(1))
.build())
);
limiter.acquirePermission(); // Throws RateLimiterFullException if limit exceeded
}
}
Composing Patterns
Patterns compose. The typical order: time limiter → circuit breaker → retry → bulkhead.
@TimeLimiter(name = "inventory")
@CircuitBreaker(name = "inventory", fallbackMethod = "inventoryFallback")
@Retry(name = "inventory")
@Bulkhead(name = "inventory")
public CompletableFuture<StockLevel> checkInventory(ProductId id) {
return CompletableFuture.supplyAsync(() -> inventoryClient.getStock(id));
}
The decorator order matters: time limiter wraps the whole thing (including retries), circuit breaker opens on too many failures (including retried failures), retries happen before the circuit opens.
The Observability Requirement
Resilience patterns are invisible unless you observe them. Add metrics:
management:
health:
circuitbreakers:
enabled: true
metrics:
export:
prometheus:
enabled: true
Dashboard what matters:
- Circuit breaker state transitions (open/closed/half-open)
- Retry attempts vs successes vs exhausted
- Bulkhead rejection rate
- Fallback invocation rate
A circuit breaker opening and immediately closing is normal behavior. A circuit breaker staying open for an extended period is an incident.
Deliberate Application
The question before adding any resilience pattern: “what failure mode am I protecting against, and is this the right mitigation?”
Circuit breaker: “This dependency goes down periodically and I want to fail fast rather than queue up waiting threads.”
Retry: “This call fails transiently sometimes but succeeds if retried, and the operation is idempotent.”
Bulkhead: “If this dependency is slow, I don’t want it to starve other dependencies of threads.”
Time limiter: “I need a hard bound on how long this call can take.”
Adding all patterns to all calls “just in case” creates operational complexity without proportional benefit. Know why each pattern is where it is.