Every system that communicates over a network is a distributed system, and distributed systems fail in ways that single-process applications don’t. This is not opinion — it’s physics. Networks drop packets. Clocks drift. Services restart. Disks fail. These things happen regardless of how carefully you write your application code.
The teams that get distributed systems right don’t eliminate these failures. They design so that their systems remain correct despite them.
The Fallacies You’ll Rediscover
Peter Deutsch’s eight fallacies of distributed computing (1994) are still relevant because they describe assumptions that feel true in development and false in production:
- The network is reliable
- Latency is zero
- Bandwidth is infinite
- The network is secure
- Topology doesn’t change
- There is one administrator
- Transport cost is zero
- The network is homogeneous
The first two account for probably 80% of distributed systems incidents. Understanding them deeply is worth the time.
The network is not reliable. This does not mean it fails often. It means it can fail at any time, in any direction, for any duration. More specifically: you cannot distinguish between these scenarios from the calling side:
- The request never arrived
- The request arrived but the service crashed before processing it
- The request was processed but the response was lost
- The service is processing it very slowly
From the caller’s perspective, a timeout looks the same in all four cases. The implications are significant: you cannot assume that a failed call means the operation did not happen.
Latency is not zero. This matters most in the aggregate. A service that makes 20 synchronous calls to downstream services before returning a response compounds latency at every step. If each call has a p99 of 50ms, the combined p99 is not 50ms. Tail latency in a chain of calls behaves much worse than the arithmetic suggests.
Partial Failure
The failure mode unique to distributed systems is partial failure: some components succeed while others fail, leaving the system in an indeterminate state.
A payment service calls an inventory service to check stock, then calls a payment processor, then calls the inventory service again to decrement stock. If the final call fails:
- Payment was charged ✓
- Stock was not decremented ✗
- Order cannot be confirmed ✗
The system is inconsistent. Not corrupted — just inconsistent. No single-process rollback mechanism helps here.
This is why distributed transactions are hard and why most production systems avoid them in favor of eventual consistency and compensating actions.
Retries and Idempotency
The obvious response to network failures is retries. This is correct. The non-obvious implication is that any operation you retry must be safe to execute multiple times.
If a payment processing request times out, you don’t know whether the payment was charged. Retrying may charge the customer twice. This is a real failure mode in production payment systems.
The solution is idempotency: design operations so that applying them multiple times has the same effect as applying them once.
The standard mechanism is an idempotency key — a client-generated unique identifier for each logical operation:
public PaymentResult processPayment(ProcessPaymentCommand command) {
// Check if this operation was already completed
Optional<PaymentResult> existing = idempotencyStore
.findByKey(command.idempotencyKey());
if (existing.isPresent()) {
return existing.get(); // Return the same result
}
PaymentResult result = paymentGateway.charge(
command.amount(),
command.paymentMethod()
);
// Store result before returning
idempotencyStore.store(command.idempotencyKey(), result);
return result;
}
The idempotency key is generated by the client — the caller — so that retries use the same key and receive the same response.
Key design considerations:
- The key must be unique per logical operation, not per HTTP request
- Store results durably (database, not in-memory)
- Set an expiry on the key (7 days is common)
- The window must exceed the maximum retry window
Retry Strategies
Retries without control make cascading failures worse. When a service is overloaded, clients retrying immediately add more load at exactly the wrong moment.
Exponential backoff with jitter is the standard approach:
public <T> T withRetry(Supplier<T> operation, int maxAttempts) {
int attempt = 0;
while (true) {
try {
return operation.get();
} catch (TransientException e) {
attempt++;
if (attempt >= maxAttempts) throw e;
long baseDelay = 100L * (1L << attempt); // 200ms, 400ms, 800ms...
long jitter = ThreadLocalRandom.current().nextLong(baseDelay / 2);
long delay = Math.min(baseDelay + jitter, 30_000L); // cap at 30s
Thread.sleep(delay);
}
}
}
Jitter prevents the thundering herd: when many clients retry simultaneously at the same interval, they all hit the recovering service at the same moment. Random jitter spreads the retry load.
Retry budgets prevent runaway retries at the system level. Rather than per-request retry counts, track retries across all requests and cap the total retry rate.
Don’t retry on 4xx errors. A 400 Bad Request will be a 400 on every retry. Only retry on transient failures (network errors, 429 rate limit with respect for Retry-After, 503 Service Unavailable).
Circuit Breakers
The failure mode circuit breakers prevent is cascading failure — one slow service causing all of its callers to exhaust their thread pools, which makes them appear slow to their callers, propagating up the dependency graph.
A circuit breaker wraps calls to a dependency and tracks the failure rate. Three states:
- Closed: calls pass through normally
- Open: calls fail immediately without hitting the dependency
- Half-open: a test call is allowed through to probe recovery
@Component
public class InventoryClient {
private final CircuitBreaker circuitBreaker = CircuitBreaker.ofDefaults("inventory");
public Optional<StockLevel> getStockLevel(ProductId productId) {
return circuitBreaker.executeSupplier(() ->
inventoryApi.getStockLevel(productId.value())
);
}
}
When a dependency is degraded, the circuit opens. Subsequent calls fail fast — in milliseconds — rather than waiting for timeouts that might take 30 seconds each. The caller can handle the failure gracefully (return a cached value, return a degraded response, queue the operation for later) rather than blocking indefinitely.
Resilience4j is the standard choice for Java. The configuration that matters:
resilience4j.circuitbreaker:
instances:
inventory:
failure-rate-threshold: 50 # Open when 50% of calls fail
wait-duration-in-open-state: 30s # Stay open for 30 seconds
permitted-number-of-calls-in-half-open-state: 3
sliding-window-size: 20 # Evaluate over last 20 calls
Timeouts: Required, Not Optional
Every network call must have a timeout. Without one, a slow downstream service causes the calling thread to block indefinitely.
The challenge is setting the right value. Too short and you have spurious failures. Too long and you don’t detect real problems fast enough.
A starting framework:
- Measure the p99 latency of the dependency under normal load
- Set the timeout at 2–3× p99
- Adjust based on operational experience
Different timeouts for different call types:
- User-facing synchronous calls: tight budget (200ms–1s)
- Background processing: looser budget (5s–30s)
- Health checks: very tight (100ms)
// Using WebClient in Spring Boot
WebClient.builder()
.baseUrl("http://inventory-service")
.build()
.get()
.uri("/stock/{productId}", productId)
.retrieve()
.bodyToMono(StockLevel.class)
.timeout(Duration.ofMillis(500)) // Hard timeout
.onErrorReturn(TimeoutException.class, StockLevel.unknown(productId));
The Outbox Pattern
The outbox pattern solves a specific distributed systems problem: how to write to a database and publish a message atomically.
The naive approach fails:
// WRONG — not atomic
repository.save(order); // Succeeds
eventBus.publish(orderPlaced); // Fails — order saved but event not published
Or:
// ALSO WRONG
eventBus.publish(orderPlaced); // Succeeds
repository.save(order); // Fails — event published but order not saved
The outbox pattern avoids this by writing the message to a database table in the same local transaction as the primary write:
@Transactional
public void placeOrder(Order order) {
orderRepository.save(order);
// Write to outbox in same transaction
outboxRepository.save(new OutboxMessage(
"order.placed",
toJson(new OrderPlaced(order.id(), order.customerId())),
Instant.now()
));
}
A separate process reads the outbox and publishes the messages:
@Scheduled(fixedDelay = 1000)
public void publishOutboxMessages() {
List<OutboxMessage> messages = outboxRepository.findUnpublished();
for (OutboxMessage message : messages) {
eventBus.publish(message.topic(), message.payload());
outboxRepository.markPublished(message.id());
}
}
This guarantees at-least-once delivery. Messages may be published more than once if the publisher fails after publishing but before marking complete — which is why consumers must be idempotent.
Transactional outbox is available in tools like Debezium (CDC-based) for more sophisticated implementations.
Ordering and Causality
Distributed systems do not guarantee message ordering across producers. If service A sends messages M1 and M2, a consumer may receive M2 before M1.
Design for out-of-order delivery:
- Use timestamps or sequence numbers to detect ordering issues
- Make your state transitions idempotent regardless of ordering
- Use a global ordering mechanism (Kafka partition key) where ordering is genuinely required
Kafka provides ordering within a partition. If all events for a given entity (order, user, product) go to the same partition (using the entity ID as the partition key), you get ordered delivery for that entity.
Clock Problems
Don’t use wall clock times for distributed coordination. Clocks drift. NTP synchronization is approximate. In a distributed system, you cannot assume that System.currentTimeMillis() on two different machines returns comparable values.
Practical implications:
- Don’t use timestamps to determine which of two distributed writes happened first
- Don’t use timestamps as unique identifiers
- Use logical clocks (Lamport timestamps) or database-generated sequences when ordering matters
- Accept that events from different services may arrive with timestamps that appear to violate causality
For most application-level code, this means: generate timestamps for human-readable purposes (when an order was placed), but use database-generated monotonic sequences for coordination.
Observability Is Not Optional
Distributed systems fail in non-obvious ways. A 5% error rate in one service, combined with a specific retry pattern, can cause a 40% error rate in a dependent service in ways that only become clear from distributed tracing.
The minimum observability stack for distributed systems:
- Structured logging with correlation IDs that flow across service boundaries
- Distributed tracing (OpenTelemetry → Jaeger/Tempo) to visualize cross-service request flows
- Service-level metrics: request rate, error rate, latency (p50/p95/p99) per service
- Dependency health: track error rates and latency of each downstream dependency separately
When something goes wrong, you need to be able to trace a failing request through every service it touched. Without distributed tracing, this is guesswork.
The Honest Summary
These aren’t hypothetical edge cases. In any system that handles meaningful production traffic, all of these problems will occur:
- Network calls will time out
- Messages will be delivered more than once
- Services will be temporarily unavailable
- Clocks will be slightly out of sync
The question is whether your system is designed to handle them gracefully or whether each occurrence causes an incident. Design for the failures you know are coming rather than discovering them in production.