Reliability and failure

Design for partial failure

A dependency can be slow while everything else is healthy. Threads, connections, and request queues can fill before an outright error appears. Track saturation and latency, not only success rates.

Decide what the product should do when each dependency is unavailable: reject safely, serve stale data, queue work, remove an optional feature, or ask the user to retry.

Timeout first, then retry carefully

A retry without a timeout simply waits longer. Use a deadline based on the caller’s remaining budget. Retry transient failures with exponential backoff and jitter, and cap both attempts and total elapsed time.

Retries multiply load during an incident. Never blindly retry non-idempotent work. Carry an idempotency key when a command may be repeated.

Contain damage with isolation

Circuit breakers stop repeatedly calling a known-unhealthy dependency. Bulkheads reserve resources so one dependency cannot exhaust every worker or connection. Rate limits protect both your system and downstream services.

These mechanisms need observable states and conservative recovery. A circuit that flaps open and closed can be more disruptive than a clear degraded mode.

Follow the latency budget

Use SLOs to choose engineering work

An SLI measures user-visible behaviour such as successful requests under a latency threshold. An SLO states the target. The error budget is the permitted unreliability over a period.

When a service burns its budget quickly, reduce change and repair reliability. When it remains comfortably healthy, the team can invest more aggressively in product delivery.