Reliability exit conditions
Candidate exit conditions for the Reliability type. See the catalog overview for how to adopt these.
Retries and backoff
| Exit condition | What it checks |
|---|---|
|
Retried operations increase their delay exponentially between successive attempts rather than retrying at a fixed interval. |
|
Backoff delays include randomized jitter so concurrent clients do not retry in synchronized waves. |
|
Every retry policy declares a finite maximum attempt count or total elapsed budget. |
|
Retries are attempted only for transient or retryable failure classes, not for deterministic client errors. |
|
A retry budget caps the fraction of total traffic that may be retries to prevent retry-amplified overload. |
|
Exponential backoff is capped at a maximum delay so retry intervals never grow unbounded. |
|
Retry waits are scheduled asynchronously and do not block worker threads while awaiting the next attempt. |
|
Exhaustion of a retry policy emits a distinct terminal signal or metric rather than failing silently. |
|
Messages that exhaust all retries are routed to a dead-letter queue or equivalent quarantine surface. |
Timeouts and deadlines
| Exit condition | What it checks |
|---|---|
|
Every outbound network or inter-service call is governed by an explicit timeout. |
|
No blocking call, lock acquisition, or await is permitted to wait indefinitely. |
|
A request deadline is propagated across service hops so downstream calls inherit the remaining budget. |
|
Each service’s timeout is shorter than its callers' timeouts to avoid wasted work on already-abandoned requests. |
|
Connection-establishment timeouts are configured separately from request or read timeouts. |
|
A cancelled or timed-out request cancels its in-flight downstream work rather than orphaning it. |
|
Every request-response (ask) interaction between entities or services specifies a bounded reply timeout. |
|
Timeout durations are externally configurable and not hardcoded at call sites. |
Circuit breaking and bulkheads
| Exit condition | What it checks |
|---|---|
|
Calls to each external dependency are wrapped in a circuit breaker that opens on sustained failures. |
|
An open circuit transitions to a half-open state that probes recovery with limited trial traffic before fully closing. |
|
Circuit-breaker trip conditions are defined by explicit failure-rate or consecutive-failure thresholds. |
|
An open circuit fails calls immediately instead of allowing them to hang against a failing dependency. |
|
Independent dependencies use isolated resource pools so exhaustion in one cannot starve the others. |
|
Concurrent in-flight calls to each dependency are capped to a bounded limit. |
|
All work queues and mailboxes are bounded so backlog cannot grow without limit under load. |
|
Streaming and ingestion paths propagate backpressure to upstream producers rather than buffering unboundedly. |
|
The system sheds or rejects excess load with a clear signal once capacity limits are reached. |
Idempotency and delivery guarantees
| Exit condition | What it checks |
|---|---|
|
State-changing operations produce the same result when the same request is delivered more than once. |
|
Externally triggered mutations carry an idempotency key used to deduplicate retried requests. |
|
Duplicate-suppression logic specifies a bounded retention window for tracked request identifiers. |
|
Consumers tolerate at-least-once delivery by handling redelivered messages without corrupting state. |
|
Effectively-once processing is achieved by combining at-least-once delivery with idempotent effects. |
|
State changes and their outgoing messages are committed atomically via an outbox to prevent lost or ghost events. |
|
Incoming messages are acknowledged only after their effects are durably persisted. |
|
Per-entity or per-key message ordering is preserved where the domain requires it. |
|
Repeatedly failing messages are isolated so they do not block progress on the rest of the stream. |
Cluster resilience and node loss
| Exit condition | What it checks |
|---|---|
|
The cluster continues serving traffic after the abrupt loss of any single node. |
|
No single instance, node, or process is the sole path for any critical operation. |
|
Sharded entities on a lost node are rebalanced and reactivated on surviving nodes automatically. |
|
A split-brain resolver deterministically downs the losing partition to prevent divergent cluster halves. |
|
Cluster membership and leadership decisions require a majority quorum to proceed. |
|
Nodes can be restarted one at a time with no loss of availability or in-flight work. |
|
A node hands off its shards and drains work before leaving the cluster. |
|
Shard rebalancing converges to a balanced distribution within a bounded time after membership changes. |
|
The cluster retains sufficient headroom to absorb the load of a lost node without saturation. |
Supervision and recovery
| Exit condition | What it checks |
|---|---|
|
Every stateful component declares an explicit supervision strategy for handling failures. |
|
Failed components restart under a backoff supervisor rather than restarting in a tight loop. |
|
An event-sourced entity reconstructs its state by replaying its persisted event log after restart. |
|
Passivated entities can be reactivated later with their state fully recovered from durable storage. |
|
Snapshots are taken so entity recovery replay time stays within an acceptable bound. |
|
Component and entity recovery completes within a defined maximum duration. |
|
A failure in one entity or component does not cascade to unrelated entities or components. |
|
Recovery detects and quarantines corrupt or unreplayable state instead of crash-looping on it. |
|
Durable in-flight work such as sagas or projections resumes from its last checkpoint after a restart. |
Health, readiness, and graceful shutdown
| Exit condition | What it checks |
|---|---|
|
The service exposes a liveness probe that reflects whether the process is healthy enough to run. |
|
The service exposes a readiness probe that gates traffic until dependencies and warmup are ready. |
|
Readiness reports not-ready when a required downstream dependency is unavailable. |
|
Shutdown drains in-flight requests and releases resources before the process exits. |
|
On shutdown the instance stops accepting new work before terminating existing work. |
|
Graceful shutdown completes within the platform-provided termination grace period. |
|
Startup waits for or degrades gracefully around dependencies that are not yet available. |
|
Liveness checks avoid heavy downstream calls that could cause false-positive restarts. |
|
Health and readiness probes have bounded timeouts so a hung check cannot stall orchestration. |
Graceful degradation and fallbacks
| Exit condition | What it checks |
|---|---|
|
A degraded but functional response is returned when a non-critical dependency is unavailable. |
|
Cached or last-known-good data is served when the source of truth is temporarily unreachable. |
|
Safe default values are used when optional data cannot be retrieved in time. |
|
The critical request path depends on the smallest possible set of must-succeed dependencies. |
|
Failures in non-essential features degrade quietly without failing the overall request. |
|
Aggregations return available partial results when some contributing sources fail. |
|
Expensive or fragile features can be disabled at runtime to preserve core availability. |
|
Operation in a degraded mode is surfaced through metrics or response metadata for observability. |
Disaster recovery and failover
| Exit condition | What it checks |
|---|---|
|
A recovery-time objective is defined and validated against measured recovery duration. |
|
A recovery-point objective is defined and validated against measured maximum data loss. |
|
Durable state is backed up on a defined schedule covering all critical data stores. |
|
Backup restoration is periodically exercised end-to-end to prove backups are usable. |
|
Critical state is replicated across regions to survive a full regional outage. |
|
Failover to a secondary region is tested and completes within the defined RTO. |
|
The system can return to its primary region after recovery without data loss or manual surgery. |
|
Persisted data is stored with sufficient replication to survive the loss of a storage node or zone. |
|
A disaster-recovery runbook exists and has been executed successfully in a drill. |
Fault injection and resilience validation
| Exit condition | What it checks |
|---|---|
|
Randomized node termination is exercised and the system maintains its availability targets. |
|
Injected downstream latency is validated to trigger timeouts and fallbacks rather than cascading stalls. |
|
A simulated dependency outage confirms circuit breakers and degradation paths behave as designed. |
|
Induced network partitions are validated against the split-brain and quorum behavior. |
|
Behavior under CPU, memory, or connection exhaustion is exercised and stays within safe limits. |
|
Correctness is validated under injected clock skew between nodes. |
|
A sudden load spike is validated to trigger shedding or backpressure without collapse. |
|
Resilience experiments define a measurable steady-state hypothesis that is asserted before and after fault injection. |
|
Fault-injection experiments confirm that a single failure’s impact stays within its expected blast radius. |