Key takeaways
- A timeout means the outcome is unknown, not that nothing happened. Reconcile the destination before deciding whether to retry a side effect.
- Give each intended business effect a stable idempotency key and bind it to the same operation and parameters for every attempt.
- Retry only failures that may change with time. Invalid input, denied permission and failed business rules need correction or escalation, not another attempt.
- Use one retry owner, a capped attempt budget, exponential backoff and jitter. Layered retries can multiply load exactly when a dependency is weakest.
- Persist attempt, effect and reconciliation states separately so recovery can distinguish not-started, in-flight, committed and contradicted work.
A timeout creates an unknown outcome
Suppose a workflow sends a write and waits. The connection closes before a response arrives. Three states are possible: the destination never received it, the effect committed but the response was lost, or processing is still in flight. Retrying immediately is safe only in the first state. The client cannot infer that state from the timeout itself.
Amazon's Builders' Library guidance warns that retries add load to a dependency already under stress and that side-effecting APIs need idempotency to be safely retried. The workflow should classify the failure, reconcile when possible and spend a limited retry budget at one layer.
| Observed result | Retry? | Next state |
|---|---|---|
| Rate limit, explicit temporary error | Yes, within budget | Wait for provider guidance or capped backoff with jitter |
| Timeout before a read | Usually | Retry the read; it has no external effect |
| Timeout after a write was sent | Only with idempotency or reconciliation | Mark effect unknown; query destination |
| Invalid schema or business rule | No | Correct input or enter exception review |
| Permission denied | No | Escalate ownership or authorization |
| Model returned unsupported action | No blind retry | Fail validation; inspect instruction and case |
| Human rejected proposal | No | Terminal rejection or a new authorized revision |
Key the intended effect, not the attempt
An attempt identifier should be new on every try. An effect identifier should remain stable across every attempt to do the same business thing. Build it from a non-sensitive business identity, operation and semantic version: ticket:SUP-4182:assign:v1, for example. Do not put email addresses, names or raw customer data in the key.
Bind the key to a payload digest or stored parameters. Reusing the same key with a different destination or amount is a conflict, not deduplication. Stripe's idempotency reference describes this important guard: the service compares later parameters with the original request and rejects mismatches.
| Record | Stable identity | Important states |
|---|---|---|
| Run | Trigger event plus workflow version | started, waiting, completed, failed |
| Attempt | New identifier for each call | scheduled, sent, response, timed_out |
| Effect | Business key plus operation and version | not_started, in_flight, committed, contradicted |
| Reconciliation | Effect key plus check time | absent, matches, partial, conflicts, unknown |
Write the retry contract before the first failure
- 01Choose one retry ownerHuman approvalDecide whether the client, queue worker or workflow retries. Disable hidden repeats elsewhere or include them in the same budget.
- 02Classify retryable outcomesCodeList exact response codes and failure states that may change without altering the request. Everything else fails closed.
- 03Set timeout and attempt budgetsHuman approvalCap calls, elapsed time and total retry load. The business deadline must exceed the nested technical deadlines.
- 04Add backoff and jitterCodeIncrease delay between attempts, cap it and randomize the wait so many workers do not return together.
- 05Reconcile ambiguous effectsCodeQuery by the business key or idempotency key. If no authoritative check exists, stop for review instead of guessing.
- 06Resume from a checkpointCodeReuse validated prior results and repeat only the failed safe step. A whole-run restart increases cost and duplicate risk.
Layered retries multiply. Three attempts across five independent layers can produce hundreds of calls at the weakest dependency. A single owner makes the load, audit trail and stopping condition legible.
Reconcile before repeating a customer-facing effect
A workflow approves a response for ticket SUP-4182 and sends it with effect key ticket:SUP-4182:reply:v3. The request times out after transmission. The run enters effect_unknown; it does not regenerate the message or restart ticket classification.
A reconciliation read finds a reply carrying the same key and matching payload digest. The effect moves to committed, and the run continues to verify ticket status. If the reply were absent, the same payload and key could be retried once after backoff. If a different payload held that key, the workflow would stop for conflict review.
Only the transport attempt repeats. The approved content, authority, business identity and intended effect stay fixed.
When reconciliation returns partial or conflicting state, follow AI agent failure recovery. A retry policy should never improvise compensation.
Limitations and when not to use this
- Idempotency prevents repeated effects only within the destination's implementation and retention window. Verify its exact semantics and expiry.
- Some external systems expose neither an idempotency key nor a reliable read-after-write check. Put ambiguous writes behind a human reconciliation queue.
- A duplicate-free wrong effect is still wrong. Approval, authorization, input validation and destination reconciliation remain separate controls.
Sources
- Timeouts, retries, and backoff with jitter — Amazon Builders' Library Accessed 4 August 2026
- Idempotent requests — Stripe Accessed 4 August 2026
- The Evolution of Automation at Google — Google, Site Reliability Engineering Accessed 4 August 2026
Design recovery by effect state
Map unknown, absent, partial and complete effects to containment, repair and safe resume.
Design recovery by effect stateUli Prantz
Builds and operates all-agents
Uli Prantz builds all-agents, the process-automation platform this site documents. He writes about the operational side of automating recurring business work: where deterministic code beats model judgment, where it does not, and where a human still has to approve.