Key takeaways
- Recovery begins by stopping new effects and preserving the run, not by asking the model to try the whole task again.
- Classify the failure by what happened in the outside world: no effect, confirmed effect, ambiguous effect or harmful effect. Each needs a different path.
- Retry only transient, idempotent operations with a budget and jitter. Never retry an ambiguous side effect until the destination has been reconciled.
- Use rollback where state can truly be restored; use a named compensating action where it cannot, and require human approval for consequential repair.
- Resume from the last verified checkpoint with the same business key and updated preconditions, then add the case to the regression set.
Classify the failure by its external effect before choosing a retry
“The agent failed” says nothing about recovery. The model may have returned invalid data before any action, a destination may have rejected a write, or the network may have timed out after the write succeeded. Those cases look similar in the orchestration log and require different responses.
Google's SRE guidance warns that retries can amplify failures and recommends bounded backoff with jitter; its incident guidance prioritizes stopping damage, restoring service and preserving evidence. Apply those principles at the workflow-run level.
| Effect state | Meaning | Allowed next move |
|---|---|---|
| No effect | Failure occurred before a destination accepted the action | Correct input or retry a transient step within budget |
| Confirmed effect | Destination proves the intended change landed once | Advance from the verified checkpoint; do not repeat |
| Ambiguous effect | Timeout or partial response leaves destination state unknown | Reconcile by business key; block retry until known |
| Wrong reversible effect | State changed incorrectly and can be restored completely | Authorized rollback, then verify restoration |
| Wrong irreversible effect | Message sent, disclosure made or action cannot be erased | Contain, notify, compensate where possible and require human incident ownership |
Use a fixed recovery sequence when the run path was variable
- 01ContainCodePause the run, stop queued effects for the same scope, trip the relevant circuit breaker and preserve the last known state. Do not erase the failed attempt.
- 02Preserve and notifyTriggerSnapshot instructions, inputs, model and tool versions, trace, approvals, idempotency keys, destination responses and timestamps. Alert the owner according to consequence and time sensitivity.
- 03Reconcile effectsCodeQuery destinations using stable business keys. Classify each planned action as absent, present once, duplicated, partially applied or unknown.
- 04Choose repairHuman approvalFor consequential cases, a named owner chooses retry, rollback, compensation, manual completion or abandonment and records why.
- 05Execute repair exactly onceCodeUse a new repair ID linked to the original run while retaining the original business key. Check authorization and current preconditions before changing anything.
- 06Verify and resumeHuman approvalConfirm postconditions, then resume from the last verified checkpoint. Re-read mutable source state; do not trust the stale snapshot as current truth.
- 07LearnCodeCreate a regression case, assign a systemic fix and track it to verified closure. Keep the incident record linked to the test.
A retry needs an eligibility rule, budget, and business key
| Field | Example | Reason |
|---|---|---|
| Eligible errors | 429, 502, connection reset before acceptance | Business or validation errors do not improve with repetition |
| Maximum attempts | 3 total | Prevents infinite work and overload |
| Backoff | Exponential with random jitter | Avoids synchronized retry storms |
| Time budget | 90 seconds or before customer deadline | A technically successful late retry may still be wrong |
| Idempotency key | ticket-4421:eng-handoff:v2 | Connects attempts to one intended effect |
| Reconciliation query | Lookup by external reference or business key | Resolves a timeout-after-success ambiguity |
| Exhausted path | Owned recovery queue with severity clock | “Failed” is not a finish condition |
Retry the smallest safe step, not the entire agent plan. If classification succeeded and only a read-only lookup failed, repeat the lookup. Replanning can choose a different path and invalidate the assumptions behind prior approvals.
Circuit breakers and load shedding protect dependencies as well as the workflow. When a destination is failing broadly, adding autonomous retries can turn a small outage into a cascading one. Pause, degrade to a manual queue, or accept less work until the dependency recovers.
Rollback restores state; compensation addresses what cannot be erased
Call an action a rollback only when it returns the system to the verified prior state without leaving a material effect. Deleting a duplicate internal draft may qualify. Sending a correction after a wrong customer email does not; the first message was still read. That is compensation and incident response.
| Failed effect | Repair pattern | Approval |
|---|---|---|
| Incorrect internal tag | Restore previous tag and verify queue state | Automatic if rule and scope are fixed |
| Duplicate internal record | Merge or retire duplicate; preserve cross-reference | Human if records have downstream work |
| Wrong permission change | Contain access, restore known-good permissions, investigate exposure | Security owner |
| Incorrect customer message | Stop follow-ups, approve a correction and preserve both messages | Customer owner |
| Money or contract effect | Use the qualified finance/legal incident and compensation process | Permanent accountable human gate |
Worked example: timeout after creating an engineering ticket
ticket-4421:handoff:v2. The client times out and the orchestrator records an ambiguous effect. The workflow pauses rather than retrying. Reconciliation queries the destination by external reference and finds ENG-944 created once with all required fields. The run records a confirmed effect, repairs only the missing reciprocal link in the support ticket using repair ID RP-102, verifies both links and resumes at “await engineering acknowledgement.” The timeout becomes a regression case that asserts one engineering item and two reciprocal references after repeated client timeouts.Without reconciliation, a whole-task retry could have created ENG-945 and paged a second team. The safest recovery did not ask a model what probably happened; it inspected the destination.
A workflow is not recoverable until its recovery path is exercised
- Every external effect has a business key, reconciliation query and owner.
- Every step states whether it is retryable, reversible, compensatable or human-only.
- Checkpoints record verified postconditions, not merely “step completed.”
- Repair permissions are narrower than ordinary execution where possible and fully audited.
- The team regularly tests timeouts, partial failure, destination outage, stale approval and rollback failure.
- Recovery measures include containment time, ambiguous-effect duration, duplicate effects, successful compensation, resume success and repeat incidence.
Preserve the failure and repair as an append-only chain: original intent, authority, events, effect reconciliation, decision, repair action, verification and regression-test reference. The AI agent audit trail guide specifies the event envelope behind that chain.
Limitations and when not to use this
- This is a general workflow-recovery pattern, not an incident plan for safety, security, privacy, financial, legal or regulated events. Use qualified owners and the applicable response process.
- Idempotency and rollback depend on the actual destination API and business semantics. Test them against current systems; a client-generated key alone does not guarantee deduplication.
- Do not allow a model to authorize its own repair, infer that an external effect occurred, or decide compensation for a person. Reconcile facts and keep accountable human gates.
Sources
- Managing incidents — Google Site Reliability Engineering Accessed 3 August 2026
- Addressing cascading failures — Google Site Reliability Engineering Accessed 3 August 2026
- Service best practices — Google Site Reliability Engineering Accessed 3 August 2026
Map the recovery path
Add checkpoints, effect verification, retry budgets, compensation and escalation to one live workflow.
Map the recovery pathUli Prantz
Builds and operates all-agents
Uli Prantz builds all-agents, the process-automation platform this site documents. He writes about the operational side of automating recurring business work: where deterministic code beats model judgment, where it does not, and where a human still has to approve.