Key takeaways
- Retry repeats an eligible step; it does not repair a wrong instruction or changed process.
- Fallback limits impact and keeps work moving, but leaves the root cause unresolved.
- Reconciliation checks what actually happened before any side effect is repeated.
- A repair should be versioned, regression-tested, and confirmed before changed behavior is released.
- When the real process changed, return to supervised learning rather than patching around a new rule.
A self-healing business workflow should detect and contain a failure, determine what happened, and choose among bounded retry, fallback, reconciliation, repair, or relearning. Retry repeats an eligible transient step. Fallback reduces impact. Reconciliation verifies external state. Repair changes implementation. Relearning changes the encoded understanding of the process. Consequential behavior changes should be versioned, regression-tested, and confirmed by an accountable owner rather than applied invisibly inside the failed run.
Use five mechanisms instead of one marketing phrase
| Mechanism | It does | It does not do |
|---|---|---|
| Retry | Repeats an eligible transient step within a budget | Fix wrong logic, changed policy, or an ambiguous side effect |
| Fallback | Uses a degraded, manual, or alternate path to contain impact | Remove the root cause |
| Reconciliation | Queries source and destination to establish what actually happened | Guess from an incomplete client response |
| Repair | Changes implementation when intended behavior is known but code or integration is wrong | Redefine the business process silently |
| Relearning | Returns changed reality to supervised examples and process decisions | Pretend a new policy is a software bug |
For an internal tool, the healing claim is credible only if the owner can see which mechanism occurred, what changed, what was tested, and who approved it.
Recover in a fixed sequence
- 01ContainCodeStop further effects for the affected scope, preserve state, and keep the failed attempt.
- 02Classify the effectTriggerDetermine no effect, confirmed effect, ambiguous effect, reversible wrong effect, or irreversible harmful effect.
- 03ReconcileCodeUse stable business keys to inspect destinations before any retry that could duplicate an action.
- 04Choose the mechanismHuman approvalApply retry or fallback under policy; route implementation repair, process relearning, rollback, or compensation to the accountable owner.
- 05Test changed behaviorAI judgmentPropose a bounded version and replay known normal, boundary, exception, and failure cases.
- 06Confirm and resumeHuman approvalApprove the changed version, verify current preconditions, then resume from the last verified checkpoint.
A retry needs eligibility, identity, and a budget
Google SRE guidance on cascading failures warns that retries can amplify failure and recommends bounded backoff with jitter. In workflows, a retry also needs a stable business identity and an exhausted path.
| Field | Example |
|---|---|
| Eligible error | Rate limit or connection failure before confirmed acceptance |
| Attempt budget | Three total attempts inside 90 seconds |
| Backoff | Exponential with random jitter |
| Business key | request-4421:approval:v3 |
| Reconciliation | Lookup the destination by external reference |
| Exhausted path | Owned exception queue with severity and clock |
Repair implementation; relearn changed reality
A workflow rejects a new document layout. The run stops before updating the destination and opens review. The owner confirms the policy and expected fields are unchanged, so this is implementation repair, not relearning. A proposed parser change is versioned and replayed against the failed document plus the existing normal, missing-field, and conflicting-value cases. The owner reviews the diff and results, approves release, and the run resumes from the verified pre-action checkpoint. If the document introduced a new business category instead, the path would return to supervised learning and policy ownership.
Evaluate self-healing claims with four questions
- Who can see the original failure, external-effect state, and complete evidence?
- What is allowed to change automatically: attempts, routing, configuration, code, instructions, policy, or authority?
- Which representative and regression cases run before the changed version is released?
- Who confirms consequential repair, relearning, rollback, compensation, and resume?
The detailed AI agent failure recovery guide covers checkpoints, idempotency, rollback, compensation, and effect reconciliation.
Limitations and when not to use this
- This business-workflow taxonomy is not a substitute for infrastructure remediation, incident response, safety systems, or regulated recovery procedures.
- Idempotency and rollback depend on destination semantics and must be tested; a client-generated key or “undo” label is not proof.
- The all-agents product boundary described here stops failed execution and requires owner confirmation for changed behavior. It does not claim inline autonomous self-modification.
Sources
- Managing incidents — Google Site Reliability Engineering Accessed 14 August 2026
- Addressing cascading failures — Google Site Reliability Engineering Accessed 14 August 2026
- Testing for Reliability — Google Site Reliability Engineering Accessed 14 August 2026
Measure the recovery gap
Separate clean runs, corrected runs, safe failures, harmful failures, and recovered failures for one recent operating period.
Measure the recovery gapUli Prantz
Builds and operates all-agents
Uli Prantz builds all-agents, the process-automation platform this site documents. He writes about the operational side of automating recurring business work: where deterministic code beats model judgment, where it does not, and where a human still has to approve.