Key takeaways
- Decide what failure rate each step can live with before you build it. Chasing zero on every step costs more than it returns, and it distracts from the few steps where zero really is the target.
- Most unreliability is variance you chose to keep. Every step still decided fresh on each run is a step that can answer differently tomorrow, so convert the ones you now understand into fixed code and the variance goes with them.
- Assume every step will be retried. Give anything with a side effect a stable key drawn from the business identity of the work, so a repeat is a no-op instead of a second payment, email or ledger entry.
- Design for detection, not only prevention. A recorded, replayable run is what turns a vague complaint into a defect you can diagnose and a fix you can test against the cases that already worked.
- An approval that is always approved is not a control. Track how often the reviewer changes something; if that number is near zero, the gate is in the wrong place or the reviewer cannot see what they are approving.
What reliability means when a model is doing the work
Ordinary software is boringly repeatable. The same input takes the same path, so a bug reproduces and a test that passed on Monday still passes on Thursday. Put a model inside a step and that guarantee goes away for that step: two identical inputs can produce two different answers, both plausible, one of them wrong. Reliability stops being a property you have or lack, and becomes a rate you manage.
Google's site reliability engineers put the point bluntly: “100% is the wrong reliability target for basically everything”. That is not permission to be sloppy but an argument about where to spend: past a point, each additional nine costs more than it returns, and the money does more good on detecting and recovering from the failures you accepted.
The target belongs to the step, not to the process. A misfiled category and a payment to the wrong account are not one event at two severities. A few steps really do have a target of zero, and the way you get there is never to try harder: it is to stop letting that step vary, or to keep a person answerable for it. Four properties get bundled together whenever somebody calls an agent unreliable, and each is fixed differently.
- Correct. The run produced the outcome the procedure specifies for that case.
- Repeatable. The same case run next Tuesday produces the same outcome, and you can say in advance which parts of the process guarantee that.
- Detectable. When a run is wrong, somebody finds out inside a known window, from your own checks rather than from a complaint.
- Recoverable. You can see what the run did, and undo or redo it without archaeology.
Teams who call their automation untrustworthy rarely mean the first one. They mean the second and third.
Six ways an automated process quietly becomes untrustworthy
The loud failures are the easy ones. A crashed run, an expired credential, a service returning 500 — these announce themselves, somebody retries, life goes on. The failures that destroy trust are quiet: the run reports success and the damage surfaces days later in a reconciliation or an audit. Read the middle column first, because recognising the symptom comes before naming the cause.
| Failure mode | What it looks like | What to do about it |
|---|---|---|
| Non-determinism | The same invoice, submitted twice, is categorised two different ways. Nobody changed anything, your test case passes, and you cannot reproduce the bad run on demand. | Stop asking a model to decide what a rule can decide. Once you have seen enough real cases to state the rule in a sentence, move the step to fixed code and leave the model the residue. Where the judgment is genuine, constrain it to a fixed set of allowed answers. |
| Silent partial failure | Four of the five systems were updated. Nothing threw an error, so the run is marked complete, and the fifth stays quietly out of date until somebody reconciles it. | Define a finish condition for the whole process and check it, instead of trusting each step's return code. Assert post-conditions — the record exists in both systems and the amounts match — and fail loudly when they do not hold. |
| Duplicate side effects on retry | A step times out after its action has already landed. The retry does it again: the same email twice, the same ledger entry twice, the same refund twice. The logs look healthy, because both attempts succeeded. | Assume retries; you cannot design them away. Give every side-effecting step a key drawn from the business identity of the work — invoice number, ticket id, pay period — and check before acting, so a repeat is a no-op. |
| Upstream drift | A vendor redesigns a form, a report gains a column, a field is renamed. The step that read the third column now reads something else, and carries on without complaint. | Validate the shape of what you read before acting on it, and treat a shape change as an exception rather than as a value. This is the failure that most deserves a human: the process should stop, show what changed, and ask. |
| Prompt and context drift | Instructions accumulate. Somebody adds a clause to fix one edge case, then another. Three months on, the guidance contradicts itself and behaviour shifts on cases nobody touched. | Keep the written procedure, not the prompt, as the source of truth, version it, and check every change against the runs you already accepted. A change that fixes today's case and breaks one from March is not a fix. |
| Rubber-stamp approvals | The gate fires, the reviewer approves, and the rate at which anything is actually changed is effectively zero. Ask what the last two hundred approvals were for and nobody can say. | Measure the edit rate, not the approval count, and keep gates few, high-consequence and legible. The AI Act's human-oversight article exists partly because over-reliance on system output is the predictable way a human check fails. |
Notice what the right-hand column has in common. Not one of these fixes is “use a better model”. Five of the six are properties of the process built around it: where the rules live, what gets checked, what happens on a retry, who is asked.
The reliable path is to shrink the surface that can vary
A step that runs on model judgment behaves like a distribution rather than a value. Sometimes that is what you want, and it is why you reached for a model on a job with messy inputs. But it is a cost paid on every future run: latency, money, and one more place the answer can move.
So the reliability question is not how to make the model more consistent. It is how much of this process still needs deciding at all — usually far less than it looked like at the start. A step seems open-ended for the first few runs because you have not yet seen enough cases to state the rule. Once you have, keeping a model on it means paying every week to re-derive what you already know.
Google's SRE book makes the case for automating operational work first on the grounds of consistency: a machine performs the same actions in the same order every time, which people demonstrably do not. The same chapter is candid about the other edge of that blade, since automation applies a mistake as faithfully as a correct rule, and at the same scale. That is why the checks and records below are not optional extras, and it gives the shape of a process that gets more reliable over time — the model all-agents is built around.
- 01Run the job with the model, and watchAI judgmentThe first runs execute with an agent's flexibility, because you do not yet know the shape of the job. Every accepted run is kept as an example: the actual inputs, the path taken, the decisions, your corrections.
- 02Approve everything irreversible while you are learningHuman approvalBefore anything leaves the building — an email sent, an external system written to — the run stops and asks. The gate does two jobs: it keeps mistakes inside, and it produces the corrections the written procedure learns from. A run you had to correct teaches more than a clean one.
- 03Decide it is ready on coverage, not on a countHuman approvalReadiness is about which branches you have actually seen — the foreign-currency invoice, the record with no email address — and how complete the procedure has become. A run count is a nudge, not a rule. The owner makes the call.
- 04Walk the procedure through and pin what has to stayHuman approvalCodification is a walk-through, not a silent compile. Going step by step against real example data surfaces the unwritten rules nobody thought to mention, and it is where you say “this step needs judgment every time” or “this checkpoint always needs a person”. What surfaces goes back into the procedure, not just the code.
- 05Run it as code, with judgment only where it is irreducibleCodeThe trigger fires and the process runs the same way every time, at a fraction of the cost, many cases in parallel. Agent steps survive only where the variety cannot be enumerated, pinned gates still stop for a person, and every run is recorded and replayable.
- 06When reality moves, the human comes back into the loopHuman approvalA codified process must not rot silently. A failure is reviewed and classified: a small change becomes a proposed patch, checked against every accepted example so it cannot quietly break what worked. A broken assumption sends the process back to supervised learning, while inbound work queues instead of being dropped.
What matters in that picture is the direction of travel. The varying surface should shrink monotonically: every answered question and every pinned rule takes one more decision out of the model's hands for good. If six months in the same proportion of your process is still decided fresh on each run, nothing has been learned, and reliability will not improve either.
Deciding step by step: fixed code, model judgment, or a human gate
Make this decision on paper, per step, before anything is built. Two questions settle most of it: can you state the rule without saying “it depends”, and what does a wrong answer cost? The first decides code versus judgment; the second decides whether a person stays on the hook regardless.
| Character of the step | Give it to | The test that settles it |
|---|---|---|
| You can write the rule as a sentence with no “it depends” in it. | Fixed code | Two colleagues applying your sentence to ten past cases agree on all ten. |
| The rule holds for the bulk of cases, but there is a tail you cannot enumerate. | Code for the body, model for the tail | You can describe the common cases exactly and are only guessing at the rest. |
| The input is free text, an image, or something a person wrote, and it genuinely varies. | Model judgment, with a constrained output | You cannot list the inputs in advance, only the answers you are willing to accept. |
| The action is irreversible, and a wrong one costs real money or affects a specific person. | A permanent human gate | Ask who would be named in an audit. If it is a person, that person belongs at the moment it happens. |
| The action is reversible, cheap to undo, and happens many times a day. | Fixed code, with a check afterwards | You can undo it without contacting anyone outside the team. |
| Nobody yet agrees on what a correct outcome looks like. | Nobody, yet — keep it manual and write it down | You cannot describe the failure case, which means you could not write a test for it. |
Running an existing process through this table is usually uncomfortable in a useful way: more steps belong in the first row than are currently there, and one or two steps nobody gated belong in the fourth. Each mistake has its own cost: too much code and the tail gets mishandled silently, too much model and you pay to re-derive rules you already know, too many gates and approval fatigue turns all of them into rubber stamps. If you cannot fill in the third column at all, that is the real finding — the procedure is not written down well enough yet, and writing it down is the work that unblocks everything else.
Worked example: the retry that posted the same invoice twice
Failure modes read as abstractions until you follow one through. Here is a single case in the shape these incidents usually take. The figures are illustrative.
The process. Twice a week, an automated process reads approved supplier invoices from a shared mailbox, posts each to the ledger, files the PDF and notifies the approver.
The run that went wrong. The ledger call for one invoice took longer than the step's timeout. The posting had succeeded on the far side; only the response was lost. The retry policy did what retry policies do, and the invoice was posted a second time. Both attempts recorded success, the run finished green, and nothing looked wrong.
How it surfaced. Nine days later, at month-end, the supplier ledger was over by the value of one invoice. The only clue was two entries with different internal ids, the same supplier and amount, and timestamps ninety seconds apart.
The wrong fixes. Raising the timeout turns a frequent duplicate into a rare one and leaves the defect in place. Removing the retry trades duplicates for dropped invoices, which is worse: a missing entry is harder to spot than a doubled one.
The actual fix, in three parts.
- An identity key. The posting step derives a key from the business identity of the work — supplier id, invoice number, accounting period — and checks for an entry under it before writing. The retry now finds the first posting and returns it.
- A post-condition. The run asserts at the end that exactly one ledger entry exists per invoice handled. With that check it would have failed loudly on the day, instead of passing quietly for nine of them.
- A regression case. The recorded run became a test: the timeout is replayed against the fixed process, and the other accepted runs are replayed to confirm nothing else changed.
The ratio that matters. The defect lived nine days and cost a journal correction plus half a day of reconstruction. The post-condition that would have caught it on day one was two lines.
Two things generalise. The incident was not a model failure at all: the defect was in how the process handled a retry. Reliability work here is mostly ordinary engineering discipline applied to a system that happens to contain a model. And the expensive part was not the bug but the nine days — the gap between the failure and anybody knowing about it.
Proving it works: records, replay, and the rubber-stamp trap
You cannot manage a rate you cannot see. The highest-return investment in an automated process is a complete record of what each run did: the inputs, the path taken, each decision and who or what made it, each approval, and every action in an external system. It is cheap to keep and impossible to reconstruct afterwards, which is why our own runs are recorded and replayable by default.
There is a regulatory floor worth knowing even if it does not yet bind you. The EU AI Act requires high-risk systems to keep automatic logs across their lifetime. Whether or not your process falls in that category, that standard describes what anyone investigating an incident needs.
The record pays for itself twice: once when something goes wrong, and again every time you change the process, because a recorded run can be replayed against a proposed fix. Human gates need the same discipline, and this is where governance-by-intention falls apart: an approval granted every time is not a control, it is a delay with a signature. Five numbers are worth watching.
- Edit rate at each gate. The share of approvals where the reviewer changed, rejected or questioned something. Near zero means the gate is in the wrong place, or the reviewer cannot see what they are approving.
- Exception rate, and its composition. A rising rate is a warning. The same exception recurring is something better: a rule waiting to be written into the procedure.
- Share of steps still decided at run time. This should fall as the process matures. If it does not, the learning loop is not closing.
- Time to detect. The gap between a bad run and somebody knowing. Usually a team's worst number, and usually the cheapest to improve.
- Rework. How often a finished run had to be redone by hand. The people doing the work already know this number, and it decides whether they trust the automation.
None of this makes a process perfect, and aiming for perfect is how teams spend their whole budget on the wrong steps. Pick the failure rate each step can live with, remove the variance you were never getting value from, keep a person answerable where somebody has to be, and make sure that when the remaining failures happen, you are the one who finds out. That is a process you can defend — the practical meaning of one you can trust.
Limitations and when not to use this
- This page is about work that repeats. A one-off analysis or a genuinely novel task has no stable shape to convert into code, so the argument about shrinking the varying surface does not apply — there you are paying for judgment because judgment is the product.
- Determinism is not correctness. Fixed code applies the rule you gave it exactly, including when that rule is wrong or has gone out of date. Everything here assumes a named person still owns the question of what correct means.
- Nothing here is legal advice. We cite the AI Act because its record-keeping and oversight articles are a reasonable design floor for any process that acts on its own, not because we can tell you whether your system is high-risk under it.
- If the process changes shape every month, do not automate it yet: you will spend more on re-specifying it than you bank in saved effort. Write the procedure down, run it by hand until it settles, and come back. The same goes for a process nobody currently owns.
Sources
- Embracing Risk — Google, Site Reliability Engineering Accessed 24 July 2026
- Automation at Google — Google, Site Reliability Engineering Accessed 24 July 2026
- Article 12: Record-keeping — EU Artificial Intelligence Act Accessed 24 July 2026
- Article 14: Human oversight — EU Artificial Intelligence Act Accessed 24 July 2026
Design the approvals that stay
Two of the six failure modes are really approval design problems. This guide sorts gates by risk and shows the patterns for each tier, including how to stop a gate becoming a rubber stamp.
Design the approvals that stayUli Prantz
Builds and operates all-agents
Uli Prantz builds all-agents, the process-automation platform this site documents. He writes about the operational side of automating recurring business work: where deterministic code beats model judgment, where it does not, and where a human still has to approve.