Key takeaways
- A completed run is not necessarily a clean run. Corrections, duplicates and harmful effects must remain visible instead of disappearing inside one success percentage.
- The calculator separates clean completion, corrected completion, safe failure and harmful failure, then reports recovery coverage independently.
- Projected monthly incidents are arithmetic based on your observed sample, not a promise about future performance.
- The first recommendation follows visible hard rules: harmful effects, unrecovered failures and duplicate effects outrank improvements to the headline clean-run rate.
Build a reliability profile from one observed window
Use one workflow version and one population over a stated period. Count each started run once in exactly one terminal outcome: clean completion, corrected completion, safe failure or harmful failure. Recovery and duplicate effects are additional observations, not terminal categories.
- Clean-run rate
- 91.0%
- Acceptable completion
- 96.0%
- Safe-failure rate
- 3.6%
- Harmful-failure rate
- 0.4%
- Recovery coverage
- 80.0%
- Duplicate-effect rate
- 0.2%
- Projected harmful / month
- 4.8
- Projected non-clean / month
- 108
- Unclassified runs
- 0
First priority: Contain harmful effects first
Reconcile each escaped effect, isolate its slice and add a hard release gate before optimizing the clean-run rate.
Projections apply observed rates to expected volume. They do not adjust for sample uncertainty, traffic mix changes or workflow updates.
The method keeps consequence and recovery visible
| Result | Formula |
|---|---|
| Clean-run rate | clean completions / started runs |
| Acceptable completion rate | (clean + corrected completions) / started runs |
| Safe-failure rate | safe failures / started runs |
| Harmful-failure rate | harmful failures / started runs |
| Recovery coverage | recovered failures / all observed failures |
| Duplicate-effect rate | duplicate effects / started runs |
| Monthly projection | expected monthly runs x observed rate |
The tool does not produce a composite reliability score. A weighted average could let hundreds of clean runs cancel one unauthorized effect, which is a poor release rule. Harmful effects therefore receive first priority whenever their count is above zero. Duplicate effects come next, then unrecovered failures, unclassified outcomes and a high correction rate.
Google SRE's testing guidance distinguishes evidence from testing and production monitoring. Use the profile as production evidence for one observed version, then build the failed cases into the versioned evaluation set.
Read rates and raw counts together
Of 500 runs, 455 complete cleanly, 25 are corrected before effect, 18 stop safely and two produce a harmful effect. Sixteen of the 20 failures are recovered, and one duplicate effect is observed.
The clean-run rate is 91.0%, acceptable completion is 96.0%, harmful failure is 0.4% and recovery coverage is 80.0%. At 1,200 expected monthly runs, the same observed mix projects 4.8 harmful failures and 108 non-clean runs. Those projections are arithmetic, not a forecast.
The first action is to contain and explain the two harmful effects. Raising clean completion while leaving that slice exposed would improve the headline and leave the material risk untouched.
Add slices and service time outside this calculator
- Slice outcomes: channel, language, action type, consequence, tool path and input shape can reveal a concentrated failure hidden by the total.
- Report uncertainty: show the raw numerator and denominator; rare-event rates from small samples move sharply after one case.
- Time the path: measure p50 and p95 trigger-to-terminal time, approval wait and detection-to-recovery time separately.
- Version everything: start a new observation window when the workflow, model, prompt, rule, tool or approval policy changes materially.
Use How to Make AI Agents Reliable to turn the first priority into a bounded change and release gate.
Limitations and when not to use this
- The calculator does not estimate statistical confidence, adjust for changing traffic mix or prove causation. Preserve counts and compare like-for-like versions and populations.
- “Harmful” must be defined by the accountable process owner before counting. Do not relabel an effect after seeing the rate merely to improve the result.
- This is an educational measurement aid, not certification, a service-level agreement or authorization to automate a consequential decision.
Sources
- Testing for Reliability — Google, Site Reliability Engineering Accessed 4 August 2026
- Monitoring Distributed Systems — Google, Site Reliability Engineering Accessed 4 August 2026
- Evaluation best practices — OpenAI Accessed 4 August 2026
Fix the first reliability gap
Use the measured result to choose a bounded control, evaluation and rollout change.
Fix the first reliability gapUli Prantz
Builds and operates all-agents
Uli Prantz builds all-agents, the process-automation platform this site documents. He writes about the operational side of automating recurring business work: where deterministic code beats model judgment, where it does not, and where a human still has to approve.