Key takeaways
- Evaluate the business decision and permitted effect, not whether the answer sounds polished. A good response that takes an unauthorized action still fails.
- Build a versioned case set from ordinary traffic, edge cases, known failures and adversarial inputs; preserve a held-out release set.
- Score deterministic checks with exact assertions, model judgment with criterion-level human labels, and agent paths with trace and side-effect checks.
- Set slice-level gates before the run. An average can pass while one language, channel or high-consequence case fails completely.
- Every corrected production miss should become a regression case, but keep test examples separate from the material used to tune the system.
Evaluate the job contract, not the model in isolation
A model benchmark can help compare general capabilities, but an operator needs to know whether one configured system can perform one bounded job. The evaluation unit therefore includes the instructions, retrieved context, tools, permissions, workflow code, approval gates and destination effects — all versioned together.
OpenAI's current evaluation guidance describes task-specific tests, production-shaped datasets, automated scoring where possible, human calibration and continuous evaluation. Its workflow example evaluates each nondeterministic interaction separately. The method here extends that logic to the surrounding deterministic path and real-world effect.
| Layer | Question | Best evidence |
|---|---|---|
| Input contract | Did the run receive valid, permitted and current information? | Schema, provenance, access and freshness assertions |
| Decision | Was the classification, extraction or recommendation acceptable? | Reference labels and criterion-level human judgment |
| Path | Did it call permitted tools in an acceptable order and stop at gates? | Trace events, tool arguments, budgets and state transitions |
| Effect | Did the outside system change exactly as authorized? | Destination reconciliation, idempotency and absence-of-effect checks |
A polished answer can pass the decision layer and fail the path because it queried an unauthorized source. A correct plan can fail the effect layer because a retry sent twice. Do not compress these into one score before diagnosing them.
Build a case set that represents ordinary work and known danger
- 01Define the objective and disallowed outcomesHuman approvalWrite the job, population, success condition, permitted actions, hard failures and owner. “Helpful” is not a testable objective.
- 02Stratify the real populationCodeMeasure channel, language, category, customer tier, input length, tool path and consequence class. Sample typical traffic using those distributions.
- 03Add edges and historic failuresHuman approvalInclude missing fields, conflicting sources, duplicates, timeouts, policy changes, prompt attacks and every corrected production miss that should never recur.
- 04Create reference decisionsHuman approvalHave qualified people label criteria, acceptable alternatives and required escalation. Preserve disagreement rather than forcing false certainty.
- 05Split development and release setsCodeTune on one partition and keep the release set hidden from the change process. Version membership, references and source dates.
Synthetic cases are useful for rare boundaries and adversarial inputs, but they must not replace production-shaped data. Track case origin so a high score on generated examples cannot masquerade as live performance.
Protect customer information with purpose limitation, redaction and access controls. A copied production transcript is not automatically safe evaluation data merely because it stays inside an engineering tool.
Match the grader to the type of claim
| Requirement | Grader | Failure example |
|---|---|---|
| Schema and allowed values | Exact validation | Output adds an unsupported category |
| Numeric or source-grounded fact | Source comparison with tolerance or exact match | Quoted amount differs from system of record |
| Policy criterion | Human-labeled criterion; calibrated model grader may assist | Explanation omits a required limitation |
| Tool path and permission | Trace assertion | Write tool called before approval |
| External side effect | Destination-state assertion | Two records created for one business key |
| Safe abstention | Required state transition | System guesses when evidence conflicts |
A model grader should judge one clearly defined criterion and return evidence, not a vague overall preference. Calibrate it against repeated human labels, measure disagreement by slice and keep a human adjudication path. Human review itself also needs a rubric and overlap set; “gold” labels are not gold when qualified reviewers interpret the rule differently.
Set release gates before seeing the candidate result
Define exact hard stops for unauthorized effects, privacy leakage, missed approval gates and duplicate writes. Then define minimum performance by important category, language or source channel, plus allowed regression from the current version. Report confidence intervals or case counts; “95% accurate” without the denominator and composition is not actionable.
Test multiple runs for nondeterministic steps. One pass shows possibility, not consistency. Preserve model/version, parameters, prompts, tool versions, timestamps and raw traces so the result can be reproduced.
Evaluation continues after release without grading customers in secret
- Shadow first: compare proposed decisions with the live path without taking the external effect.
- Canary narrowly: release to a bounded, reversible slice with a stop rule and named owner.
- Sample continuously: review random cases and high-risk triggers, separating their statistics.
- Detect drift: watch input mix, abstention, corrections, tool errors, latency and slice-level quality.
- Close the loop: preserve a corrected miss as a versioned regression case, then prove the fix against cases that already worked.
The evaluation record should name the workflow version, dataset version, graders, thresholds, results, failures, decision owner and release outcome. This is the evidence behind “we tested it,” not a dashboard screenshot.
Use AI agent failure recovery to design what happens when a production run crosses one of the boundaries the evaluation was meant to protect.
Limitations and when not to use this
- Passing an evaluation supports a bounded release decision for the tested workflow, population and version. It is not certification of a model or proof of performance in every context.
- Benchmarks, synthetic cases and model-based graders can complement but not replace representative local data and qualified human judgment for consequential work.
- Do not use evaluation results to make automated decisions about workers or customers. Define separate qualified review and governance for any individual consequence.
Sources
- Evaluation best practices — OpenAI Accessed 3 August 2026
- Working with evals — OpenAI Accessed 3 August 2026
- Artificial Intelligence Risk Management Framework 1.0 — NIST Accessed 3 August 2026
Inventory the variable steps
Classify one workflow step and export the minimum test and control contract for it.
Inventory the variable stepsUli Prantz
Builds and operates all-agents
Uli Prantz builds all-agents, the process-automation platform this site documents. He writes about the operational side of automating recurring business work: where deterministic code beats model judgment, where it does not, and where a human still has to approve.