AI Agent Evaluation: Test the Decision, the Path, and the Effect

Evaluate the job contract, not the model in isolation

A model benchmark can help compare general capabilities, but an operator needs to know whether one configured system can perform one bounded job. The evaluation unit therefore includes the instructions, retrieved context, tools, permissions, workflow code, approval gates and destination effects — all versioned together.

OpenAI's current evaluation guidance describes task-specific tests, production-shaped datasets, automated scoring where possible, human calibration and continuous evaluation. Its workflow example evaluates each nondeterministic interaction separately. The method here extends that logic to the surrounding deterministic path and real-world effect.

A polished answer can pass the decision layer and fail the path because it queried an unauthorized source. A correct plan can fail the effect layer because a retry sent twice. Do not compress these into one score before diagnosing them.

Build a case set that represents ordinary work and known danger

1. 01Define the objective and disallowed outcomesHuman approval
   Write the job, population, success condition, permitted actions, hard failures and owner. “Helpful” is not a testable objective.
2. 02Stratify the real populationCode
   Measure channel, language, category, customer tier, input length, tool path and consequence class. Sample typical traffic using those distributions.
3. 03Add edges and historic failuresHuman approval
   Include missing fields, conflicting sources, duplicates, timeouts, policy changes, prompt attacks and every corrected production miss that should never recur.
4. 04Create reference decisionsHuman approval
   Have qualified people label criteria, acceptable alternatives and required escalation. Preserve disagreement rather than forcing false certainty.
5. 05Split development and release setsCode
   Tune on one partition and keep the release set hidden from the change process. Version membership, references and source dates.

Synthetic cases are useful for rare boundaries and adversarial inputs, but they must not replace production-shaped data. Track case origin so a high score on generated examples cannot masquerade as live performance.

Protect customer information with purpose limitation, redaction and access controls. A copied production transcript is not automatically safe evaluation data merely because it stays inside an engineering tool.

Match the grader to the type of claim

A model grader should judge one clearly defined criterion and return evidence, not a vague overall preference. Calibrate it against repeated human labels, measure disagreement by slice and keep a human adjudication path. Human review itself also needs a rubric and overlap set; “gold” labels are not gold when qualified reviewers interpret the rule differently.

Set release gates before seeing the candidate result

Define exact hard stops for unauthorized effects, privacy leakage, missed approval gates and duplicate writes. Then define minimum performance by important category, language or source channel, plus allowed regression from the current version. Report confidence intervals or case counts; “95% accurate” without the denominator and composition is not actionable.

Test multiple runs for nondeterministic steps. One pass shows possibility, not consistency. Preserve model/version, parameters, prompts, tool versions, timestamps and raw traces so the result can be reproduced.

Make the release run reproducible from one packet

A result cannot be reproduced from a score alone. Freeze the case manifest, fixtures, candidate, graders and gates together. Each case names its origin and slices, expected decision, permitted tool path, expected effect or required absence of effect, grader and hard-gate status.

Run from a declared starting state, repeat nondeterministic steps the planned number of times, capture raw traces and reconcile the destination. Score decision, path and effect separately before aggregating. The named owner records release, reject or repair-and-rerun; a later edit creates a new version rather than changing the old record.

# AI agent evaluation release packet

Use this packet to make one workflow release reproducible. Replace the simulated
examples with approved, redacted cases from the real operating population. Keep
development cases separate from the held-out release set.

## 1. Version contract

- Workflow/job:
- Candidate version:
- Current version:
- Dataset version and frozen timestamp:
- Instruction, retrieval, tool and policy versions:
- Runtime/model settings:
- Decision owner:
- Evaluators and adjudicator:
- Intended population and excluded populations:
- Permitted effects:
- Hard failures:
- Rollback owner and stop route:

## 2. Case manifest

Create one row per case with these fields:

| Field | Required content |
| --- | --- |
| `case_id` | Stable non-sensitive identifier |
| `partition` | `development`, `release` or `regression` |
| `origin` | Production-shaped, historic failure, edge or adversarial |
| `slices` | Channel, language, category, consequence and other declared slices |
| `input_ref` | Access-controlled pointer to the frozen input fixture |
| `expected_decision` | Allowed label, acceptable alternatives or required abstention |
| `allowed_tool_path` | Permitted tools, order constraints and approval boundary |
| `expected_effect` | Exact destination state or required absence of an effect |
| `graders` | Exact assertions and criterion-level human rubric |
| `hard_gate` | Whether one failure blocks release |

Do not put secrets or raw customer records in the manifest. Point to controlled,
redacted fixtures and document the permitted use.

## 3. Grader contract

For every requirement, record:

1. the claim being tested;
2. the grader type: exact, trace, destination-state or human criterion;
3. the evidence the grader may use;
4. pass, fail and abstain conditions;
5. tolerance, if a numeric comparison legitimately needs one;
6. the adjudication route for disagreement; and
7. the slice-level threshold and whether it is a hard gate.

A model grader should assess one criterion at a time and return the evidence for
its judgment. Calibrate it against overlapping qualified human labels before it
contributes to a release decision.

## 4. Reproducible run order

1. Freeze the candidate, dataset, graders and thresholds.
2. Reset the simulator or test destination to its declared starting state.
3. Execute every release case for the predeclared repeat count.
4. Capture inputs, outputs, traces, tool arguments, approvals and effect receipts.
5. Reconcile the destination state; an accepted tool response alone is not proof.
6. Score decision, path and effect separately.
7. Aggregate by declared slice while retaining case counts.
8. Compare the candidate with the current version on the same frozen cases.
9. Record every hard-gate result before reviewing the average.
10. Have the named owner sign release, reject or return-for-repair.

## 5. Simulated eight-case smoke suite

This is a shape example, not a universal benchmark.

| Case | Condition | Required result | Hard gate |
| --- | --- | --- | --- |
| S01 | Ordinary valid request | Allowed decision and one verified effect | No |
| S02 | Required field missing | Abstain and request the named evidence | Yes |
| S03 | Sources conflict | No write; route to the declared owner | Yes |
| S04 | Duplicate delivery | Reuse the business key; no duplicate effect | Yes |
| S05 | Tool timeout before receipt | Reconcile before any retry | Yes |
| S06 | Approval required | No write before approved version exists | Yes |
| S07 | Disallowed instruction in retrieved text | Ignore it and remain in permission envelope | Yes |
| S08 | Valid edge category | Acceptable decision with correct slice label | No |

## 6. Release record

- Run identifier and timestamp:
- Case count and repeat count:
- Results by decision, path and effect:
- Results by declared slice:
- Current-version comparison:
- Hard-gate failures:
- Human/model grader disagreement:
- Known limitations:
- Decision: release / reject / repair and rerun
- Decision owner and date:
- Canary scope and stop rule:
- Regression cases created after repair:

Passing this packet supports only the tested workflow, version, population and
effect boundary. It is not a certification of a model or a substitute for the
specialist review required by consequential work.

Evaluation continues after release without grading customers in secret

• Shadow first: compare proposed decisions with the live path without taking the external effect.
• Canary narrowly: release to a bounded, reversible slice with a stop rule and named owner.
• Sample continuously: review random cases and high-risk triggers, separating their statistics.
• Detect drift: watch input mix, abstention, corrections, tool errors, latency and slice-level quality.
• Close the loop: preserve a corrected miss as a versioned regression case, then prove the fix against cases that already worked.

The evaluation record should name the workflow version, dataset version, graders, thresholds, results, failures, decision owner and release outcome. This is the evidence behind “we tested it,” not a dashboard screenshot.

Use AI agent failure recovery to design what happens when a production run crosses one of the boundaries the evaluation was meant to protect.