Key takeaways
- Measure cost and elapsed time for the whole business run, including tool calls, retries, approval waits and recovery, rather than one model request.
- Separate the critical path from work that can run in parallel or after the user-visible result. Removing a sequential round trip often beats shaving milliseconds from every call.
- Route settled extraction, validation and formatting to deterministic code; reserve higher-cost model work for decisions that show a measured quality gain.
- Set budgets and stop rules before execution. An agent that can keep searching, retrying or reflecting has unbounded tail cost and latency.
- Optimize against the same evaluation set. A cheaper or faster configuration that crosses a quality or safety gate is a failed optimization.
Price and time the whole run
A model request that costs cents can sit inside a run that costs much more. A search tool may charge per call. A timeout may trigger two more calls. A reviewer may wait six minutes for the evidence packet. A malformed effect may take an hour to repair. Put every one of those under the same run identifier or the optimization target will be wrong.
| Entry | Cost field | Time field |
|---|---|---|
| Model interaction | Model, input, cached input, output and reasoning usage | Queue, first token and completion |
| Tool call | Provider charge and internal compute | Start, timeout and response |
| Retry | Repeated model and tool spend | Backoff plus new attempt |
| Human gate | Reviewer minutes | Queue wait and handling time |
| Effect | Write and reconciliation calls | Proposal-to-verified-state time |
| Recovery | Repair calls and operator time | Detection-to-restored-state time |
Report median and tail latency. An average hides the small group of runs that hit multiple tool rounds or wait at an approval gate. Separate machine execution from human queue time; buying a faster model cannot repair a reviewer queue.
Draw the critical path before optimizing a call
- 01Remove work that should not happenHuman approvalDelete duplicate context, unused tools, ornamental summaries and model checks that a validator can perform exactly.
- 02Move stable logic into codeCodeUse deterministic extraction, lookup, policy and formatting where accepted examples now reveal a fixed rule.
- 03Parallelize independent readsCodeFetch sources that do not depend on each other together, then join their validated results before the decision.
- 04Choose the smallest sufficient configurationHuman approvalCompare model and reasoning settings on representative cases. Keep the least expensive option that clears the same slice gates.
- 05Bound the runCodeSet call, token, tool, retry and elapsed-time budgets. Enter a named exception state when a budget expires.
OpenAI's current latency guide prioritizes fewer requests, fewer generated tokens, parallel work and making the user wait only for what is necessary. That ordering matters: removing a full sequential round trip is usually more valuable than tuning a request that should not be on the critical path.
Cache repeated context, route by task and stop deliberately
Place stable instructions and examples before run-specific data so providers that support prefix caching can reuse them. OpenAI's prompt-caching documentation exposes cached token usage, which lets a team verify the saving instead of assuming it. A cache is an optimization; version the prompt and rerun evaluations when the prefix changes.
Route by the job's measured difficulty. A simple allowed-label classification need not share a configuration with an ambiguous exception analysis. Do not route using a model's confidence claim alone. Use observable input features, hard-risk slices and a fallback that enters review rather than silently accepting a weak result.
| Change | Expected gain | Release proof |
|---|---|---|
| Remove one model step | Lower cost, latency and variance | Exact code check matches accepted cases |
| Use a smaller model | Lower per-call spend and often lower latency | Same held-out gates by important slice |
| Shorten output | Lower generation time and output spend | Required fields and evidence remain complete |
| Cache prompt prefix | Lower repeated input work | Cache usage is observed; version invalidation works |
| Parallelize tools | Shorter critical path | Reads are independent and join validation catches partial failure |
| Lower attempt budget | Lower tail cost and time | Transient recovery remains above its gate |
A run ledger turns a vague complaint into three fixes
A weekly reporting run takes a median 84 seconds and p95 226 seconds. The trace shows four sequential data reads, two model calls, and an automatic retry of the whole run after any timeout. The first model call converts fixed column names into a schema; the second writes a short exception note.
The team replaces the schema conversion with code, runs the four independent reads in parallel and retries only the failed read. The exception note stays on the same model because evaluation shows a meaningful quality loss on the cheaper configuration. Median time falls to 31 seconds and p95 to 74 seconds in the simulated comparison; total model usage drops because one call disappeared.
The numbers are illustrative. The method is the asset: trace, change one cause, rerun the same cases, then compare cost, median, tail and quality together.
Use the automation ROI calculator only after the operating ledger is measured. Its monthly automation cost input should come from completed runs, not a model price copied from a pricing table.
Limitations and when not to use this
- Provider prices, model behavior and caching rules change. Recheck current primary documentation and store the effective price and model version with each measurement.
- Faster is not better when it removes evidence, skips an approval or lowers a hard-risk slice below its gate.
- Queueing, network locality, provider rate limits and human review can dominate latency. A model-only benchmark does not predict end-to-end service time.
Sources
- Latency optimization — OpenAI Accessed 4 August 2026
- Prompt caching — OpenAI Accessed 4 August 2026
- Models — OpenAI Accessed 4 August 2026
- The Evolution of Automation at Google — Google, Site Reliability Engineering Accessed 4 August 2026
Build a run ledger
Instrument the path before optimizing it so quality, effect and recovery remain visible.
Build a run ledgerUli Prantz
Builds and operates all-agents
Uli Prantz builds all-agents, the process-automation platform this site documents. He writes about the operational side of automating recurring business work: where deterministic code beats model judgment, where it does not, and where a human still has to approve.