Key takeaways
- AI can shorten discovery and handle messy language, but it does not remove the need for a defined outcome and owner.
- Move settled rules and side effects into deterministic code; keep model judgment bounded and evaluated.
- Consequential actions need explicit authority and durable human gates.
- Production readiness includes examples, regression tests, observability, failure handling, and rollback or fallback.
- When reality changes, stop and propose a tested patch or return to supervised learning instead of mutating behavior invisibly.
Build internal tools with AI by using the model where language and legitimate variation make rules brittle: process discovery, extraction, classification, comparison, drafting, and repair proposals. Keep state transitions, permissions, calculations, side effects, reconciliation, and release controls deterministic where possible. Evaluate model behavior on representative examples, preserve human approval for consequential actions, and stop failed runs for tested, owner-confirmed repair or supervised relearning.
AI changes the cost of specification and judgment
Traditional internal software becomes expensive when a process owner must translate every tacit rule and exception into a specification for a builder. A conversational learning loop can collect examples and clarify decisions closer to the work. At run time, a model can interpret messy text or documents that would otherwise need a large rule tree.
That can lower the threshold for an internal tool. It does not eliminate the system around the model. NIST's AI Risk Management Framework emphasizes managing AI risk across governance, mapping, measurement, and management rather than treating model output as self-validating.
Separate deterministic execution, bounded judgment, and authority
| Layer | Good work | Boundary |
|---|---|---|
| Deterministic code | Validation, calculations, matching, state transitions, API calls, idempotency, verification | Cannot resolve unwritten policy |
| AI judgment | Extraction, classification, comparison, summarization, drafting, anomaly explanation | Must be bounded by examples, output contract, abstention, and evaluation |
| Human authority | Policy, novel exceptions, sensitive decisions, consequential approval, release | Needs enough evidence and time to make a real decision |
Use a learn-to-run loop that makes change visible
- 01Teach with casesHuman approvalDemonstrate normal work, corrections, exceptions, and permanent approval boundaries. Remove sensitive details from reusable examples.
- 02OperationalizeCodeTurn settled behavior into versioned steps and isolate the minimum model judgments with explicit inputs and outputs.
- 03EvaluateAI judgmentUse representative, adversarial, boundary, missing-input, and changed-condition cases. OpenAI's official evaluation guidance recommends task-specific evaluations and continuous evaluation as systems change.
- 04Observe and approveHuman approvalCompare proposed results with current work, inspect disagreements, and retain gates before consequential effects.
- 05Run with evidenceTriggerRecord the version, inputs, decisions, approval, attempted and confirmed effects, corrections, and outcome.
- 06Repair or relearnHuman approvalAfter a stopped failure, confirm a regression-tested patch when implementation is wrong or return to supervised learning when reality changed.
A demo proves possibility; production requires an operating case
| Demo shows | Production evidence adds |
|---|---|
| One successful prompt | Versioned instructions and a representative evaluation set |
| A generated interface | Identity, permissions, validation, accessibility, and error states |
| An API call | Idempotency, reconciliation, timeouts, retries, and least privilege |
| A plausible answer | Correctness criteria, abstention, review, correction, and drift monitoring |
| A changed workflow | Regression results, authority review, owner approval, release, and rollback |
Choose a first AI internal tool with bounded consequence
- Frequent enough to collect examples and measure improvement.
- One named process owner and a small user group.
- Messy language or documents where bounded interpretation adds value.
- A deterministic finish state and external effects that can be verified.
- Actions that are drafts, internal, reversible, or protected by approval.
- A manual fallback that can absorb exceptions and outages.
Limitations and when not to use this
- AI-generated code or interfaces require the same security, licensing, accessibility, testing, and operational review as other software.
- Model evaluations estimate behavior on selected cases; they do not prove correctness for every future input.
- This page does not promise that a production tool can be created in a fixed conversation length or that maintenance and API changes are free or autonomous.
Sources
- AI Risk Management Framework — NIST Accessed 14 August 2026
- Evaluation best practices — OpenAI Accessed 14 August 2026
- Testing for Reliability — Google Site Reliability Engineering Accessed 14 August 2026
Map the first workflow
Label each step as deterministic code, bounded AI judgment, human approval, or trigger before choosing an interface.
Map the first workflowUli Prantz
Builds and operates all-agents
Uli Prantz builds all-agents, the process-automation platform this site documents. He writes about the operational side of automating recurring business work: where deterministic code beats model judgment, where it does not, and where a human still has to approve.