AI Agent Governance: A Practical Control Framework

What agent governance has to answer that ordinary software governance does not

Ordinary business software does what it was told to do: you review the rules once, and the rules are what runs. An agent differs in one way that matters for control design. You give it a goal and a set of tools, and it decides the path. What you approved was the boundary, not the behaviour, so you are no longer signing off on a sequence of steps. You are signing off on a scope: what it can reach, how far it can go alone, and what it leaves behind.

The NIST AI Risk Management Framework organises this work into four functions — govern, map, measure and manage — and treats GOVERN as the one that cuts across the others rather than a phase you finish. Its instruction for teams like yours is unglamorous and correct: have policies that define and differentiate human roles and responsibilities in human-AI configurations. In operator language, decide who is accountable for what, per process, before the first unattended run.

Governing the technology as a whole leaves you nothing to test. Governing one process does: it has a named owner, a known blast radius, a written procedure and a record. Everything below assumes one process at a time, and five questions asked about it — access, autonomy, record, escalation and review.

The control sheet: five questions and what a complete answer looks like

This is the framework. One sheet per process, filled in before it runs unattended and revisited whenever it changes. The third column is deliberately a minimum rather than an ideal: below that line you do not have a control, you have an intention.

One structural rule sits underneath the last row. The person who approves an action should not be the one who requested it or who is measured on the throughput of the process. Separating those roles is standard practice in financial controls, and automation changes nothing about it. If the same person defines the tiers, approves the exceptions and reports on the results, you have documentation rather than control.

What the agent may access: scoped credentials, not borrowed ones

Access is the control that works when nobody is watching. Every other control on the sheet depends on someone noticing something. Least privilege caps the damage from a step that misfires at three in the morning, which is why it comes first.

The most common mistake by a distance is running the process under a person's account because it was quicker. That one shortcut breaks two controls: the scope becomes whatever that person can do, usually everything they have accumulated over three years, and the audit trail now says a human did it.

• Give the process its own identity. Not an employee's login, and not a shared account used by four automations. Reading a log later, you want to know which process acted.
• Scope to the procedure, not the person. Grant only the systems, objects and fields the written procedure names. If it never mentions customer bank details, the credential should not reach them.
• Split reading from writing. Broad read access is usually cheap; write access should be narrow and enumerated. Most steps only read, match and prepare.
• Keep secrets out of the procedure text. The written procedure is read by people and by the model. It should never contain a key, a token or a password.
• Rotate and prune on a schedule. Rotation that only happens after an incident is an incident response. Review scope when the procedure changes, and remove permissions it no longer uses.

A useful test before you turn anything on: write down the worst single action this credential permits if a step goes completely wrong. If that sentence is uncomfortable to read, the scope is too wide, and no amount of approval design fixes it later.

What may run unattended, and what stops for a person every time

Autonomy belongs to each action a process can take, never to the process as a whole. A supplier payment process reads documents, matches records, drafts messages and releases money, and those four do not deserve the same treatment. Sorting them by consequence is the real design work, and the part usually skipped.

Sort by two things only: how visible the action is outside your team, and how hard it is to take back. Cost and frequency are distractions. A cheap action that reaches a customer and cannot be recalled belongs above the line; an expensive internal recalculation you can rerun does not.

Tier 3 is the row people argue about, usually as a confidence question: surely once it is accurate enough, the gate can go? That misses why the gate is there. Some decisions need a person on the hook for responsibility, however good the system has become. Where a decision produces legal or similarly significant effects for an individual, Article 22 of the GDPR gives that person the right not to be subject to a decision based solely on automated processing, and where such processing is permitted, the right to obtain human intervention, to express a view and to contest the outcome. A gate you can switch off is not an answer to that.

The second failure mode is subtler, and it is what turns governance into theatre: a gate exists, someone clicks it, and nothing is actually reviewed. Article 14 of the EU AI Act addresses this for high-risk systems: they must be designed so natural persons can effectively oversee them, overseers need the competence, training and authority to do so, and the design must work against automation bias, the tendency to over-rely on system output. That translates into concrete interface decisions.

• Show the payload, not a summary. The approver sees the actual message, amount or record being changed. A request that says “approve step 4” is not oversight.
• Make rejection cheap and editing possible. If the only options are approve or abandon the run, people approve. Correcting the proposal and continuing is what turns a gate into a control.
• Give the approver real authority. They must be allowed to say no, to stop the process, and to not be measured purely on how fast the queue clears.
• Watch the rejection rate and the queue length. A gate never failed across hundreds of runs means the tier is wrong or nobody is reading. Forty approvals a day means attention is already decaying.

What gets recorded, and what the record has to make answerable

After an incident, three questions get asked in the same order. What did it do? What was it working from? Who said it could? A record that cannot answer all three is an operational log rather than a governance record, and the difference only becomes apparent on the day you need it.

The regulatory direction of travel is the same. Article 12 of the EU AI Act requires high-risk systems to allow the automatic recording of events over the system's lifetime. Whether it applies to you depends on how the system is classified and on your role, which is a question for counsel rather than for this page. The design it implies is worth adopting regardless: it is the same design that lets you debug your own process.

• Run identity, trigger and inputs as received. The original artefact, not a cleaned-up interpretation. Interpretations are what you are checking.
• Procedure version. Which version this run executed against. Without it you cannot separate a bug from a rule change.
• Every step, attributed. Whether it was fixed logic, model judgment or a human decision, and for tool calls, what was sent to which system and what came back. Those three kinds of step have different failure modes and different remedies.
• Approval events. Who approved, what they were shown, whether they edited it, when, and the reason on rejection.
• Outputs that left the building, in the form they were sent.
• Exceptions, escalations and retention. What stopped, who was told, how it was resolved, and how long the record is kept.

The stronger version is a record complete enough to replay the run. A narrative summary leaves you guessing; the actual inputs and decisions let you investigate, and let you check a proposed fix against everything that already worked.

How exceptions escalate, and who reviews the arrangement on a schedule

An exception is not a failure of the design. It is the design working: the process met something its procedure does not cover and stopped instead of improvising. What matters is that the stopping goes somewhere, within a stated time, and leaves the procedure better than it found it.

1. 01The process stops instead of guessingTrigger
   An unfamiliar input, a tool that failed, a case the written procedure does not cover. The correct behaviour is to halt that run and preserve its state, not to pick the most plausible option and carry on.
2. 02It routes to a named person, then escalates on a clockCode
   Not a shared alias, and not a dashboard someone might look at. A named first responder gets the run, the inputs and the point of failure, with a pick-up limit matching the tier. Past the limit: a second name, then a defined fallback to doing the work manually. Work quietly waiting on an unavailable person is the most common way an automated process fails unreported.
3. 03A person resolves the case and states the ruleHuman approval
   Handling the case is half the job. The other half is saying what the rule should have been, so the answer lands in the written procedure rather than in one person's memory.
4. 04Repeat exceptions become procedure, not habitHuman approval
   The same exception arriving a third time is a missing rule. Catching that is the difference between a process that improves and one where somebody permanently patches the same gap by hand.

Review keeps the other four controls honest, and it decays first, because a process that appears to be working generates no prompt to look at it. Put it on a calendar with a named owner and a specific question for each item.

Where these controls attach in a process that starts agentic and becomes code

A control sheet is only worth filling in if the system running the process can enforce it. Here is where each control lands in the approach all-agents takes, because the sequence changes which controls matter when.

1. 01Learn: the process runs with the operator watchingAI judgment
   The first runs execute agentically, with the operator present and every irreversible action approved before it happens. This is when your tier list meets reality: actions you filed under tier 1 turn out to reach a customer, and you find out while a person is still approving everything.
2. 02Decide it is understood well enough to pin downHuman approval
   Each accepted run is captured as a concrete example, and the written procedure is refined with what it taught. A person, not a counter, decides when it is complete enough to codify. That decision is itself a governance event worth recording.
3. 03Codify: walk the procedure step by stepHuman approval
   A step-by-step walk-through turns the understood procedure into deterministic code. Steps that genuinely need judgment stay with the model. Steps the human pins keep a permanent approval gate, which is where your tier 3 rows belong. What it surfaces is written back into the procedure, so the document stays the thing you audit against.
4. 04Run: deterministic, gated where it was pinned, recordedCode
   The codified process runs the same way every time, still stopping at the pinned gates, every run recorded and replayable. Determinism is itself a control: a step that behaves identically on every run is one you review once rather than sample forever.

Two consequences are worth naming. The number of steps left to model judgment should fall over the life of a process, and each one remaining marks a place the procedure could not be pinned down: a governance metric as much as a cost one. And permanent gates are not a transitional measure a mature process earns its way out of. They are a property of the decision, and they survive codification by design.

If the procedure is not written down yet, start there instead: the procedure guide covers what makes one precise enough to check an automation against. For gate design, the human-in-the-loop guide goes deeper, and the reliability guide covers the failure modes these controls exist to catch.

Limitations and when not to use this

• Nothing here is legal advice. Whether the EU AI Act applies to your system, whether it counts as high-risk, which obligations fall on you as provider or deployer, and how Article 22 of the GDPR meets your legal basis all depend on jurisdiction, sector and role. We cite the primary sources so you can read them; take advice before relying on any of it.
• This framework is sized for operational processes owned by a small team. In a supervised sector, your regulator's requirements are more specific than anything here and come first. Treat it as a floor, not a programme that satisfies a supervisor.
• Controls need something to check against. If you cannot yet write down what a correct run looks like and what should happen when one goes wrong, the honest next step is the written procedure, not the automation.
• Some processes should not be automated at all, and governance cannot rescue them. If nearly every case is a judgment call about an individual, leave the work with a person and spend the effort on making their job easier.

Sources

1. AI RMF 1.0 (NIST AI 100-1) — NIST Accessed 24 July 2026
2. Article 14: Human oversight — EU Artificial Intelligence Act Accessed 24 July 2026
3. Article 12: Record-keeping — EU Artificial Intelligence Act Accessed 24 July 2026
4. Art. 22 GDPR — Automated individual decision-making — GDPR Accessed 24 July 2026

Related reading

• Human-in-the-Loop AIThe approval side in detail: how to build a gate a person actually reads, and which decisions keep one permanently.
• AI Agent ReliabilityThe failure modes these controls exist to catch, and what to do about each one.
• SOP AutomationControls need a written procedure to check against; this is how to write one precise enough to test.
• SecurityHow credentials, approvals and run records are handled inside all-agents itself.