Enterprises are deploying AI agents. Not experimenting — deploying. Into customer service, financial operations, healthcare administration, supply chain management. The question is no longer whether agents will be part of enterprise operations. They already are.
The governance infrastructure hasn’t kept up. Most organizations deploying agents today rely on a combination of careful prompting, thorough (design time) testing, and retrospective (runtime) logging. Build the agent well, watch what it does, investigate when something goes wrong.
We think this model is insufficient. And with the EU AI Act’s provisions for high-risk AI systems taking effect in August 2026 — five months from now — it is also increasingly a regulatory liability.
The core architectural gap is this: agent governance must operate on execution paths — the sequence of steps an agent takes at runtime — not just on static specifications or individual outputs. This post proposes a three-layer control framework and translates it into practical terms for enterprise teams deciding how to govern their agents.
You Cannot Fully Specify Agent Behavior at Design Time
Traditional software is deterministic. Same inputs, same outputs. You can verify correctness before deployment; at design time. AI agents are fundamentally different. The same prompt, the same context, the same tools — and you get meaningfully different behavior across runs. This is not a bug. It is how large language models work and it is part of their power.
The consequence is underappreciated: you cannot guarantee agent behavior at design time. You can constrain it, guide it, test it extensively — but the agent will, eventually, do something you didn’t anticipate. The question is not whether agents will make mistakes. It is what happens when they do.
And not all mistakes are equal. An agent that drafts an awkward email is a minor issue. An agent with write access to a financial system that executes an unauthorized transaction is categorically different. The relevant variable is not error frequency — it is the potential cost of each error, weighted by the probability that it occurs.
This is why the right frame for enterprise agent governance is not “prevent all errors” — that goal is unachievable — but “minimize the cost of the errors that will inevitably occur.”
That reframe has architectural implications.
Design Time Is Necessary. It Is Not Sufficient.
If agents will make mistakes, and the cost varies by action type and context, then you need a system that intervenes before high-cost actions are taken — not one that detects them afterwards.
Design-time governance — careful prompting, guardrails in the system prompt, thorough testing, restricted tool access — reduces the probability of bad behavior. It is necessary.
Runtime governance — intercepting every prospective action and evaluating it against organizational policy before execution — reduces the cost of bad behavior. It catches what design-time missed.
These are complements, not substitutes. Fewer violations, each less costly. Current enterprise practice is almost entirely focused on the first. Most “AI governance” products on the market are prompt filtering, output moderation, or compliance documentation tools. None of them actually stop an agent from doing something it shouldn’t, at the moment it is about to do it.
Three Layers, Three Questions
A credible governance stack works at three levels, each addressing a different failure mode:
Layer 1 — Prompt-level: Does the agent know the rules? Applicable policies are surfaced to the agent as context at the start of each task. This reduces accidental violations — an agent that knows it cannot send external emails is less likely to attempt it. But this layer is instructional, not enforceable. A prompt that says “do not send emails” does not prevent the agent from invoking the email tool. It is a speed advisory, not a speed governor.
Layer 2 — Access-level: Is this action available? Tool whitelisting, API authentication, role-based access control. This restricts what the agent can do but not how it uses what’s available. An agent with read access to a customer database and write access to email has everything it needs to exfiltrate customer data. The violation lies in the sequence, not in any individual tool permission. This is not easy to solve: making tool access permissions sufficiently tight such that they “properly” constrain the agent (at design time), will render the agent itself practically useless.
Layer 3 — Runtime-level: Is this specific action permitted right now? Every prospective action is evaluated against policy before execution, in the context of the full execution path — everything the agent has done so far in this task. If the action violates policy, it is blocked before it happens. If it requires human approval, execution pauses until approval is granted. The agent never gets to execute the problematic step.
Each layer alone is insufficient. An informed agent can still misbehave. Access control can’t detect violations that arise from permissible actions in impermissible sequences. Runtime enforcement, when combined with a clear step by step audit trail is the only thing that should satisfy regulators.
The critical gap in most current deployments is Layer 3. Layers 1 and 2 are becoming standard practice. Layer 3 — the thing that actually evaluates actions in context, at runtime, before they execute — barely exists as a category.
The Organizational Policy Problem
There is a compounding issue. In any enterprise of meaningful size, AI agents are not a single system — they are many systems, built by different teams, running on different frameworks, serving different functions. A bank might have agents handling customer queries, processing loan applications, monitoring fraud, and generating reports. Each built differently, each with different risk profiles.
The policies that govern all of them — data handling, approval workflows, escalation thresholds, regulatory constraints — are organizational. They belong to the enterprise, not to any individual agent.
Today, those policies are either embedded in each agent individually (fragmented, impossible to update consistently) or they exist only as system prompts that the agent may or may not follow.
What is missing is a layer that sits above all agents — framework-agnostic, organizationally scoped — that enforces consistent policy across everything, and generates the evidence to prove it did.
Evidence, Not Intentions
This brings us again to the regulatory dimension. The EU AI Act doesn’t just require that you have policies. It requires that you can prove you enforced them. Article 12 requires automatic logging. Article 14 requires human oversight. Article 9 requires continuous risk management.
For AI agents, meeting these requirements means logging not just what the agent did, but the complete governance record: which policies were evaluated at each step, what risk scores were computed, what enforcement decisions were made, and — critically — evidence that violations were detected and corrected before harm occurred. An immutable, hash-chained audit trail is not a nice-to-have. It is the legal foundation.
A log that records only agent actions shows what happened. A log that records governance decisions demonstrates that oversight was active. That distinction matters enormously when a regulator asks: “Were you in control of this system?”
An Honest Assessment
There are things that even the runtime enforcement layer does not solve. It does not make agents perfectly reliable; policies enforced at runtime simply reduce the cost and impact of agent failures. Runtime enforcement does not replace careful agent design and testing — design-time and runtime governance multiply each other’s effectiveness. Runtime enforcement will not work equally well across all platforms; deep integration is possible in programmable environments like LangGraph or custom Python agents, while low-code platforms offer shallower hooks and less complete enforcement options.
And it does not answer the hardest question in enterprise AI governance, which is not technical: who in your organization is accountable for the behavior of your agents, and what does that accountability actually mean in practice?
The EU AI Act’s high-risk provisions take effect in August 2026. Organizations that cannot demonstrate runtime governance for their agent deployments will face a choice: pull agents out of high-risk contexts, or operate in non-compliance. Neither option is attractive. The enterprises that build using layer 3 enforcement now — i.e., those that implement not just logging, not just testing, but actual runtime enforcement with auditable evidence — will be the ones that can deploy agents confidently into higher-stakes contexts while their competitors are still too nervous to try.
The practical infrastructure to do this is what we’re trying to build.
Maurits Kaptein is the founder of Kyvvu, a runtime compliance platform for AI agents and the author of AI Agents at Work.
*Want to discuss how runtime governance applies to your agent deployments? Reach out.