An AI Governance Framework for Agentic Systems

Most AI governance is written to be read, not enforced. It lives in a policy document, a risk register, and a quarterly committee review. That was adequate when AI meant a model that produced a prediction and a human decided what to do with it. It is not adequate now. In April 2026, a coding agent deleted a production database and its volume-level backups after it hit a credential mismatch, an incident widely reported as a plain case of excessive agency. No policy PDF stopped it, because a PDF has no enforcement surface at the moment a system acts.

There is a lazy debate about this, and both sides are wrong. "Move fast and break things" is unserious about power grids, supply chains, and money. "Slow down and regulate" is managed decline dressed as caution. The adult position is the third one: adopt fast, and make the method temperate. That is only possible if your governance runs at the moment of action, not on a slide. This page defines AI governance, shows why the standard frameworks break the instant a system can act, and gives you a nine-control framework you can require of any vendor, including us.

What is AI governance?

AI governance is the set of policies, controls, and accountability structures that keep an AI system safe, compliant, and aligned with an organization's legal and operational obligations across its lifecycle. In practice it answers three questions: what is the system allowed to do, who is accountable when it acts, and how do you prove both after the fact.

Most established programs are built on the same five pillars: transparency, accountability, fairness, human oversight, and data protection. The reference frameworks map to those pillars. The NIST AI Risk Management Framework organizes the work into Govern, Map, Measure, and Manage. The EU AI Act sorts uses into risk tiers and puts most obligations on high-risk systems. ISO/IEC 42001 defines a management system for the whole program.

These are the correct baseline. Adopt them. But notice what they have in common: they assume a model that makes a prediction, offline, for a human to review. They govern outputs. They say almost nothing about how to bound a system that plans, calls tools, writes files, and takes actions on your behalf. The moment your AI stops predicting and starts acting, the paper stops working.

Why traditional AI governance frameworks break on agentic systems

An agentic system is not a smarter question-answering model. It plans a multi-step task, selects and calls tools, reads and writes data, and executes, often under a set of credentials it borrowed from a human. The unit you have to govern is no longer a prediction. It is a decision chain with real blast radius. Committee governance and model cards have no runtime hook into that chain.

The 2026 data shows how wide the gap is.

• 48% of production AI agents run unsecured, with no monitoring or logging of their actions, according to industry security reporting. You cannot govern what you cannot see.
• 88% of organizations reported a confirmed or suspected AI agent security incident in the past year, per the same body of research.
• Prompt injection is the top agentic vulnerability: OWASP and 2026 production data show that a single instruction hidden in a document or webpage can redirect an agent, with no malware and no stolen credential involved.

Two failure modes matter most to a security owner. The first is excessive agency: a system that can act is a system that can act wrongly at machine speed, as the deleted production database showed. The second is credential misattribution: when an agent runs under a human's login, your audit log records that the human deleted the file. Your provenance is a lie by construction, and separation of duties is gone.

This is also why pilots die. Roughly 88% of agentic AI pilots never reach production, and the model is rarely the reason. The blocker is governance and integration. If you cannot get a system past your own security review, it does not matter how capable it is. The same problem sinks do-it-yourself builds on open-source agent frameworks, which ship without governance and leave the RBAC, audit, and sandboxing for you to assemble. For a fuller definition of the category, see our briefing on agentic AI.

What an agentic AI governance framework actually requires

Governance for a system that acts is a runtime architecture, not a policy document. The distinction is the whole thing. A policy describes intended behavior and hopes for it. An architecture makes unintended behavior structurally impossible.

That requires inverting the default trust model. The standard approach trusts the model's output and cleans up afterward. The correct approach is the reverse: the administrator explicitly enables each capability, and everything not enabled is inaccessible. Deny by default. The system does not get the benefit of the doubt.

The right mental picture is not a flowchart you edit. It is robotics safety. The controls you need look like kill switches wired into the system's brain, ways to blind it, mute it, take its hands, and pull its power pack, not "delete workflow step two." This is governed AI as a runtime architecture, and it rests on three things the paper frameworks skip:

1. Enforcement at the moment of action: The control fires when the system tries to do the thing, not in a review three weeks later.
2. Inside the perimeter: In regulated environments, data cannot leave the boundary, so cloud-tenant agent execution is a non-starter. The perimeter is also where the control has teeth. Your buyer already carries scars from cloud vendor lock-in; this is the same lesson applied to AI.
3. A distinct identity per worker: If the system acts under its own identity, actions are attributable and separation of duties survives.

Vendor-agnostic inference belongs here too, as a governance control rather than a feature. Running on Anthropic, OpenAI, or a self-hosted model, in any combination, means no single-vendor dependency becomes a supply-chain risk. The model providers are the engine. The governance is the vehicle around it. They are not the thing you are governing against.

The framework: nine real-time governance firewalls

Here is the framework as an enumerated set of controls. Each one is a real-time firewall: it enforces a boundary while the system runs, not a rule you audit later. This is the exact architecture Mission Control deploys, and you should require every layer of any agentic vendor you evaluate.

1. Identity: Every synthetic worker gets a unique, verifiable identity. No shared service accounts, no borrowed human credentials. This is what makes every downstream control and every audit entry meaningful.
2. Human RBAC: Access is defined by an administrator, using the role-based access controls your team already runs. Humans grant; the system never assumes.
3. Synthetic RBAC: The worker carries its own credentials and permission scopes, provisioned like an employee in your existing identity provider. If you can reason about an Okta login, you can reason about this. Nothing new to learn.
4. Execution sandbox: Package and code execution is whitelist-based. The runtime blocks the os, subprocess, and sys modules. There is no arbitrary code execution, which closes the most direct path from a compromised prompt to a compromised host.
5. Audit logging: Every model call and file access is logged with complete provenance, attributable to the worker's own identity and SOC2-attested. This is the direct fix for credential misattribution: the log records what the worker did, not a human it was impersonating.
6. Delegation controls: A worker cannot escalate its own permissions or pass them to another worker. Privilege is granted from above, never manufactured from within.
7. Scheduling: A worker acts only inside defined operating windows. Outside them it is inert, which shrinks the time surface an attacker can use.
8. Communications: All messaging between workers is logged. Multi-worker systems do not get a private channel your security team cannot see.
9. Capability whitelist: Any capability that has not been explicitly defined is inaccessible. The blast radius is bounded by construction, not by good behavior.

Where these controls end. Be clear-eyed about this, because overselling governance is how you lose a security review. These nine firewalls bound what a worker can do, make every action attributable, and keep the blast radius small and reversible. They do not make the worker's judgment perfect, and they are not a substitute for human review of high-stakes actions. Governance guarantees provenance and containment. It does not guarantee that every decision inside the bounds is the one you would have made. A vendor who claims otherwise is selling you a slide.

How to apply this AI governance framework

The value of a framework is that it travels. Turn the nine firewalls into a checklist and take it into every agentic vendor evaluation, ours included. If a vendor cannot answer these plainly, that is your answer.

Map each row to your own regime. This is also the backbone of a broader vendor assessment; pair it with our guide on how to evaluate AI agents and, when you are comparing named platforms, enterprise AI agents ranked on governance. Different categories carry different governance profiles, which is why the difference between RPA and AI agents changes what you need to control, and why definitions matter enough that we keep a plain one for what an AI agent actually is. For where these controls land in practice, see the industrial AI briefing on regulated operations and the named workers in AI agent examples.

Governing synthetic workers with Swarm

Swarm implements all nine firewalls, on-premises, inside your perimeter, with vendor-agnostic inference. This is the reframe that matters for a regulated buyer. A copilot that lives in one SaaS tenant and calls an external model API cannot meet these controls, because the data leaves your boundary and the actions run under a borrowed login. A synthetic worker on your own systems, provisioned in your identity provider, behind your firewall, can.

The architecture makes unauthorized actions structurally impossible rather than merely discouraged. Capabilities are enabled by an administrator, undefined capabilities are inaccessible, and the runtime blocks arbitrary code execution outright. Every action lands in a SOC2-attested audit trail, via Drata, tied to the worker's own identity. Customer data never leaves the customer environment, which is what lets the same architecture serve NERC CIP, ITAR, DFARS, and HIPAA settings without a separate premium tier. On-premises is the default here, not an upsell.

The point is not that a synthetic worker is more capable than an agent. It is that it is governable in the places an agent is not. Capability without governance is what fills the pilot graveyard. Governance is what gets a system into production.

Conclusion

There is one idea under all of this. Paper governs predictions. Architecture governs actions. The frameworks you already run, NIST, the EU AI Act, ISO 42001, remain the right baseline, and they remain insufficient the moment your AI can act on your systems.

A short decision framework:

• If your AI system can take actions, not just make predictions, you need runtime firewalls, not a policy document.
• If it touches data that cannot leave your perimeter, it has to run inside the perimeter.
• If it runs under shared or human credentials, fix identity first, because nothing else is trustworthy until you do.

If all three are true, you are past the point where a governance PDF protects you. See how Swarm deploys inside your perimeter, with all nine firewalls running in your environment.

Related reading

• Industrial Artificial Intelligence: From Copilots to Synthetic Operators
• AI Agents Examples: What Synthetic Workers Actually Do
• How to Evaluate AI Agents for the Enterprise
• All posts on the blog