How to Evaluate AI Agents for the Enterprise

Most AI agent evaluations are lost in the demo. A vendor shows a system booking a flight, closing a ticket, or reconciling an invoice, everyone nods, and the hard questions get deferred to a security review that happens after the contract is half-signed. That is backwards. For a regulated buyer, the demo is the easy part. The evaluation that decides whether a system reaches production is the one your security team runs, and the systems that die there rarely die on capability.

This is a guide on how to evaluate AI agents the way a security or platform owner has to: architecture first, deployment boundary first, evidence over adjectives. The market calls these systems AI agents, and we will use that term because it is what you are searching for. But notice the reframe underneath it. For an operator running a substation or a depot, an agent that lives in one SaaS tenant and calls an external API is not the thing you are evaluating. You are evaluating whether an autonomous system can run inside your perimeter, under its own identity, with every action provable and its blast radius bounded. The scarce, defensible asset in this whole exercise is the institutional knowledge captured inside your perimeter before it walks out the door, and the security boundary it has to be captured inside.

The good news: the criteria that matter are knowable and portable. Below is a checklist you can take into any vendor evaluation, ours included.

What is AI agent evaluation?

AI agent evaluation is the structured process of scoring an autonomous system against defined criteria, security, integration, performance, cost, and governance, before you deploy it. For a regulated buyer that standard rubric is not enough: the deciding criteria are whether the agent runs inside your perimeter, under its own identity, with every action audited and its blast radius bounded.

Hold onto that second sentence. Every vendor rubric on the internet covers the first one. The gap that sinks deployments is the second.

Why most AI agent evaluations fail

The failure pattern is consistent, and the numbers are stark. Roughly 88% of agentic AI pilots never reach production, per IDC and LangChain research, and Gartner projects that over 40% of agentic AI projects will be canceled by the end of 2027. Those sources report the mortality. They do not all name the cause, so read the next sentence as our conclusion, not theirs: the pilots that die rarely die because the model was not smart enough. They die on governance and integration, at the review gate you are now standing at.

The security picture explains why that gate is unforgiving. Industry security reporting found that 48% of production AI agents run unsecured, with no monitoring or logging of their actions, and that 88% of organizations reported a confirmed or suspected AI agent security incident in the past year. Prompt injection is the top agentic vulnerability, which means a single instruction hidden in a document can redirect a system that has real permissions. An ai agent security review that starts after the commercial commitment is signed is a review that will be overruled. Start it first, and give it a rubric with teeth.

Two failure modes should anchor that rubric. Excessive agency: a system that can act is a system that can act wrongly at machine speed. Credential misattribution: when an agent runs under a human's login, your audit log records that the human took the action, so your provenance is a lie by construction and separation of duties is gone. Everything below is, in effect, a defense against those two.

Start with the deployment boundary

Before you score anything else, ask one disqualifying question: does the buyer's data stay inside the buyer's security boundary?

If you operate under NERC CIP, ITAR and EAR, DFARS, CMMC, or HIPAA, regulated data cannot leave the perimeter, which makes cloud-tenant agent execution a non-starter regardless of how the system performs. Many otherwise capable platforms fail here on the first question, and it is better to find that out in week one than in week ten. How strict the boundary needs to be is genuinely buyer-context-dependent: a financial-services back office and a classified program sit at different points on that line. Establish your own line before you talk to a single vendor, then rank every option against it honestly.

The AI agent evaluation criteria that actually gate a security review

Here are eight requirements, ordered as an evaluation methodology. Treat them as portable ai agent evaluation criteria: for each one, know what to require, what to ask, and what a good answer sounds like. Where a criterion maps to a specific runtime control, that mapping is noted, because a control you can point at in the architecture is worth more than a promise in a slide. This is the exact architecture Mission Control deploys, and you should require every layer of any vendor you evaluate, us included.

1. Deployment boundary

Require: the system runs where your data already lives, on-premises or inside your own cloud tenancy, with no external callbacks. Ask: can it deploy fully inside our perimeter, and does any data leave our environment at any point, including for inference? Score: a vendor whose only deployment is their multi-tenant cloud fails for regulated data. Swarm deploys on-premises inside your perimeter, and customer data never leaves the environment. It is worth stating the limit plainly: on-premises is not the same as air-gapped, so if your program requires an air-gap, make that an explicit line item and require a specific answer rather than a marketing phrase from anyone, including us.

2. Identity and RBAC

Require: each worker has its own verifiable identity, provisioned and scoped in the identity provider you already run. Ask: does each worker authenticate as itself, or does it borrow a human's credentials, and can we manage it in our existing SSO/OIDC directory? Score: this maps to the first three governance firewalls, Identity, Human RBAC, and Synthetic RBAC. If you can reason about an Okta login, you can reason about a correctly built synthetic worker. Shared service accounts and borrowed logins are the red flag, because they destroy separation of duties before the system does anything.

3. Audit trail

Require: every action logged, attributable to the worker's own identity, and exportable for your auditor. Ask: when the auditor asks what the system did, when, with what data, and why, is the answer a structured query or a reconstruction? Score: this is the Audit Logging firewall, extended by the Communications firewall that logs messaging between workers. It is the direct fix for credential misattribution. In Swarm, every action is logged to a full audit trail, attributable to the worker's own identity, under SOC2 attestation via Drata. Note the precision there: Drata attests to the control, it does not do the logging, and a vendor who blurs that distinction is a vendor to watch.

4. Blast-radius bounding

Require: the system cannot do anything it was not explicitly enabled to do, and you can stop it. Ask, of every vendor including us: can you stop a running worker in real time, pause it, revoke its access, take its hands? Do not accept a slide. Require a live demonstration. Score: this maps to four firewalls at once, the Execution Sandbox (package whitelist, no arbitrary code execution, with os, subprocess, and sys blocked at the interpreter level), Delegation Controls (a worker cannot escalate or hand off permissions it does not have), Scheduling (it runs only inside defined windows), and the Capability Whitelist (anything undefined is inaccessible). The correct posture is deny by default: enable specific capabilities, and everything else does not exist for that worker. A system that trusts model output and executes it as code has an unbounded blast radius no matter how good the model is.

5. Governance as a runtime architecture

Require: governance that enforces at the moment of action, not a policy document reviewed quarterly. Ask: are your controls runtime firewalls or written policies, and what physically happens when a worker tries to exceed them? Score: the nine firewalls above are the set, and the point is that they fire while the system runs. This is governed AI as a runtime architecture, and it is the whole framework laid out in our AI governance framework for agentic systems. A policy PDF has no enforcement surface at the instant a system acts. Require the surface.

6. Inference-layer independence

Require: freedom to choose and swap the inference provider without a rebuild. Ask: can we run on Anthropic, OpenAI, or a self-hosted model, in any combination, and change that with a configuration, not a re-implementation? Score: single-vendor AI is now a supply-chain risk your own procurement apparatus will flag. Vendor-agnostic inference removes it. The model providers are the engine; the system you are evaluating is the vehicle around them. A platform hard-wired to one model provider concentrates a risk you are otherwise working to remove.

7. Cross-system reach

Require: the system touches every system a human operator touches, not just one ecosystem. Ask: does it work across Splunk, HubSpot, Salesforce, spreadsheets, email, and the legacy application built in 2007 that will never get an MCP server, or is it locked to a single platform? Score: real operational work spans systems. A single-platform embedded agent does the fraction of the job that lives in its own ecosystem and stops at the boundary. If the work you need done crosses systems, an option that cannot cross them is not actually in the running.

8. Learning mechanism

Require: clarity on whether the system runs fixed scripts or can acquire a new skill from demonstration. Ask: can it learn a departing expert's process by being shown it once, or does every task have to be programmed? Score: this one is honestly buyer-context-dependent. If all you need is a single fixed, scripted task, you may not need demonstration learning, and you should not pay for it. But it is the criterion that separates knowledge reanimation from automation. Documentation captures what was done; it never captures why. A synthetic worker that watches a 60 to 90 second screen-share, writes its own procedure, and improves with each correction captures the reasoning before the person who holds it retires. This is the requirement no pre-scripted system meets, and it is worth deciding on purpose whether you need it.

The portable checklist

Take this table into every evaluation. If a vendor cannot answer these plainly, that is your answer.

Map each row to your own regime, then use it to build a named shortlist. Our ranking of enterprise AI agents scored on deployment and governance applies exactly these criteria to specific platforms, and if you want the category definition underneath the term, start with what an AI agent actually is.

Run it as a process, not a demo

Evaluating AI agents for enterprise use is a sequence, and the order protects you. Issue a short requirements document to four to six vendors and cut to a shortlist on the disqualifying criteria first, deployment boundary before features. Then run the shortlist against your own real workflows, not the vendor's rehearsed demo, because the demo is engineered to pass and your workflow is not. Run the security review before the commercial commitment, never after. Finish with reference calls to buyers in your own regulatory regime, who will tell you what broke in month three.

The do-it-yourself path deserves the same rubric. Building on open-source agent frameworks is a real option, but those frameworks ship without governance and leave the identity, audit, sandboxing, and blast-radius controls for you to build and maintain. Score your own build against the eight criteria as strictly as you would score a vendor, and count the maintenance honestly.

Where Swarm lands on this checklist

Fairness requires we hold ourselves to the artifact we just handed you. Swarm is Mission Control's platform of synthetic workers, and here is the honest scorecard.

It meets the deployment boundary by running on-premises inside your perimeter, with customer data that never leaves the environment. It meets identity, RBAC, audit, and blast-radius bounding through the nine real-time governance firewalls, with RBAC provisioned in your existing identity provider and every action logged under SOC2 attestation via Drata. It meets inference-layer independence with vendor-agnostic inference across Anthropic, OpenAI, and self-hosted models. It meets cross-system reach because a synthetic worker touches every system a human operator touches. And it meets the learning mechanism through show-it-once demonstration learning, which is the one criterion no pre-scripted system satisfies.

Now the limits, because overselling is how a vendor loses a security review. Swarm is a newer category, not a self-serve download; deployment runs through a 12-week pilot with a forward-deployed engineering team, which is a heavier engagement than installing a tool. On-premises is not air-gapped, and if your program requires an air-gap, that is a specific conversation, not a checkbox. And the governance firewalls bound what a worker can do and make every action provable; they do not make its judgment perfect. Governance guarantees containment and provenance. It does not guarantee that every decision inside the bounds is the one you would have made, and any vendor who tells you otherwise is selling you a slide.

Conclusion

A usable evaluation comes down to three questions, in order:

• Can it run inside your perimeter? If not, for regulated data the evaluation is over.
• Can you prove and bound what it does, with its own identity, a full audit trail, and a bounded blast radius? If not, it will not survive your security review, and it should not.
• Do you need it to learn a process rather than run a script? Decide that on purpose, because it changes which category you are even shopping in.

Take the checklist to every vendor, hold each of them to it in plain language, and hold us to it too. If you want to see the nine firewalls running in an enterprise environment, see how Swarm deploys inside your perimeter.

Related reading

• What Is an AI Agent?
• What Is Agentic AI?
• What Is a Synthetic Worker?
• All posts on the blog