Capability Is Not a Launch Gate: The Evidence Your Agent Needs Before Production

A good agent demo is usually a story about capability. The model found the right record, called the right tool, drafted the right response, and made the workflow look easy. That matters. But it is not the same as being ready for production.

Production is where the agent stops performing for a clean prompt and starts operating inside an organization. It inherits messy context, stale memory, partial permissions, uneven data quality, ambiguous handoffs, human escalation paths, compliance obligations, and real side effects. A model can look capable in a test run and still be unready for that environment.

That gap is the useful provocation in Fouad Bousetouane's new arXiv paper, “Stop Shipping AI Agents on Faith: Capability Is Not Production Readiness.” The paper introduces the ProofAgent Index, or PAI, as a readiness frame for agent launches. Its central point is simple and overdue: a launch decision should not collapse into “did the agent score well?” It should ask what evidence exists across evaluation, context, compliance, and governance.

That four-part split is valuable because it breaks one of the most common mistakes in agent programs. Teams often treat evaluation as the whole release gate. They run a benchmark, record a pass rate, maybe collect a few golden-path demos, and call the system safe enough to pilot. But evaluation mostly tells you what was observed under a defined test condition. It does not automatically prove that the surrounding operating environment is stable, authorized, auditable, or recoverable.

The PAI paper's abstract reports that validation across healthcare and finance separated higher-risk from lower-risk configurations, and that context engineering strongly changed reliability. That should surprise no one who has operated agents for more than a week. The same model behaves differently when retrieval scope changes, when a tool schema is tightened, when memory is allowed to accumulate, when instructions compete, or when a user asks for an edge-case exception. Capability improves behavior, but it does not determine readiness by itself.

For builders, the first practical takeaway is to make context a release artifact. Do not just ask whether the agent can solve the task. Ask what environment produced that result. Which tools were available? Which fields could be read or written? Which memories were loaded? Which retrieval corpus was in scope? Which policies were injected into the prompt, and which were enforced outside the model? Which failure cases were simulated? Which human approvals were required before side effects?

If the answer lives only in a prompt, a Slack thread, or a staff engineer's memory, it is not production evidence. It is tribal knowledge wearing a lab coat.

The second takeaway is that compliance and governance cannot be pasted on after launch. NIST's AI Risk Management Framework is organized around governing, mapping, measuring, and managing AI risk. That language is intentionally operational. It pushes teams to know the context, measure relevant risks, assign responsibilities, and manage the system over time. For an agent, that means a release packet should show not only model behavior, but also authority boundaries: who approved the workflow, what the agent is allowed to do, what data it can touch, when it must stop, how incidents are reviewed, and how changes are rolled back.

This is especially important because agents are not just answer engines. They are action engines. They retrieve, transform, decide, call APIs, update systems, and hand work to other agents or humans. Once an agent can create side effects, the organization needs runtime evidence, not policy theater. A PDF saying “human oversight exists” is weak evidence. A trace showing where the human approval was required, what was approved, what tool call followed, and what audit record was produced is much stronger.

That is why tracing should be treated as part of readiness infrastructure. OpenAI's Agents SDK documentation, for example, describes tracing for agent runs, tool calls, handoffs, and workflow debugging. The specific SDK matters less than the pattern: production agents need a black-box recorder. If a customer record changes, a ticket escalates, or a workflow stalls, operators should be able to reconstruct the path without guessing. The readiness question becomes: can we observe the agent well enough to trust, debug, and constrain it?

Anthropic's “Building effective agents” makes a complementary point from the design side: the strongest implementations tend to use simple, composable patterns rather than elaborate frameworks. That maps directly to readiness. If the workflow is too tangled to explain, it is usually too tangled to govern. Narrow tools, explicit handoffs, boring state machines, and clear human checkpoints are not signs of immature agent design. They are how autonomy becomes operable.

A useful production readiness review can be blunt. Before launch, ask for four folders of evidence.

First: evaluation. Show task suites, adversarial cases, regression tests, human review samples, and known failure modes. Include what was not tested.

Second: context. Show the exact tool catalog, retrieval scope, memory policy, prompt hierarchy, permission boundaries, and environment assumptions. Include how the system behaves when context is missing, stale, or contradictory.

Third: compliance. Show the applicable rules, internal policies, data-handling constraints, retention expectations, and approval requirements. More importantly, show where those constraints are enforced in the runtime instead of merely described in a document.

Fourth: governance. Show ownership, monitoring, incident response, audit logs, kill switches, rollback paths, change control, and the criteria for expanding or reducing autonomy.

The point is not to create bureaucracy for its own sake. The point is to stop asking a single number to answer four different questions. A high evaluation score says the agent can perform under tested conditions. It does not say the workflow is scoped, the data path is compliant, the authorization boundary is correct, or the operations team can recover when the agent behaves strangely.

The healthiest shift is cultural. Treat every agent launch as an evidence review, not a faith exercise. The review does not need to be heavyweight for every internal assistant. But it should be explicit. The more sensitive the tools, data, and consequences, the more visible the evidence packet should become.

That is the line between a promising agent and an operational system. A promising agent can do the task. A production-ready agent can do the task inside a bounded environment, under monitored authority, with enough evidence for the organization to explain what happened and enough control to stop it when needed.

Capability gets the agent into the room. Readiness decides whether it gets the keys.