Runtime Governance Is the Missing Agent Primitive

The useful question is no longer, “Do we trust this AI agent?” It is, “What happens between the agent’s intent and the external effect it is about to create?”

That distinction matters because enterprise agents are becoming less like employees using software and more like temporary workloads moving through software. They appear for a task, assemble context, call tools, create side effects, and disappear. A human identity system can tell you which employee owns an account. A service identity system can tell you which long-running workload is allowed to call an API. Neither pattern fully answers the agent-era question: should this newly discovered, model-directed actor be allowed to perform this specific action, with this payload, against this provider state, right now?

A recent preprint, “Five Primitives for Governing Autonomous AI Agents at Runtime”, frames this as a runtime governance problem. The authors argue that agents strain existing control models in three ways: their principals can be ephemeral, their actions are selected by models rather than fully enumerated programs, and their populations are often discovered rather than provisioned. Their proposed primitive set is refreshingly operational: discovery, identity, governance, attestation, and supply chain.

That list is useful because it moves the conversation away from vague “agent safety” and toward deployable platform capabilities.

Discovery asks: what agents, subagents, scripts, workflows, and tool-calling surfaces are actually operating in this environment? Without discovery, governance starts from an incomplete inventory. A security team may approve the chatbot while missing the scheduled agent, the browser worker, the MCP-connected helper, or the coding assistant that can mutate infrastructure through a terminal. In practice, discovery should include runtime registration, tool-access mapping, owner attribution, and enough metadata to distinguish a demo agent from one that can send email, update customer records, or spend money.

Identity asks: who is acting, for whom, under what delegation, and for how long? A durable API key stuffed into an automation runner is convenient, but it is a poor fit for agents that should have task-scoped authority. The stronger pattern is short-lived identity bound to a user, tenant, workflow, policy version, and action vocabulary. That does not make the model “safe.” It makes the action attributable and constrainable before it reaches a provider.

Governance is the runtime decision point. It is where the system checks whether the action belongs to the approved vocabulary, whether the current state still matches the approval, whether required human or policy prerequisites are satisfied, and whether the payload contains only what the tool needs. This layer has to be active, not decorative. A dashboard that shows risky actions after they happen is observability. A governance point that can deny, rewrite, reserve, or require escalation is control.

The supporting research around this theme is pointing in the same direction. AID-Guard focuses on stateful authorization for delegated agent effects. Its core warning is that approval at admission is not enough when retries, response loss, concurrent requests, recovery flows, and provider-state changes can occur before commit. The paper proposes revalidating both the approved request and provider state at commit, retaining a reservation under ambiguity, and allowing a successor only after a terminal result or certified no-effect. Whether or not a team adopts that exact protocol, the design lesson is clear: the authorization boundary should cover the full path from approval to effect, not just the first API call.

Privacy research makes the same point from another angle. ToolMinimize reports that, in controlled measurements, default LLM tool calls often include privacy-sensitive data beyond what the tool requires. The authors describe middleware that rewrites arguments through removal, generalization, substitution, and truncation while preserving task validity in their evaluated setting. That is a concrete example of runtime governance doing more than saying yes or no. Sometimes the right control is to transform the action so the agent can complete the job without exporting unnecessary sensitive context to a third-party tool.

Handoff research exposes a quieter failure mode. In “When ‘Must’ Becomes ‘Maybe’”, the authors study how agent workflows can preserve the topic of a constraint while weakening its operational force. A blocker can survive in a summary as background information but stop functioning as a prerequisite. Their results are a reminder that governance state must be preserved as binding state. If a safety condition begins as “do not execute until X is resolved,” it cannot be compressed into “consider X” and still be treated as equivalent.

Attestation is the evidence layer that makes these controls inspectable. It should answer what policy was applied, what identity was used, what payload was sent, what was denied or rewritten, what provider response came back, and which artifacts were available at the time. The “Five Primitives” paper describes a hash-linked signed ledger in its architecture. The broader enterprise version does not have to start with a perfect cryptographic audit system, but it should be replayable enough for incident response, compliance review, and vendor-independent verification. If the only record of an agent’s behavior is a chat transcript, the platform is under-instrumented.

Supply chain is the primitive teams are most likely to postpone and later regret. Agents do not act alone. They inherit prompts, skills, tools, MCP servers, browser extensions, retrieval indexes, workflow templates, model versions, and evaluation harnesses. Each component can change what the agent is able to do or how it interprets authority. Treating those dependencies as supply-chain inputs means tracking provenance, versioning, review status, and allowed deployment contexts. It also means refusing to let an unreviewed tool schema quietly expand an agent’s authority just because the model can read it.

For builders, the practical maturity path starts small. Pick the handful of tools where agent mistakes would create real consequences: payment actions, outbound communications, record updates, infrastructure changes, data exports, and irreversible customer-facing operations. Put those behind a runtime mediation layer. Require ephemeral identity. Define the action vocabulary. Minimize payloads before execution. Revalidate at commit. Preserve binding prerequisites across handoffs. Store evidence that can be replayed without trusting the agent’s own explanation.

This is not a substitute for model evaluation, prompt quality, or human review. It is the layer that makes those investments survivable in production. Models will still misunderstand instructions. Workflows will still compress the wrong detail. Tool calls will still include more context than necessary. Retries will still happen after ambiguous failures. Runtime governance assumes those things and designs the execution path so they do not automatically become external damage.

The best agent platforms will not merely ask models to behave. They will make every meaningful action pass through an accountable boundary where identity, policy, state, payload, and evidence are checked together. That boundary is where autonomous systems become operable enterprise software.