Stop Serving Agents Like Chatbots

The easiest mistake in agent infrastructure is to treat an agent like a chatbot with a longer timeout.

That assumption works just long enough to be dangerous. A chatbot request is mostly a model-serving problem: receive input, retrieve context, stream tokens, log the response. A production agent is different. It loops. It accumulates state. It waits on tools. It may hold a browser, terminal, file workspace, database session, or sandbox alive between model calls. It burns tokens in bursts, then sits idle while some external system responds. It creates a trail of decisions that has to be debugged, governed, and sometimes replayed.

That is why the new paper “Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Framework” is worth paying attention to. Aries is not another leaderboard arguing that one model is slightly better at acting than another. It is a systems paper asking a more operational question: what if the serving layer itself is wrong for agents?

The Aries authors frame autonomous agents as workloads that combine repeated inference, persistent context, and sandboxed tool execution. That combination breaks a lot of assumptions inherited from conventional LLM gateways. If you only measure prompt tokens, completion tokens, and first-token latency, you miss the time spent waiting on tools, the resource profile of sandboxes, the cost of retaining state, and the trajectory-level failures that happen between model calls.

Their findings are a useful warning for anyone building enterprise automation. Token-centric metrics can miss non-inference bottlenecks. Keeping more context around can improve accuracy only up to a point, while still reducing serving capacity. Tool sandboxes can alternate between long idle windows and short resource bursts. Snapshot-based state management can make aggressive suspension expensive. And the sandbox is not just a performance concern; it is part of the attack surface.

In other words: the agent is the unit of work, not the individual model call.

That sounds obvious until you look at how many systems still price, schedule, observe, and debug agents as if the model call were the only thing that mattered. A support agent that spends 40 seconds waiting for a CRM update is not “slow” for the same reason a chat completion is slow. A coding agent that keeps a large workspace and terminal alive is not just a big prompt. A healthcare workflow assistant that checks records, calls eligibility tools, drafts a message, and waits for human review is not a single request with extra steps. These are trajectories.

A trajectory has phases. It has model reasoning, context assembly, tool selection, tool execution, idle wait, state mutation, policy checks, retries, and final handoff. Each phase has a different cost and risk profile. If the platform cannot see those phases, it cannot optimize them.

This also connects to OpenForgeRL, another recent paper that looks at agents through the lens of their real harnesses. OpenForgeRL starts from a training problem: modern agents run inside elaborate inference harnesses such as Claude Code, Codex, and OpenClaw, but standard SFT and RL stacks do not naturally express stateful, multi-process agent execution. The paper’s answer is to decouple training and inference with a lightweight proxy that serves harness model calls while recording them as training data for RL infrastructure such as veRL, plus a Kubernetes orchestrator that runs each rollout in an isolated container.

The specific benchmark results are interesting, but the bigger pattern is more important for builders: training and serving are converging around the actual runtime. If you want to improve agent behavior, you need traces from the harness where the agent really plans, calls tools, verifies work, and fails. If you want to serve agents reliably, you need the same kind of trace visibility in production.

That does not mean every company needs to build a research-grade agent cloud. Anthropic’s “Building Effective Agents” remains the right counterweight: use simple, composable patterns when they solve the job, and add autonomy only when it earns its complexity. The lesson from Aries is not “make everything more complicated.” It is “do not pretend complex agent workloads are simple chat workloads.”

A practical agent-native serving plan starts with measurement. Log the full trajectory, not just the final answer. For each run, capture model-call timing, token growth, retrieved context, tool latency, sandbox lifecycle events, retries, policy decisions, human approvals, and final outcome. Make those events correlate across components so a team can reconstruct what actually happened.

Next, separate task semantics from execution configuration. The business task should not be welded to one timeout, one context policy, one sandbox size, or one retry strategy. Aries uses that separation for experimentation, but the production lesson is broader. Teams need to test whether a workflow benefits from longer retained context, cheaper models in intermediate steps, more aggressive sandbox suspension, or stricter tool isolation without rewriting the task itself.

Then treat context as a capacity budget, not a free memory drawer. More retained context can help an agent avoid repeated discovery, but it also consumes serving capacity and may deliver diminishing returns. Mature platforms will need context policies: what persists, what compresses, what gets reloaded from source, what expires, and what must be revalidated before action.

Tool execution deserves the same attention. If sandboxes spend most of their lives idle but occasionally burst, the scheduler should know that. If suspension and resume are expensive because state snapshots are heavy, that cost has to appear in planning. If a tool environment can access files, networks, credentials, or browsers, the serving layer has to treat sandbox minimization as a security requirement, not an afterthought.

For enterprise teams, the near-term move is not to buy a mythical “agent platform” and hope it solves everything. It is to add trajectory awareness to the systems they already run. Give each agent run a durable ID. Join model logs to tool logs. Track wait time separately from inference time. Keep enough state to replay failures. Review context-retention policies. Measure sandbox cost and risk. Make human approvals and policy denials first-class events.

The platform of record for agents will look less like a model gateway and more like a runtime for governed work. It will still serve models, but it will also schedule tools, manage state, constrain sandboxes, preserve evidence, and expose the whole path from intent to action.

That is the infrastructure shift Aries is pointing at. Agents are not bigger chatbots. They are stateful workers. Serve them accordingly.