Agents That Wait Need a Serving Layer, Not Just a Queue

The easiest way to misunderstand an approval-gated AI agent is to treat it like an ordinary background job. A user asks for work, the agent prepares a plan, a manager approves the risky step, and the system resumes. Put the run in a queue, send a notification, and call it production-ready.

That mental model breaks as soon as the agent is no longer waiting for seconds. Real enterprise approvals often take minutes, hours, or the better part of a workday. Legal wants a second look. Finance has to confirm a threshold. A clinician, operations lead, or account owner needs to approve the action before the agent touches the system of record. During that pause, the agent is not simply idle. It is carrying conversational state, tool state, partially completed intent, evidence, policy context, and usually a very expensive inference footprint.

A new arXiv paper, “Adaptive KV Retention for LLM Agents at Human-Approval Timescales,” makes the infrastructure problem concrete. The authors study agentic LLM requests that are suspended while waiting for human approval. Their central tradeoff is simple but painful: keeping a suspended request’s KV state around can make resumption fast, but it consumes serving capacity that could have handled active work. Evicting the state frees capacity, but the resumed request may pay a heavy latency penalty. In their abstract, the authors report that retaining suspended KV can reduce active-serving goodput by 41%, while eviction can lead to nearly 10x higher resume latency when the request returns.

That is not a niche scheduler detail. It is a warning about how production agents will actually behave.

For the first wave of chatbots, the serving layer could optimize for short turns: receive prompt, generate answer, release resources. For tool-using agents, serving became more complicated because the model pauses for tool calls and then continues. But approval-gated agents stretch the pause from a tool round-trip into a human process. They do not just call a search API and wait a few seconds. They may wait for a person in a different department, a mobile push approval, a ticket comment, or a compliance review.

At that point, “just queue it” is not enough. A queue knows order. A serving layer for agents that wait must know what is safe to retain, what can be reconstructed, what must be revalidated, and what evidence has to travel with the run.

The KV-cache example is the most visible cost because it touches GPU economics. If every suspended agent run holds onto scarce accelerator memory, the system punishes active users. If every suspended run is evicted immediately, the system punishes the user who finally approves the task and expects the agent to continue with the same context. The Adaptive KV Retention paper proposes a tiered controller around GPU opportunity cost and reports substantial goodput improvements over several baselines on human-scale approval workloads. The exact technique will matter to serving-system researchers, but the broader lesson is already actionable: approval latency belongs in capacity planning.

That changes the questions an enterprise team should ask before launching agents. How long do approvals usually take? Which approvals are seconds, which are hours, and which expire overnight? What state is worth keeping hot? What can be compressed into a durable resume packet? What has to be recalculated because the world may have changed? If the agent prepared a payment, order, clinical note, deployment, or customer message before the pause, does the resumed run still have authority to execute it?

This is where the serving layer becomes more than cache management. It becomes a resumption contract.

A useful resumption contract should say: here is the user intent, here is the proposed action, here are the tools and data consulted, here are the policy checks already performed, here is the approval requested, here is what changed while the run waited, and here is what must be rechecked before action. Without that contract, a resumed agent can look coherent while silently operating on stale assumptions.

The same point appears from a different angle in “ATLAS: Dual-Horizon Diagnostic Evaluation for Industrial Tool-Use Agents.” ATLAS argues that industrial agents need evaluation across both the current request trajectory and the longer user interaction horizon. Final outcome scoring can hide where the deficiency occurred and whether later service remains aligned with earlier context. That is exactly the problem approval boundaries create. The system needs to know not only whether the final action succeeded, but whether the agent preserved the right context across the pause, resumed at the right step, and respected any changed conditions.

A waiting agent also needs monitoring that is not based on the agent’s own confidence. “CURA: Certified Runtime Alarms for Computer-Use Agents” is a useful reminder. In that paper’s OSWorld setting, a capable computer-use agent pipeline still ended most failures with a success claim. CURA’s proposed answer is an external runtime alarm using harness-visible telemetry rather than model internals, extra LLM calls, or prompt changes. The enterprise takeaway is direct: when an agent resumes after an approval, do not rely only on the agent saying, “I am still on track.” Carry forward the trace and let external monitors inspect the trajectory.

This matters because the approval step can create a false sense of safety. A human clicked approve, so the system feels governed. But the click may only validate one narrow decision: “Yes, send this quote,” or “Yes, open this ticket,” or “Yes, proceed with this database migration plan.” It does not automatically validate every hidden assumption in the agent’s working context. It does not prove that the target record is unchanged, that the credential scope is still valid, that the tool result was not stale, or that the agent did not drift during reconstruction.

The architecture pattern is straightforward, even if the implementation is not. First, separate suspended model state from durable operational state. KV retention is an optimization; the action record is the source of truth. Second, make each approval request carry an evidence bundle, not just a yes/no prompt. Third, define expiration rules for approvals and for resumed context. Fourth, measure resume latency and active-serving goodput as first-class SLOs. Fifth, hand runtime monitors the full trajectory around the pause, including what changed while the agent was waiting.

OpenAI’s background mode documentation is another signal that long-running model work is becoming a normal platform shape rather than an edge case. As backgrounded and agentic APIs become more common, teams will be tempted to hide complexity behind async primitives. That is fine for the transport layer. It is not enough for governance, cost, or reliability.

The companies that get this right will not describe their agent infrastructure as a pile of prompts behind a queue. They will be able to answer operational questions crisply. What happens to suspended state? What does it cost to keep? What is reconstructed on resume? What is revalidated? What can an operator see? What is the audit record if the resumed action fails?

Agents are becoming long-running operational processes. Long-running processes need lifecycle management. The pause is part of the product.

Before scaling the next approval-gated workflow, instrument what happens while the agent waits. That waiting room is where cost, context, authority, and accountability quietly pile up.

Sources

• https://arxiv.org/abs/2608.30830
• https://arxiv.org/abs/2608.30685
• https://arxiv.org/abs/2608.27808
• https://platform.openai.com/docs/guides/background