The Next Agent Safety Layer Is the Trajectory, Not the Prompt

Most agent safety discussions still sound like chatbot safety discussions with tools bolted on. We ask whether the prompt is safe, whether the next tool call is allowed, or whether the model should refuse the current request. Those checks matter, but they miss the shape of the problem. A capable agent does not fail in one sentence. It fails over time.

A multi-turn agent accumulates context, partial goals, delegated subtasks, files, retrieved evidence, credentials, and user approvals. By turn twelve, the dangerous part of the workflow may not be visible in the current instruction at all. It may be distributed across six harmless-looking steps: collect this list, transform that record, open this admin page, compare these identifiers, prepare a message, submit the form. Each step can look plausible in isolation while the trajectory as a whole points somewhere the system should not go.

That is why a new arXiv paper, “Reassembling Distributed Risk: Trajectory-Conditioned Action Generation for Multi-Turn Agent Safety,” is worth attention from builders even before its specific technique is ready for everyday deployment. The paper’s core observation is practical: tool-using agents move risk from text into external action, and multi-turn decomposition attacks can spread harmful intent across otherwise ordinary requests and tool calls. The authors propose ReDiR, a generation-time defense that compresses the current trajectory into a compact safety representation before each action and injects that signal into the frozen base model. In their reported evaluations across two agent-safety benchmarks, three model families, and eight held-out tool domains, ReDiR reduces attack success rates below 8% while preserving benign task fidelity with low overhead.

The exact mechanism is a research result, not a plug-and-play enterprise control. But the architectural lesson is immediate: production agents need memory of risk, not just memory of facts.

A useful way to think about this is a trajectory risk ledger. Every meaningful step in the run should update a small, structured picture of what the agent appears to be doing, what authority it is using, what evidence it has relied on, and what unresolved risk is still accumulating. That ledger should travel with the agent. It should be consulted before the next action is generated, before a tool call is allowed, before a credential is used, and before a workflow crosses from reversible preparation into irreversible side effect.

This is not the same as dumping the entire transcript into a larger context window. Long context can preserve information, but it does not automatically preserve judgment. A transcript says what happened. A trajectory control layer says what matters about what happened: the inferred objective, the active permissions, the sensitive resources touched, the mismatch between user intent and agent action, the number of failed attempts, the quality of the evidence, and the point at which the workflow should stop.

A second recent paper, “Tunable Tool-Call Rates in LLM Agents via Representation Steering,” reinforces the same operational theme from a different angle. Tool use is not a binary feature; it is a rate and a policy. The paper argues that unnecessary calls add latency, cost, and side-effect risk, while missed calls leave a model confidently wrong when it should have retrieved or computed. The authors report that a single residual-stream direction can steer tool-call propensity from near 0% to over 90% while keeping calls well formed, and that sweeps over this dial can trace cost/accuracy trade-offs during live tool execution.

Again, most teams will not be steering residual streams in closed production models tomorrow. The valuable takeaway is that tool propensity belongs in the control plane. Some workflows should be tool-eager because correctness depends on live evidence. Others should be tool-conservative because the tools are expensive, sensitive, or capable of side effects. A serious agent platform should know the difference.

Put these two ideas together and the safety layer starts to look less like a warning banner and more like runtime infrastructure. A trajectory-aware agent platform needs at least five capabilities.

First, it needs action preflight that understands history. Before an agent sends an email, edits a record, runs a shell command, or calls an external API, the system should evaluate not only the current action but the path that produced it. Was the goal stable? Did the agent switch objectives without user confirmation? Did it combine data from sources with different trust levels? Did earlier tool results contradict the current plan? Has the user actually authorized this class of action, or only the investigation that preceded it?

Second, it needs risk budgets. Every tool call should have a cost profile: money, latency, reversibility, data exposure, compliance sensitivity, and blast radius. A read-only documentation search is not the same as a production database update. A draft message is not the same as a sent message. The ledger should make those differences explicit and reduce the remaining budget as the workflow approaches side effects.

Third, it needs tool scopes that are real boundaries. The Model Context Protocol authorization specification is a reminder that agent tooling still depends on ordinary security fundamentals: authorization servers, protected resources, client identity, and scoped access. A trajectory layer does not replace authorization. It decides when authorization should be requested, challenged, narrowed, or revoked.

Fourth, it needs escalation gates that are specific enough to be useful. “Ask a human when unsure” is too vague. Better gates sound like: stop if the agent’s inferred objective changes after a privileged tool is introduced; require confirmation before joining data across customer accounts; require reviewer approval after three failed tool attempts; downgrade to read-only mode when retrieved evidence is stale; block irreversible actions when the current step depends on an unverified model-generated identifier.

Fifth, it needs audit evidence. NIST’s AI Risk Management Framework emphasizes governance, mapping, measurement, and management. Those verbs are hard to satisfy with a black-box transcript and a vague statement that “the model decided.” Teams need a record that can answer: what did the agent know, what authority did it have, which risk signals were active, why was this tool call allowed, and which policy would have blocked it if conditions were different?

OWASP’s LLM application guidance points in the same direction. Risks such as prompt injection, excessive agency, sensitive information disclosure, and insecure output handling are not solved by a clever system prompt alone. They are application failures as much as model failures. The control layer has to live where prompts meet tools, data, identity, and operations.

For builders, the practical move is to start outside the model. Create a minimal trajectory object for every agent run. Record the user goal, tool calls, permission grants, evidence sources, side-effect class, failures, retries, and policy decisions. Add a small evaluator that updates risk after each step. Use it to annotate prompts, restrict tools, require confirmation, or stop execution. Keep the representation compact. The point is not to summarize everything; it is to carry forward the few facts that should change what the agent is allowed to do next.

This also changes how teams should evaluate agents. Static task success is not enough. You want tests where harmful or mistaken objectives are distributed across turns, where tool results are ambiguous, where a benign start becomes a privileged action, and where the right behavior is to pause rather than proceed. Measure whether the agent can reassemble the risk of the trajectory, not merely whether it can classify a single prompt.

The next wave of agent reliability will not be won by teams with the longest prompts. It will be won by teams that can explain the path from intent to action. If an agent takes action N, the system should know what turns 1 through N-1 implied, what authority was consumed, what risk accumulated, and why this was still the right next move. That is the difference between an impressive demo and an agent you can safely put near real work.