Train Your Tool Agent Where It Actually Breaks

A failed tool-using agent run is tempting to read like a failed conversation. The transcript starts cleanly, the agent makes a few reasonable moves, a tool response comes back, the next answer drifts, and by the end the workflow is wrong. The dashboard records a failure. The team opens the prompt, adds another instruction, maybe fine-tunes on the whole trace, and hopes the next run behaves better.

That is a blunt instrument for a system whose failure may have been caused by one model call.

A new arXiv paper, “Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Use,” is useful because it names a problem production teams already feel: in multi-turn tool use, not every bad-looking step is equally trainable, and not every reward signal points at the step that actually matters. The paper’s core claim is practical. When the outcome depends on later interactions, reward variation can be dominated by downstream randomness rather than by the quality of the current action. If you train the wrong call, you may spend compute, create regressions, and still leave the real failure untouched.

The authors propose Critical-State RL as a diagnostic for candidate calls inside a trajectory. Instead of treating the whole run as one unit, the method asks whether a local reward captures the effect of that action on task success and whether improvement over a reference policy is actually possible at that state. It then uses nested sampling to separate action-dependent reward variation from continuation noise, and trains the selected states with contextual-bandit optimization.

That sounds academic, but the operational translation is straightforward: before you patch an agent, locate the decision point where a better model action would have changed the run.

The paper reports experiments on the Berkeley Function Calling Leaderboard v4, a benchmark family built around structured function-calling and tool-use behavior. In the authors’ missing-function setting, the diagnostic selected the response after the tool becomes available. In the missing-argument setting, it selected the response before the missing argument is supplied. Those are different intervention points. The abstract reports that training the selected responses improved performance, including about 14 percentage points on the missing-function task, while training alternative states stayed flat or got worse.

The exact number should not be inflated into a universal enterprise promise. BFCL is a benchmark, not your billing system, claims workflow, CRM, or clinical operations queue. But the pattern matters: the best place to train may not be the loudest-looking error in the transcript. It may be the state where the agent first had enough information to act correctly, or the state just before it failed to ask for missing context.

That should change how teams design agent observability.

A lot of production agent logging is still transcript-shaped. You store the user request, model responses, tool calls, tool outputs, and final status. That is necessary, but it is not enough if the next question is, “What should we train?” For training and targeted remediation, the trace needs causal structure. Which calls were candidate decision points? What tool and memory state did the model see? What local reward or verifier was applied? Did the rest of the workflow introduce enough randomness that the local signal is unreliable? Was the observed failure caused by the current call, a later continuation, or missing infrastructure?

This is where tracing systems become more than debugging aids. OpenAI’s Agents SDK tracing documentation, for example, describes traces that include LLM generations, tool calls, handoffs, guardrails, and custom spans. That kind of span-level record is the raw material for the next layer: not just “what happened,” but “which decision was trainable, under what evidence, and with what expected effect.”

For enterprise teams, this creates a useful separation between three loops that are often mixed together.

The first loop is the runtime loop. It handles the immediate run: tool permissions, approvals, retries, revocation, escalation, and safe failure. If an agent is about to make a payment, mutate a record, or send an external message, the runtime loop decides whether that action is allowed now.

The second loop is the evaluation loop. It decides whether a version of the agent is ready to ship, whether regressions appeared, and whether the agent can handle a class of workflows. This is where final success rates, task suites, scenario coverage, and release gates belong.

The third loop is the training loop. It decides which model behavior should change. Critical-State RL is interesting because it pushes that loop down from whole-run judgment to candidate-state diagnosis. A failed trajectory is not automatically a training example for every token in the trace. It is evidence to localize a narrower intervention.

Confusing these loops creates bad habits. A runtime failure becomes a prompt patch. A benchmark regression becomes a retry rule. A whole transcript becomes fine-tuning data even though only one call had useful signal. The result is agent systems that accumulate instructions, exceptions, and hidden coupling without becoming easier to trust.

A better operating model looks more like a training ledger.

For every important agent workflow, capture the trace. Mark the candidate model calls that could plausibly affect success. Attach local checks where possible: argument completeness, tool availability, policy compliance, schema validity, user-consent state, retrieved evidence, or task-specific verifier output. Then ask whether the local reward really reflects the action or whether the outcome is polluted by later stochastic behavior. Only after that should the team choose an intervention: prompt edit, tool contract change, memory fix, evaluator update, supervised data, or RL-style optimization.

This also reframes what “agent eval” should produce. A leaderboard score or pass/fail release gate is useful for comparison, but production teams need diagnostic artifacts. They need to know whether the agent failed because it did not know a tool existed, did not ask for a missing argument, misread a tool result, chose an unsafe action, or recovered poorly after an exception. Those are different engineering problems. They deserve different fixes.

NIST’s AI Risk Management Framework uses broad lifecycle language around measuring, managing, and monitoring AI risk. Critical-State RL does not replace that governance layer, and NIST does not prescribe this technique. But the paper gives teams a concrete example of what “measure before manage” can mean inside an agent trajectory: not just measuring the final outcome, but measuring where a system is actually improvable.

The immediate checklist is simple.

Before fine-tuning a tool agent, ask: Which call are we trying to improve? Did the model have the needed tool, context, and authority at that moment? Does our reward isolate that call’s contribution, or are we measuring downstream noise? If we train this state, what regression suite proves we helped? If the problem is not trainable, is it actually a product, permission, retrieval, or workflow-design issue?

That checklist will save teams from a lot of expensive prompt churn.

The next generation of reliable agents will not come only from larger models or longer traces. It will come from narrower corrections. Better agents will be trained where they actually break.