Stop Rewarding Decorative Tool Use in AI Agents

A tool call is not evidence just because it appears in an agent trace.

That distinction matters more as AI agents become multimodal. The modern demo often looks convincing because the model does visible work: it crops an image, zooms into a screenshot, calls OCR, runs a notebook, searches a document store, or executes code. The transcript feels grounded. The problem is that the final answer may not actually depend on the artifact the tool produced. The tool call can become stagecraft: useful for the audience, expensive for the system, and misleading for anyone trying to audit what happened.

A cluster of new visual-agent research makes that failure mode easier to name. The lesson is not limited to vision models. It applies to every workflow where an agent can perform side-effect-free “reasoning support” actions and then produce a confident answer. If an enterprise team only rewards final-answer accuracy, or only checks whether tools were invoked, it can accidentally train and approve agents that look operationally rigorous while skipping the harder discipline of evidence.

FaithEyes, a July 2026 paper on agentic vision-language models, frames the issue directly. Agentic VLMs can interleave reasoning with explicit tools such as cropping and code-based image manipulation. But the paper argues that tool use can be unfaithful: the model may crop the wrong region, miss the queried target, or generate process images that are irrelevant to the question, while still arriving at a correct answer. In that case, the system receives credit for a tool-assisted trajectory even when the assistance did not help.

The practical idea inside FaithEyes is simple and valuable: judge whether each process image helps answer the question. That judgment is fed back into the reasoning context and also affects reward through a helpful-tool ratio. In other words, the tool call has to earn its place. The model should not get the same credit for a decorative crop that it gets for a crop that exposes the specific evidence needed to answer.

For production teams, that suggests a useful shift in telemetry. Do not just log “tool X was called.” Log what claim the tool output was supposed to support, whether it plausibly supported that claim, and whether later reasoning cited it. A trace that says an agent opened a patient record, fetched a policy, or inspected a screenshot is incomplete unless it also records why that action mattered.

Beacon, another July 2026 visual-reasoning paper, adds a second control: necessity. The authors separate tool use into two dimensions: Mode Adaptiveness, or whether the model knows when tools are needed, and Tool Effect, or whether the tools actually improve performance rather than introduce new mistakes. Their analysis warns that tool gains on hard examples can be offset by harm on easy examples that the model could already solve without tools.

That is a familiar production pattern. Teams add tools because tools make agents feel more capable. Then every easy question becomes slower, more expensive, and occasionally less reliable because the agent insists on consulting machinery it did not need. A retrieval call can pull in distracting context. A browser action can observe stale UI state. A code interpreter can transform a simple arithmetic or formatting task into an unnecessary execution path. More tools do not automatically mean more reliability.

The enterprise control is to test the negative case: when should the agent not call the tool? Evaluation suites should include easy tasks where tool use is unnecessary, ambiguous tasks where a clarifying path is safer, and hard tasks where tool use should measurably improve the result. If the agent uses a tool, the trace should show both the expected necessity and the observed effect.

LedgerMind, a third paper from the same late-July wave, points toward the architecture behind that control. It argues that final-answer accuracy cannot tell whether a multimodal agent reached a correct answer through grounded evidence, language priors, or accidental error cancellation. Its proposed pattern is a structured evidence ledger: tool outputs become normalized evidence entries; downstream claims may cite only active ledger entries; grounding is checked at entity and numeric levels; repair happens through typed state transitions that cannot introduce unsupported content.

The phrase “evidence ledger” is useful because it moves agent reliability out of vibes and into data modeling. A tool output is not a blob in a transcript. It is a record with scope, type, freshness, source, and claim relationships. A repair step is not a new paragraph of model reasoning. It is a transition from one evidence state to another. A final answer is not merely fluent. It is a set of claims that can be traced back to active evidence.

This is where the research connects to agent operations. OpenAI’s Agents SDK documentation emphasizes tracing of agent runs, including artifacts such as generations, tool calls, guardrails, and handoffs. OpenTelemetry’s GenAI semantic conventions show the broader push toward standard observability for generative-AI systems. Those traces are necessary, but they are not sufficient. A trace full of tool spans can still hide decorative tool use unless the spans carry evidence semantics.

A practical implementation does not need to be elaborate. Start with four fields on every high-value tool call: intended claim, evidence object, usefulness judgment, and downstream claim reference. For example: “OCR invoice total,” “image crop at coordinates X,” “useful because it contains the queried total,” and “cited by final billing summary claim.” For retrieval: “find policy exception,” “document chunk ID,” “useful because it states the exception criteria,” and “cited by authorization recommendation.” For code execution: “verify transformation,” “test output hash or assertion result,” “useful because it falsified the prior assumption,” and “cited by deployment note.”

Then add release gates that punish decorative behavior. Sample traces where the answer is correct but the evidence is irrelevant. Review cases where tools were called and not cited. Track the ratio of necessary to unnecessary tool calls. Mark “correct answer, unsupported path” as a failure class, not a success. The point is not to bury teams in compliance paperwork. The point is to make the agent’s reasoning path inspectable enough that a reviewer can tell whether the system earned its confidence.

Anthropic’s guidance on building effective agents has repeatedly favored simple, transparent workflows over unnecessary complexity. Tool-faithfulness controls fit that philosophy. They do not require turning every agent into a giant orchestration graph. They require small contracts at the boundary where the agent asks the world for help: what are you trying to prove, what did the tool return, did it help, and what claim now depends on it?

The next generation of dependable agents will not be defined by how many tools they can call. It will be defined by whether each tool call leaves behind a usable proof trail. Every crop, retrieval, screenshot, command, and calculation should earn its place in the trace. If it cannot, the system should treat it as noise — even when the final answer happens to be right.