Your Agent Eval Has to Cross the API/Knowledge Boundary

The easiest agent demo to approve is also the easiest one to misunderstand.

A user asks for something. The agent calls a tool. The final answer looks right. The benchmark records a pass. Everyone moves on with a pleasant green checkmark.

But most useful enterprise agents do not live inside a single clean tool call. They have to combine structured system state with messy policy text, retrieved documents, user context, authorization boundaries, and sometimes the uncomfortable fact that the correct answer is “do not act.” If the eval never crosses those boundaries, it is not measuring the product you are about to ship.

That is why VAKRA, a new benchmark paper titled “Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies,” is worth paying attention to. The important lesson is not that every team should copy the benchmark wholesale. The lesson is that agent readiness needs to be tested where agents actually break: between APIs, knowledge, and policy.

VAKRA is built around a practical observation. Enterprise agents often need both structured API interaction and document-grounded reasoning. A support agent may need account status from an API, cancellation rules from a help article, and permission constraints from an internal policy. A procurement agent may need supplier records, contract language, and exception handling rules. A healthcare operations assistant may need schedule data, user role boundaries, and documented workflow policy.

Testing any one of those in isolation is useful, but incomplete.

According to the arXiv abstract, VAKRA includes more than 8,000 executable APIs across 62 domains. Its tasks span increasingly difficult settings: diverse API interaction styles, multi-hop reasoning over structured APIs, and multi-source reasoning with natural-language tool-use policy constraints. The benchmark also verifies correctness by re-executing predicted tool calls against live APIs, while allowing more than one valid path.

That last point matters. For production agents, the trace is not decoration. The trace is the behavior. It determines cost, latency, data exposure, auditability, and the exact failure mode when something goes wrong. A final answer can sound polished while the agent used the wrong identifier, skipped a policy constraint, queried an unnecessary source, or took an action it should have abstained from taking.

VAKRA’s reported results reinforce the gap. In a fixed ReAct harness, the best model reaches 70.4% on single-hop endpoint-style tasks, but drops to roughly 50–51% on compositional APIs. The abstract also reports that performance degrades by more than 50% as reasoning depth increases, and that policy-constrained questions reveal severe failures, including results as low as 2.4% on unanswerable queries.

Those numbers should make builders uncomfortable in a useful way. They suggest that “can call the tool” is not the same as “can operate the workflow.” The agent may understand the endpoint schema and still fail to resolve the right entity. It may retrieve a relevant document and still fail to apply the exception. It may produce an answer when the policy says the available evidence is insufficient.

The paper’s trace analysis points in that direction too: failures concentrate around language-mediated reasoning such as entity disambiguation and cross-source grounding, not merely around the mechanics of invoking a tool. That is exactly the seam many production evals under-test.

A lot of internal agent scorecards still look like unit tests for tools. They ask: did the model call the right function? Did it return the right JSON shape? Did the happy path complete? Those checks are necessary, but they are not enough. They prove the socket works. They do not prove the agent can decide what the socket is allowed to do after reading a policy page and reconciling three pieces of state.

A better internal eval starts by designing tasks that force boundary crossing.

Do not only ask the agent to “look up customer status.” Ask it to look up customer status, retrieve the relevant policy, notice that the user’s case falls under an exception, and choose the least privileged permitted action. Do not only test whether the agent can update a record. Test whether it refuses when the retrieved document and the API state conflict. Do not only grade the final paragraph. Re-execute the tool calls where possible, inspect the selected records, and score whether the trace used the minimum necessary data.

This is also where unanswerable cases belong. In real systems, agents frequently encounter missing records, stale documents, ambiguous names, insufficient permissions, and policy conflicts. If the eval set contains only solvable tasks, it teaches a dangerous habit: keep going until a plausible answer appears. VAKRA’s policy-constrained failures are a reminder that abstention is not a nice-to-have behavior. It is part of the operating envelope.

There is a governance angle here, but it should stay concrete. The Model Context Protocol authorization specification is one example of the industry treating tool access as a formal boundary rather than a bag of callable functions. NIST’s AI Risk Management Framework gives teams a broader vocabulary for mapping, measuring, managing, and monitoring AI risk. Benchmarks like VAKRA do not replace those practices. They can, however, produce better evidence for them.

For an engineering team, that evidence might look like a release packet with four artifacts.

First, a task set that reflects actual workflow joins: API state, retrieved documents, user role, and policy text. Second, executable scoring that checks the tool trace instead of trusting the final answer. Third, a failure taxonomy that separates endpoint misuse from entity confusion, grounding failure, policy misread, and improper abstention. Fourth, a regression loop that keeps those cases alive as prompts, models, tools, and policies change.

That last artifact is easy to skip. Agent systems drift. APIs change. Documentation changes. A model upgrade can improve final-answer fluency while weakening a policy-sensitive branch. A retrieval change can make the right document harder to find. A permission change can silently expand what the agent is able to do. If the eval does not preserve cross-boundary cases, the team may not notice until a customer, auditor, or incident report does the testing for them.

There is also a product design implication. If your agent cannot reliably join API state and policy evidence, the answer is not always “bigger model.” Sometimes the answer is a narrower workflow, better identifiers, explicit policy objects, pre-action checks, or a UI that asks the human to resolve ambiguity before the tool call. Evaluation should make those design choices visible. A benchmark that only returns a pass rate hides the lever you need to pull.

The practical takeaway is simple: stop treating agent evals as chatbot exams with tools attached. A production agent is a decision system operating across boundaries. Its eval should cross those boundaries too.

Before shipping, ask three questions.

Can the agent combine structured API state with retrieved knowledge without losing the thread? Can it apply natural-language policy constraints when the happy path is tempting but wrong? Can it abstain, escalate, or ask for clarification when the evidence does not support action?

If the answer is unknown, the system is not ready. It may be impressive. It may be useful in a sandbox. But it has not earned the green checkmark that matters: evidence that it can operate the workflow, not just perform the demo.