Your Agent Eval Needs an Evidence Lake, Not Another Leaderboard

Most AI teams can tell you which model looked best in the last demo. Fewer can tell you why it looked better, whether the scaffold changed, which verifier judged the run, how much the improvement cost, or whether the same behavior survived a rerun two weeks later.

That gap is becoming the real bottleneck in agent deployment.

For chat-style models, a benchmark score is already an imperfect summary. For agents, it is even thinner. The model is only one part of the system. Tool schemas, browser state, retrieval quality, retry logic, permissions, workspace files, handoff rules, and evaluator design all shape the final behavior. If those ingredients are not preserved, the result is not evidence. It is a memory of a performance.

A recent arXiv paper, “Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation,” points toward a more useful direction. The authors describe the current evaluation landscape as fragmented across tasks, scaffolds, verifiers, and scoring rules. Their response is not simply another benchmark. It is a unified corpus: 957,253 records spanning 30 benchmarks, 714 agents, 11,891 tasks, and 74,205 verifiers. The important word is not “large.” It is “records.”

That is the shift enterprise teams should pay attention to. Agent evaluation is moving from leaderboard thinking to evidence infrastructure.

The leaderboard is too flat for agent work

A leaderboard compresses a run into a rank. That can be useful for scouting. It is not enough for release decisions.

Imagine two coding agents with the same pass rate. One uses a narrow tool surface, clean diffs, and deterministic tests. The other edits broadly, retries until something sticks, and leaves a higher cleanup burden. On a flat scoreboard, those systems may look equivalent. In production, they are not equivalent at all.

The same problem shows up in enterprise automation. A browser agent that completes a procurement workflow once may fail when the page changes, a session expires, or an approval step appears. A data agent may answer correctly on a test query while silently using the wrong table in a similar real request. A support agent may resolve an easy ticket and mishandle the escalation boundary that actually matters.

The missing artifact is a structured record of the run: what task was attempted, what version of the agent attempted it, what tools it could use, what the environment looked like, what verifier judged the outcome, what trace was produced, what it cost, and what changed from the last run.

Without that record, teams are left comparing anecdotes.

What an agent-evaluation evidence lake stores

An evidence lake does not need to be fancy at first. It needs to be disciplined.

At minimum, every evaluation run should preserve:

• the task identifier, task family, and dataset or scenario version;
• the model, agent scaffold, prompt, policy, and tool-contract versions;
• the verifier or grader used, including its version and failure modes;
• the trajectory: tool calls, observations, retries, errors, and handoffs;
• the result, including partial success and unsafe near-misses;
• latency, token use, external service cost, and human review time;
• links to artifacts such as diffs, screenshots, logs, or generated files;
• governance notes: approval status, release gate, known caveats, and owner.

This sounds operational because it is. Agent reliability is not only a modeling question. Anthropic’s practical writing on building effective agents makes a similar point from the system-design side: useful agents depend on clear workflows, well-defined tool use, and appropriate orchestration. If the scaffold is part of the system, then the scaffold has to be part of the evaluation record.

This is where agent evaluation differs from a clean multiple-choice benchmark. The same model can behave differently when the tool contract changes. The same task can become easier or harder when the verifier changes. The same pass rate can hide a cost regression, a permission regression, or a trace-quality regression. A useful evaluation system has to keep those dimensions visible.

Messier is a signal, not a finished operating model

Messier matters because it treats cross-benchmark comparability as a data problem. The paper’s corpus joins public benchmark results with additional runs across underrepresented professional and scientific domains. That kind of normalization gives researchers and builders a way to ask better questions: which task families changed, which verifiers disagree, where do agents improve without rerunning everything from scratch?

Enterprise teams should not copy any research corpus blindly. Internal evaluation has different constraints: private workflows, compliance requirements, vendor contracts, local tools, and domain-specific risk. But the pattern transfers well.

Do not just ask, “Did our agent pass the benchmark?” Ask:

• Did it improve on the task families our business actually uses?
• Did the improvement survive the same verifier and environment?
• Did cost or latency rise enough to erase the operational gain?
• Did the trace become easier or harder for a human to audit?
• Did the agent require broader permissions to get the same result?
• Did the failure mode move from obvious errors to silent corruption?

Those are evidence-lake questions, not leaderboard questions.

Borrow the standardization instinct from HELM

Stanford CRFM’s HELM is not an agent benchmark, but it is useful precedent. Its core lesson is that evaluation becomes more valuable when scenarios, metrics, and reporting are standardized enough to compare results responsibly.

Agents need the same instinct with more metadata. A model benchmark can often record prompt, answer, and score. An agent run needs the surrounding machinery: tool schemas, state, verifier, control loop, permissions, and artifacts. The run is not just text in and text out. It is a small operational event.

That framing also connects to governance. NIST’s AI Risk Management Framework emphasizes measurement, documentation, monitoring, and risk management. An evidence lake gives those ideas a concrete place to live. It becomes the shared substrate for release reviews, vendor comparisons, regression analysis, audit response, and post-incident learning.

The point is not bureaucracy. The point is memory.

A small-team version is enough to start

You do not need a warehouse team to begin. Start with a table and a folder.

For every eval run, create a stable run ID. Store the metadata in JSON. Save the trace and artifacts under that ID. Record the verifier version. Add a short human review field for “why this passed,” “why this failed,” and “what would make this unsafe in production.” Track cost and latency from day one, even if the numbers are rough.

Then make the comparison views boring and useful:

• pass rate by task family;
• regressions by scaffold version;
• cost per successful task;
• failures by verifier;
• unsafe near-misses by permission scope;
• human review time by workflow;
• trace completeness by agent version.

That is where the operational value appears. The evidence lake turns agent evaluation into a system of record. It lets leaders ask whether a new model actually helped, whether a vendor claim survives local tasks, whether a safety control created a bottleneck, and whether last month’s “improvement” was just a verifier change.

The next eval artifact is the record

The agent world does not need fewer benchmarks. It needs fewer orphaned benchmark results.

A single score can still be useful as a signal. But if the score cannot be traced back to the task, scaffold, tool surface, verifier, trajectory, cost, and environment, it should not carry much weight in a production decision.

Messier is a useful reminder that evaluation maturity often looks like better recordkeeping before it looks like better dashboards. For enterprise AI teams, that is the next practical step: build the evidence layer that lets every agent run become comparable, replayable, and reviewable.

The leaderboard can tell you who looked good today. The evidence lake tells you what changed, what broke, what improved, and whether you can trust the result tomorrow.