Data Agents Need Workflow Evals, Not Text-to-SQL Scores

The easiest way to overestimate a data agent is to watch it write a clean SQL query.

The prompt looks reasonable. The generated query joins the right-looking tables. The answer arrives in a confident paragraph. For a demo, that can feel like the whole game. For an enterprise analytics workflow, it is closer to the opening move.

Real data work is rarely “turn this sentence into one query.” It is a sequence: understand the business question, inspect the available data, decide which fields are trustworthy, handle missingness, test assumptions, run the analysis, explain uncertainty, prepare a decision, and leave enough evidence that another person can reproduce or challenge the result. A data agent that only gets scored on SQL syntax can pass the quiz and still fail the job.

That is why the new Argo-Bench paper is a useful signal. The authors describe enterprise analytics as work that spans dozens of tables, statistical analysis, and action-oriented conclusions. Their proposed benchmark contains 210 data science and analytics tasks, motivated by the gap between public benchmark settings and sensitive real enterprise warehouses. The important idea is not that one benchmark settles the field. It is that the evaluation target is shifting from query generation to workflow completion.

That shift matters because many teams are about to make the same mistake with data agents that they made with chatbots: they will test the part that is easiest to score, then deploy the system into the part that is hardest to observe.

Text-to-SQL benchmarks still matter. Projects like BIRD helped push the field beyond toy schemas and toward more realistic database reasoning. If a model cannot reliably map a question to the right tables, filters, joins, and aggregation logic, it is not ready to operate inside a data platform. Query-generation skill is a real dependency.

But it is a dependency, not a release gate by itself.

A production data agent needs a layered evaluation harness. At the bottom are unit tests: can the agent produce valid SQL, call the right tools, respect schema constraints, and avoid obviously unsafe operations? Above that are task tests: can it answer a realistic analytical question from messy data and show its work? Above that are workflow tests: can it move from ambiguous request to reviewed recommendation while preserving the evidence trail? And above that is operational monitoring: when the system fails, can the team tell whether the failure came from retrieval, schema interpretation, statistical reasoning, stale data, tool execution, or a bad handoff?

The workflow layer is where many “good” agents become dangerous. A query can be correct and still answer the wrong question. A dashboard number can be accurate and still be irrelevant to the decision. A statistically significant result can be presented without the caveats that determine whether anyone should act on it. A data agent can cite a table, skip the data-quality issue, and hand a manager a conclusion that looks finished because the prose is polished.

That is the core evaluation problem: enterprise data agents do not just produce answers. They produce decisions-in-waiting.

A better evaluation starts by scoring decomposition. Given a business request, does the agent break the job into auditable steps? Does it identify which tables, metrics, time windows, and assumptions are required before it starts executing? Does it ask for clarification when the question is underspecified, or does it silently choose a convenient interpretation?

Next, score grounding. The agent should be able to explain why a table or field is relevant, where the metric definition came from, and what it did when two sources disagreed. This is especially important in enterprise environments where semantic meaning often lives outside the warehouse: in finance definitions, sales operations rules, reimbursement logic, support taxonomies, or old analyst notebooks. A data agent that cannot bind its output to the organization’s source of truth is not doing analytics. It is narrating database access.

Then score statistical behavior. Did the agent choose an appropriate comparison? Did it distinguish correlation from causation? Did it surface sample size, missing data, outliers, and uncertainty? Did it explain the difference between a directional signal and a decision-ready result? These checks are not academic decoration. They are the line between “the model found a pattern” and “the business should change behavior.”

The fourth layer is reproducibility. Every serious data-agent run should leave a trail: prompt or task, retrieved context, selected schema, generated queries or code, outputs, transformations, assumptions, intermediate artifacts, and final recommendation. If the result cannot be replayed or inspected, the organization has created an analytics black box with a friendly interface.

The fifth layer is handoff quality. The agent’s final answer should not only say what it found. It should say what a human reviewer needs to verify before action, what decision the result supports, which caveats matter, and which downstream systems would be touched if the recommendation becomes execution. This is where agent evaluation connects to governance. NIST’s AI Risk Management Framework is not a data-agent benchmark, but its measure-map-manage mindset is exactly the posture teams need: define the context, measure the system, manage the risk, and keep monitoring after deployment.

This architecture also matches the practical agent-building advice coming from teams shipping real systems. Anthropic’s guidance on effective agents emphasizes workflows, tools, feedback, and clear orchestration patterns rather than treating the model as a magical monolith. For data agents, that means the harness is part of the product. The benchmark is not a spreadsheet of final answers; it is a controlled environment where the agent must use tools, preserve evidence, recover from ambiguity, and make its uncertainty visible.

The practical takeaway is simple: keep text-to-SQL tests, but demote them to unit tests.

A serious data-agent release checklist should include questions like these:

• Can the agent explain the business metric before querying it?
• Can it identify when a request needs clarification?
• Can it work across multiple related tables without flattening the problem into a convenient shortcut?
• Can it run or request the right statistical checks?
• Can it produce a replayable evidence packet?
• Can reviewers see which assumptions changed the conclusion?
• Can the system fail safely when the data is stale, incomplete, or contradictory?
• Can the final recommendation be separated from the raw analysis so humans know what is evidence and what is proposed action?

That last distinction is easy to miss. Many analytics agents blur analysis and execution. They answer the question, draft the email, update the dashboard, open a ticket, or recommend a campaign change in the same fluent motion. The more capable the agent becomes, the more important it is to know where analysis ends and authority begins.

Argo-Bench is interesting because it points evaluation toward that messier reality. Enterprise data work is not a single SQL translation problem. It is a workflow with dependencies, uncertainty, and consequences. The benchmark conversation needs to follow the work.

The teams that get this right will not ask, “What is our model’s text-to-SQL score?” as the primary release question. They will ask, “Can this agent complete the analyst workflow in a way we can inspect, reproduce, govern, and safely act on?”

That is a harder question. It is also the one that matters.