Your Cybersecurity Agent Needs Command Evals Before It Gets a Shell

The scariest failure mode for a cybersecurity agent is not always a dramatic hallucination. Sometimes it is one plausible command with one wrong flag.

That is what makes command-level evaluation worth taking seriously. Security teams are not asking agents to write poetry about incident response. They are asking them to translate analyst intent into tool calls, shell commands, filters, scans, reports, and follow-up actions. In that environment, "mostly right" is a weak standard. A command can use the right tool family and still bind the wrong argument, scan the wrong scope, omit the evidence-producing option, or produce output that looks authoritative while answering a different question.

A recent benchmark called KaliBench gives this problem a sharper shape. The project evaluates natural-language-to-command translation for Kali Linux tools, using 8,504 verified query-command pairs across 1,642 tools, 23 capability dimensions, and 5 security phases. Its authors frame the gap directly: many existing evaluations focus on knowledge or broad end-to-end tasks, while operational cybersecurity work depends on the exact construction of executable commands. KaliBench separates tool selection from argument construction so teams can see where command generation actually breaks.

That distinction matters for anyone building or buying security agents.

An end-to-end demo can hide a lot. The agent receives a task, thinks aloud, runs a tool, prints something convincing, and declares progress. If the environment is toy-sized, the credential scope is fake, and a human is watching closely, the demo may be useful. But it does not answer the production question: can this system reliably construct the command it is about to run, with the right tool, right options, right target, right output expectation, and right safety boundary?

KaliBench is useful because it treats that question as measurable. The project page says the dataset is grounded in official tool documentation, checked through sandboxed execution and human review, and split into training and held-out evaluation data. The benchmark also uses deterministic canonicalization and alias-aware scoring, which is important because command lines have many equivalent spellings. A serious eval cannot reward only one string if two commands are semantically identical, but it also cannot wave away incorrect option binding as a formatting issue.

The reported results are a useful warning label, not a reason to panic. In the unrestricted setting described in the arXiv abstract, no evaluated open-weight model exceeded 42% exact-command accuracy. That does not mean every security workflow is doomed, and it does not mean a proprietary model, constrained tool menu, or retrieval-augmented agent would perform the same way. It means the hard part is not just "knowing cybersecurity." The hard part is turning an ambiguous request into an executable, scoped, auditable command without silently changing intent.

For enterprises, the lesson is straightforward: do not give a cyber agent a shell just because it can pass a chat evaluation.

Build a command-eval harness first.

The harness should score tool selection separately from argument construction. If the model chooses the wrong tool, that is one class of failure. If it chooses the right tool but misplaces a flag, drops a required value, or changes the target scope, that is another. Those failures deserve different remediation paths. Tool-selection errors may point to missing documentation, weak retrieval, or poor task classification. Argument-construction errors may require structured schemas, constrained decoding, examples, or a tool-specific compiler layer.

The harness should also evaluate scope. Security agents operate near sensitive systems, so the command is not complete until the target boundary is explicit. A safe system should be able to explain whether it is acting on a lab host, a customer tenant, a production subnet, a single artifact, or a read-only log index. If the request is underspecified, the right answer may be to ask for scope, not to invent one.

A mature harness also needs sandbox validation. KaliBench highlights the value of execution checks and deterministic signals. Enterprise systems can adapt that idea without running dangerous commands against real infrastructure. Generate the candidate command, canonicalize it, compare it against expected patterns, test it in a sandbox or dry-run environment when possible, and require a receipt before production execution. The receipt should capture the user intent, selected tool, normalized command, scope, approval state, and expected output. If the agent cannot produce that receipt, it should not run the command.

This is where governance frameworks become practical rather than ceremonial. OWASP's GenAI security work is useful because it frames agentic AI as part of the security surface, not a novelty bolted onto it. NIST's AI Risk Management Framework is useful because it reminds teams to map, measure, manage, and govern risk continuously. A command-eval harness turns those ideas into an engineering control. It gives the security organization something concrete to measure before a model, prompt, tool wrapper, or documentation bundle changes.

The same harness should run after every meaningful change. New model version? Re-run the command suite. Updated tool documentation? Re-run it. Added a new connector? Re-run it. Changed the prompt that decides whether the agent asks for approval? Re-run it. In production, most agent failures will not arrive as clean research-paper categories. They will show up as regressions after someone improves context, adds a shortcut, or updates a dependency.

The goal is not to create a leaderboard for its own sake. The goal is to create a release gate.

A useful release gate can be blunt. For example: the agent cannot run unrestricted shell commands; high-risk tools require restricted or hinted modes; commands touching production assets require human approval; commands with ambiguous targets fail closed; and any command that cannot be normalized, explained, and logged is blocked. That may sound less glamorous than a fully autonomous SOC agent, but it is far closer to how reliable automation earns trust.

It also changes how teams should interpret agent demos. The impressive question is not, "Can the agent open a terminal and do a task?" The impressive question is, "Can the agent prove that the command it is about to run matches the authorized intent?" If the answer is yes, the organization gets a path toward controlled autonomy. If the answer is no, the demo belongs in a lab, not in an incident channel.

Cybersecurity agents will keep improving. They will get better tool documentation, better retrieval, better constrained generation, and better training signals. KaliBench's runtime-free verifiable reward framing points toward that future. But the operational takeaway is already available: exact command construction is a safety boundary.

Before the agent gets a shell, make it pass the command eval.