Your Agent’s Tool Calls Are Becoming Code

JSON tool calls made AI agents practical. They gave teams a clean way to ask a model for structured arguments, validate those arguments, run application code, and send the result back. For a lot of production work, that is still the right pattern. It is boring in the best possible way: explicit schemas, predictable handlers, easy logs, and familiar API boundaries.

But the next layer of agent reliability may not come from making that form-filling loop slightly prettier. A new arXiv paper, “The Bitter Lesson of Tool Calling,” argues for a more direct approach: expose tools as typed Python stubs and let capable models call them through code. The paper calls this programmatic tool calling. Instead of asking the model to emit one JSON object at a time, the model can write a small program that chains tools, fans out calls, handles intermediate values, and returns a result.

That distinction sounds academic until you map it onto a real workflow. A sales agent rarely needs one isolated CRM lookup. It may need to search accounts, filter records, compare two plans, fetch policy constraints, draft a response, and stop if a permission boundary is crossed. A healthcare operations agent may need to check eligibility, find order context, compare a scheduling rule, and produce an auditable recommendation without touching anything it is not allowed to touch. JSON tool calling can do this, but the orchestration often lives awkwardly between the model, the application loop, and a growing pile of glue code.

The arXiv result is interesting because it tests this representation choice on BFCL v4, not on a toy prompt. The Berkeley Function Calling Leaderboard was built to evaluate whether LLMs can call functions and tools across agent and enterprise-style use cases. Its public description includes parallel calls, multiple calls, several languages, function relevance detection, cost, latency, and later versions that added enterprise-contributed, dynamic, multi-turn, and multi-step scenarios. In that setting, “The Bitter Lesson of Tool Calling” reports that programmatic tool calling matched or exceeded native JSON tool calling in 11 of 14 models. It also reports stronger behavior under parallel fan-out, where it matched or outperformed the baseline in 13 of 14 models, and more stable behavior under context-rot conditions where the JSON baseline degraded on average.

The lesson is not “throw away JSON.” The lesson is that tool interfaces are becoming execution architecture.

Most teams still talk about tool calling as if it is an API formatting problem. Define a schema. Add a tool description. Let the model decide when to call it. Anthropic’s documentation frames tool use this way: Claude determines when to call a tool from the user request and tool description, then returns a structured call that either the application executes or Anthropic executes. OpenAI’s function-calling docs similarly describe function calling as a way for models to connect to external data and systems. That is the mainstream mental model, and it has served the industry well.

Programmatic tool calling bends that model. If the model can compose typed stubs in code, the interface stops being a single request envelope and starts becoming a small execution surface. That can be more natural for tasks with loops, branches, retries, parallel work, and local transformations. It can also be easier for strong coding models, because they already know how to express procedural intent in code.

For builders, the opportunity is obvious: less brittle orchestration, fewer awkward “call one tool, wait, re-prompt, call another tool” loops, and better use of parallelism when a task naturally splits. The risk is just as obvious: code-shaped tool use can blur the line between planning and acting.

If an agent can write the orchestration, you need to be much more explicit about the box it writes inside. Typed stubs are only the beginning. The runtime should still enforce least privilege. The tool layer should know which tenant, user, patient, project, or account the agent is allowed to touch. Expensive fan-out should have budgets. Side-effecting calls should require preflight checks, idempotency keys, or human approval. The execution trace should preserve not only final tool inputs, but the small program, intermediate values, tool outputs, denied calls, retries, and stop reasons.

That trace is the difference between “the agent did something weird” and “we can replay the exact branch where it exceeded the intended scope.” Programmatic tool calling without replay is just a cleverer black box.

The practical architecture is not complicated, but it has to be intentional. Start with a narrow namespace of typed tools. Generate stubs from the same source of truth as your production API contracts. Run the agent’s program in a sandbox with no ambient credentials. Attach a policy engine at the tool boundary, not only in the prompt. Set budgets for calls, time, tokens, rows returned, and side effects. Log enough to replay. Then promote workflows gradually: first read-only, then reversible writes, then high-confidence writes with approval, and only then autonomous action in low-risk lanes.

That approach also keeps JSON tool calling in the toolbox. For many workflows, a single structured call is safer, cheaper, and easier to certify. A password reset, address lookup, or appointment search may not need a generated mini-program. The point is not to make every tool call look like software development. The point is to notice when the workflow already is software development, just hidden inside repeated model turns.

The deeper shift is cultural. Agent teams should stop treating tool schemas as prompt accessories. They are control surfaces. A JSON schema controls shape. A typed programmatic interface controls shape plus composition. That extra power has to be paid for with stricter runtime boundaries and better evidence.

So if programmatic tool calling keeps winning benchmarks, the production response should not be panic or hype. It should be a cleaner contract: yes, the agent may compose tools as code, but only through typed stubs, inside a sandbox, under policy, with budgets, traces, and replay.

The future of agent tool use may look less like filling out a form and more like executing a tiny workflow. That can be a real reliability upgrade — but only if the workflow is governed like production code, not trusted like a chat response.