The Bitter Lesson of Tool Calling

Technical Summary: The Bitter Lesson of Tool Calling

Problem Statement
Large Language Models (LLMs) increasingly function as agents that invoke external tools via APIs. The current industry standard requires models to emit structured JSON objects for each function call. While prior theoretical work (e.g., CodeAct) suggests that executable code offers advantages in multi-tool scenarios, a systematic empirical evaluation comparing programmatic tool calling (PTC) against native JSON tool calling has been lacking. Specifically, it remains untested whether PTC can match or exceed JSON performance across diverse model generations, particularly under real-world constraints such as sequential chaining, high parallel fan-out, and adversarial context loads.

Methodology
The authors conducted a controlled evaluation on a representative 309-entry subset of the BFCL v4 benchmark, spanning eight task categories. The study involved 14 language models from Anthropic and OpenAI, released between November 2024 and July 2026.

Two primary paradigms were compared:

1. JSON Tool Calling: The standard approach where the model emits structured JSON objects via an API endpoint.
2. Programmatic Tool Calling (PTC): The model is provided with typed Python stubs corresponding to the function schemas. It writes a Python script to import these stubs and execute the necessary calls. The script is run in a native shell subprocess, capturing stdout for scoring. This design ensures that PTC and JSON calling consume the same number of LLM inference turns per entry.

The evaluation included three specific ablation studies targeting known limitations of JSON tool calling:

• Chaining: Sequential multi-hop calls (chain lengths 2–20).
• Parallelism: Independent fan-out calls (counts 7–48, with probes up to 100).
• Context Rot: A "flood" condition injecting 128 total function schemas (including decoys) to test robustness against context noise.

Accuracy was measured deterministically by matching the model's output against ground-truth function calls, stripping punctuation and lowercasing values.

Key Results
The study found that PTC is a viable and often superior alternative to JSON tool calling, with performance heavily correlated to model generation rather than model family.

• Overall Performance: On the main BFCL v4 subset, 11 of the 14 models matched or exceeded the accuracy of JSON tool calling when using PTC. The GPT-5.6 family achieved the largest gains, with GPT-5.6-Sol and GPT-5.6-Terra showing absolute improvements of 10.6% over their JSON baselines.
• Model Generation Divide:
  • Anthropic Models: All five tested models (Haiku 4.5 through Sonnet 5) matched or exceeded JSON baselines across all conditions.
  • OpenAI Models: The three newest GPT-5.6 variants outperformed JSON baselines. However, three older models (GPT-4o, GPT-4.1, and GPT-5.4-mini) performed significantly worse under PTC (drops of 19.7% to 26.9%). The authors attribute this to a specific failure mode where these models output literal \n escape sequences instead of actual newlines in multi-line scripts, causing syntax errors in the subprocess.
• Chaining Ablation: PTC demonstrated a distinct advantage in sequential tasks, with the accuracy gap widening as chain length increased. At chain lengths ≥12, PTC achieved an 18.8% absolute advantage over JSON. This is attributed to PTC resolving calls in a single turn using parametric knowledge for intermediate values, whereas JSON requires an additional inference turn per link.
• Parallelism Ablation: PTC matched or outperformed JSON in 13 of 14 models. Notably, JSON tool calling exhibited a hard structural limit for Anthropic models; Claude Sonnet 5 began dropping calls at a fan-out of N=72, reaching 0% enumeration accuracy at N=100. PTC maintained 100% enumeration accuracy at N=100.
• Context Rot Ablation: Under context flooding, PTC remained stable, showing an average absolute improvement of 5.5% compared to the filtered condition. In contrast, JSON tool calling degraded by an average of 2.3%. A filesystem-based discovery approach (used as a reference) degraded by 32%, highlighting PTC's robustness to adversarial context loads.

Significance and Claims
The paper claims to provide the first systematic, standardized benchmark evaluation of programmatic tool calling against native JSON tool calling across multiple model families and generations.

The authors conclude that:

1. Viability: Programmatic tool calling is a robust alternative to JSON, matching or exceeding baseline accuracy in the majority of tested models.
2. Generation Dependency: The success of PTC is a function of model generation capability. Newer generations (specifically GPT-5.6 and recent Anthropic models) have internalized the ability to generate correct multi-line code, whereas older generations struggle with formatting constraints (e.g., newline handling).
3. Structural Advantages: PTC offers inherent structural benefits for complex tasks. It eliminates the latency of sequential turns in chaining and removes the token-count and serialization limits that cause JSON tool calling to fail at high parallel fan-out.
4. Practical Implication: As model generations advance, the "bitter lesson" suggests that relying on rigid JSON structures may be suboptimal for capable code-writing models, which can leverage programmatic execution to achieve higher fidelity in tool use.

Limitations
The authors note four primary limitations:

1. The benchmark uses echo-return stubs rather than real API calls, measuring argument serialization accuracy rather than end-to-end execution correctness.
2. Ablation study sample sizes are small (n=31–52), meaning individual model results should be viewed as directional rather than statistically conclusive.
3. The benchmark's ground truth may contain noise, as previous audits have noted misalignment between LLM judges and human evaluators.
4. PTC incurs a fixed input-token overhead due to the inclusion of full Python stubs in the system prompt, which can be less efficient than JSON for low fan-out tasks, though this reverses at high fan-out.