🤖 AI

When Does Restricting a Coding Agent to execute_code Help? A Regime $\times$ Agent-Design Ablation

This paper demonstrates that restricting coding agents to a single `execute_code` tool is often as effective as and frequently cheaper than using tool-rich environments, revealing that the optimal tool surface is jointly determined by the interaction between task regimes and specific agent designs rather than by either factor alone.

Original authors: Hong Yang, Qi Yu, Travis Desell

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Hong Yang, Qi Yu, Travis Desell

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot assistant whose job is to fix broken code or solve math puzzles. Right now, there's a huge debate in the tech world about how this robot should talk to the computer to get the job done.

Some say, "Give it a full toolbox with special buttons for every task!" (Like a fancy IDE).
Others say, "Just give it a command line and let it type shell commands!" (Like a hacker in a movie).
A third group says, "Nope! Just let the robot write a Python script and run it all at once!" (The execute_code approach).

This paper is like a giant, fair referee that sets up a race to see which method actually saves money and gets the job done. They didn't just guess; they ran the exact same robot (two different models: Claude and Codex) on the exact same tasks, but changed only the tools they were allowed to use.

Here is what they found, broken down into simple stories.

The Main Discovery: It Depends on the Robot and the Job

The biggest surprise? There is no single "best" tool. The winner changes depending on who the robot is and what it's doing.

Think of it like this:

The "Math Puzzle" Job (Artifact Tasks): If the robot is doing calculations or data crunching, the "write a script and run it" method (code_only) is the clear winner. It's like hiring a chef who just writes one perfect recipe and cooks the whole meal in one go.
- For the Claude robot, this method saved 24.6% on costs.
- For the Codex robot, it saved about 6.7% (though the paper says this one is a bit shaky, like a coin flip).
- Verdict: For math and data, the script method is cheaper and just as good at solving the problem.
The "Fixing a Messy Codebase" Job (SWE-bench Tasks): This is where it gets tricky. These tasks involve editing many files in a complex project.
- If you use the Codex robot: The script method is still the champion! It saved 19.9% on costs. Why? Because the script method lets the robot bundle many small requests into one big package, like a delivery truck carrying 50 packages instead of a person walking 50 times.
- If you use the Claude robot: Uh oh. The script method actually got more expensive (by 14.4%), though the paper notes this wasn't a statistically "proven" difference, just a strong trend. Why? Because for Claude, writing a script to edit a file is like trying to change a tire by building a new car first. It takes way too many words (tokens) to explain the edit in code, creating "edit friction."

What the Paper Rules Out

The paper explicitly argues against the idea that one tool surface is always better for everyone.

It rules out: "Special IDE buttons are always required." (Because the script method won on Codex).
It rules out: "Bash commands are always enough." (Because the script method was cheaper on Claude for math tasks).
It rules out: "Code execution is always the cheapest." (Because it was more expensive for Claude on complex code fixes).

The authors are very clear: You can't just pick a tool based on the tool itself. You have to look at the combination of the robot's brain and the type of work.

The "Pass Rate" Surprise

Here is the most important part: The cost changed, but the success rate didn't.

Imagine two runners. One runs in heavy boots (expensive tools), and one runs in sneakers (cheap tools). The paper found that both runners finished the race at the exact same speed.

Whether the robot used the fancy toolbox, the bash commands, or the script method, the percentage of tasks they solved correctly was almost identical (within 3 percentage points).
The "script" method didn't make the robot smarter or dumber; it just changed how it walked to the finish line. Sometimes that walk was a sprint (cheap), and sometimes it was a stumble (expensive).

Why Did the Costs Change?

The paper digs into why the script method was cheaper or more expensive.

The "Edit Friction" Tax (For Claude): When Claude had to use the script method to fix a file, it had to write a long Python script just to say "change line 5." This used up a lot of "output tokens" (words the robot had to type). It was like paying a toll every time you wanted to turn a page. This happened mostly on tasks where the robot failed or struggled, making those specific runs very expensive.
The "Batching" Bonus (For Codex): When Codex used the script method, it could pack many small commands into one script. Instead of asking the computer "Read file A," then "Read file B," then "Read file C" (three separate trips), it asked "Read A, B, and C" in one go. This saved a ton of "input tokens" (words the robot had to read).
The "Doomed Run" Effect: The extra cost for Claude on the code-fixing tasks mostly happened when the robot was already going to fail. It was like a car running out of gas while driving in circles. The script method didn't cause the failure; it just made the failed attempt cost more money.

How Sure Are We?

The authors are very confident in the math tasks and the Codex robot results. They ran these tests on 93 math tasks and 100 code-fixing tasks, using three different "seeds" (random starting points) for each to make sure the results weren't just luck. The savings for Codex on code-fixing tasks were statistically significant (a 19.9% drop with a p-value of 2.0 × 10⁻⁹, which is basically zero chance of being random).

However, for the Claude robot on code-fixing tasks, the result is a bit murkier. The cost went up by 14.4%, but the statistical test said this wasn't a "slam dunk" proof (p-value of 0.12). The authors call this "directional," meaning the trend is there, but they'd need to run it more times to be 100% certain it's not a fluke.

The Bottom Line

If you are building a coding agent:

Don't guess. The "best" tool depends on your specific robot and your specific job.
For Math/Data: Try the "write a script" method. It's cheaper and works just as well.
For Code Fixes: It depends. If you use Codex, the script method is great. If you use Claude, you might want to keep the standard editing tools, especially for hard problems, because the script method might get bogged down in "edit friction."
Don't worry about intelligence: Changing the tools won't make your robot smarter or dumber; it just changes the price tag of the journey.

The paper concludes that the "cheapest" way to run an agent isn't a universal rule; it's a puzzle where you have to match the tool to the robot and the task.

Technical Summary: When Does Restricting a Coding Agent to `execute_code` Help?

Problem Statement

The field of coding agents currently operates under three mutually incompatible claims regarding the optimal "tool surface" (the set of interfaces exposed to the agent):

Specialized IDE Primitives are Required: Structured interfaces (e.g., Agent-Computer Interface or ACI) significantly outperform raw shell access, as argued by SWE-agent.
Bash is Sufficient: Minimal bash-only scaffolds can achieve competitive performance on software engineering benchmarks without IDE primitives.
Code Execution is Optimal: Routing agent actions through a single execute_code tool (via Model Context Protocol, MCP) drastically reduces token usage compared to calling native tools directly.

Prior work has failed to resolve these contradictions because existing studies fix a single variable (e.g., comparing ACI vs. shell on a specific model) without crossing the task regime (computation vs. code modification) and agent design (different model families). Consequently, the field lacks a unified understanding of which conditions favor which tool surface.

Methodology

The authors conduct a "missing crossed comparison" using an integrity-clean, three-arm ablation study. The experimental design is a $3 \times 2 \times 2$ factorial structure:

Three Tool Surfaces:
- Baseline: The agent's default tool surface (e.g., Read, Grep, Edit, Write, Bash).
- Bash_Only: Restricted to native bash plus read-only browsing (Read, Glob, Grep); all file-editing primitives (Edit, Write) are removed.
- Code_Only: The entire native surface is replaced by a single MCP tool (mcp__codebox__execute_code) running a persistent Python+Bash REPL. All native built-ins are disallowed.
Two Agents: Claude Code (model claude-sonnet-4-6) and OpenAI Codex CLI (model gpt-5.5).
Two Task Regimes:
- Computation: A new "Artifact" suite ( $n=93$ ) consisting of self-contained tasks defined by a workspace, hidden grader, and reference output.
- Modification: SWE-bench Mini ( $n=100$ ), a curated subset of repository modification tasks.

Experimental Controls:

Integrity: The harness enforces a strict SWE-bench protocol where test patches are applied after the agent terminates. File system states are isolated between arms using fuse-overlayfs to prevent cross-contamination.
Cost Accounting: To address server-side prompt caching variability, the authors employ a "cache-adjusted cost" metric. This applies a median-floor adjustment to the first-turn input tokens, normalizing the "warm/cold" cache lottery without claiming full cache isolation.
Statistical Rigor: The unit of inference is the task, not the seed. Results are analyzed using paired Wilcoxon signed-rank tests on per-task differences across three independent seeds.

Key Contributions

1. A Four-Cell Agent-Conditional Cost Structure

The study reveals that the cost-effectiveness of restricting an agent to execute_code is not universal but jointly determined by the task regime and the agent design.

Three of four cells favor code_only:
- Artifact/Claude: Significant cost reduction ( $-24.6\%$ , $p=7.4 \times 10^{-14}$ ).
- Artifact/Codex: Directional cost reduction ( $-6.7\%$ , not significant).
- SWE-bench/Codex: Significant cost reduction ( $-19.9\%$ , $p=2.0 \times 10^{-9}$ ).
One cell reverses the trend:
- SWE-bench/Claude: code_only is directionally costlier ( $+14.4\%$ , not significant).

This finding demonstrates that the three prior claims in the literature are not contradictory but are true only under specific (regime, agent) combinations.

2. Three-Mechanism Causal Decomposition

The authors decompose the observed cost variances into three distinct mechanisms:

Path-Cost (Edit Friction): On Claude for SWE-bench, code_only forces every file edit to be expressed as a Python script. This scales output tokens with edit volume (Spearman $\rho=0.488$ ), creating a "tax" on high-edit tasks.
Failure-Cost: The cost overrun for Claude on SWE-bench is localized to "doomed-run" trajectories (instances where all arms fail). On the subset of unanimous-pass instances, the cost gap collapses from $+14.4\%$ to $+4.1\%$ (not significant).
Pricing Asymmetry (Batching): On Codex for SWE-bench, code_only achieves significant savings ( $-19.9\%$ ) primarily through input token reduction ( $-24.8\%$ ). This is driven by tool-call batching (one Python script replaces multiple native tool calls) and programmatic output suppression (the agent constrains output volume, avoiding the hard output caps of native tools).

3. Capability Invariance

Across all four cells and three tool surfaces, pass rates are statistically tied (differences $<3$ percentage points, all not significant).

The agreement matrix shows that for the majority of instances ( $\ge 91\%$ under majority vote), all arms either pass or fail together.
This confirms that tool surface selection alters the path and cost of solving a task, but not the capability to solve it, consistent with the "Capability Overlap Principle."

Results Summary

The paper provides an "at-a-glance" map of when code_only is beneficial:

Computation Tasks (Artifact): Restricting to execute_code is cheaper for both agents, with a significant win for Claude.
Modification Tasks (SWE-bench) with Codex: Restricting to execute_code is significantly cheaper due to input token batching and output control.
Modification Tasks (SWE-bench) with Claude: Restricting to execute_code is directionally more expensive. This is driven by "edit friction" (higher output tokens for edits) and "failure costs" (expensive failed trajectories), rather than a penalty on successful edits.

Significance and Claims

The paper claims to resolve the "three contradictory claims" of the field by demonstrating that no single tool surface is optimal across all contexts.

Cost vs. Capability: The headline signal of tool surface choice is cost, not pass rate. Since pass rates are invariant, the choice of surface is a cost-engineering decision rather than a capability enhancement.
Joint Determination: The optimal surface is determined by the intersection of the task regime (computation vs. modification) and the specific agent's implementation (how it handles batching, output caps, and edit primitives).
Empirical Receipt: The study provides the first empirical evidence supporting the "tool-use tax" inequality in coding agents: a tool surface delivers net benefit only when its task-specific capability gain exceeds the per-call cost of its definition.

The authors explicitly state that their findings are cost-engineering claims, not capability claims. They do not propose that code_only is universally superior, nor do they claim their results generalize to all models or external MCP tools without further validation. The significance lies in the methodological framework that allows for the disambiguation of agent design and regime effects, moving the field from anecdotal industry reports to controlled, crossed ablation studies.