🤖 machine learning

What Context Does a Coding Agent Actually Need to Act?

This paper demonstrates that for coding agents editing code, the essential context is strictly limited to the specific files being modified, as natural language summaries and surrounding file content contribute negligibly to issue resolution compared to the source code itself, while also revealing a significant noise floor in benchmark results caused by non-deterministic API inference.

Original authors: Brian Sam-Bodden

Published 2026-07-14✓ Author reviewed ⓘ

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Brian Sam-Bodden

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to fix a leaky faucet in a massive, 100-story skyscraper. You don't need to read the blueprints for the entire building, the cafeteria menu, or the security logs to find the wrench. You just need to know exactly which pipe is dripping and see the immediate area around it.

That's the surprising lesson from this new study about "coding agents"—the AI bots that write and fix software. For a long time, the tech world assumed these bots needed to swallow the entire codebase of a project (sometimes millions of lines) into their "brain" to do their job. The thinking was: "More context is always better."

But this paper says: Stop stuffing the brain. It turns out that for the actual act of fixing code, the AI needs almost nothing but the specific lines it's about to change.

The "Find" vs. The "Act"

The researchers split the problem into two parts:

The Find: Locating the broken code.
The Act: Actually fixing it once you've found it.

To test this, they used a "magic map" (an oracle) to tell the AI exactly where the broken code was. This meant the AI didn't have to guess where to look; it just had to focus on how to fix it. They then gave the AI different "views" of that code to see what worked best.

The Big Disappointment: Summaries Don't Work

One popular idea was: "Let's just give the AI a natural language summary of the code, like a book blurb."

The Test: They asked the AI to answer tricky questions about how the code behaves, using either the full source code or a summary written by a super-smart AI (a "frontier model").
The Result: The full source code got 27 out of 45 questions right. The summaries? They got only 4 out of 45 right.
The Twist: It didn't matter if the summary was written by the smartest AI in the world or a tiny, basic model. They both failed equally. The problem wasn't the writer; the problem was the format. A summary just can't carry the specific "behavioral" details needed to fix a bug. It's like trying to fix a car engine by reading a travel brochure about the car; the brochure is nice, but it doesn't tell you which bolt is loose.

The "Skeleton" vs. The "Whole Body"

Next, they tested if the AI needed the full, fleshy code or just the "skeleton" (the structure, like function names and signatures).

The Setup: They took 70 real-world coding problems. For some, they gave the AI the full files. For others, they gave it just the "skeleton" (UML diagrams and signatures) or a "keep/drop" version (keeping only the essential parts and deleting the rest).
The Result: The "skeleton" and "keep/drop" versions solved just as many problems as the full files. In fact, the "keep/drop" method solved slightly more (25 out of 70) than the full files (19 out of 70), though the difference was small enough that it might just be luck.
The Cost: Here's the kicker. Solving a problem with the full files cost the AI 94,000 tokens (a unit of text). Solving it with the compressed "keep/drop" method cost only 19,000 tokens. That's a massive savings—about 3 to 3.7 times cheaper—with no loss in performance.

The "Noise" Warning

The researchers also found something weird and important: even when they ran the exact same test with the exact same settings, the AI sometimes gave different answers. About 9% of the time, the outcome flipped between runs. This means that if you see a tiny difference between two methods (like a 2% improvement), it might just be random noise, not a real breakthrough.

The Bottom Line

The paper concludes that for the specific task of fixing code, less is more.

What works: Giving the AI the exact lines of code it needs to edit, stripped down to their essentials.
What doesn't work: Flooding the AI with summaries, full file histories, or complex structural diagrams.
The Verdict: The "signal" lives in the code itself, not in the story we tell about the code. By cutting out the fluff, we can fix bugs for a fraction of the cost, without needing to read the whole skyscraper to fix one leaky faucet.

The authors are very careful to say this applies to "single-shot" fixes (where the AI tries once and doesn't get to re-read or ask for help). But for that specific, high-stakes moment of editing, the data is clear: you don't need the whole library; you just need the right page.

Technical Summary: What Context Does a Coding Agent Actually Need to Act?

Problem Statement

Modern coding agents possess context windows large enough to hold entire code repositories, yet evidence suggests that much of this context is unused. While retrieval-augmented generation (RAG) and sophisticated scaffolding are often employed to manage context, benchmarks like ContextBench indicate that these complex systems often barely outperform trivial baselines. The core question addressed is not how much context an agent can hold, but what minimal context is actually necessary for an agent to successfully edit code, and in what representation.

The authors distinguish between two sub-problems often conflated in agent design:

Find: Locating the specific functions or files where work must occur.
Act: Given the location, possessing sufficient information to perform the edit correctly.

This study isolates the Act phase. By holding localization fixed (using an oracle to provide the exact gold-standard files to be edited), the authors vary only the representation of that code to determine the minimal context required for resolution.

Methodology

Experimental Rig: `resolve@cost`

The authors constructed an audited instrument to measure resolution rates against context costs.

Dataset: 70 multi-file instances from SWE-bench Verified (after excluding one instance where the gold patch failed its own environment validation).
Oracle Localization: To isolate representation effects from retrieval errors, every experimental "arm" (condition) receives the exact same set of gold-changed files. The only variable is how the content of those files is rendered.
Agent: A single-shot run of Claude Sonnet 4.6 (Temperature 0, max 8000 tokens). The agent outputs SEARCH/REPLACE blocks, which are converted to unified diffs by a deterministic builder (difflib) and verified via a Docker harness.
Validation:
- Expressibility Gate: Before any agent call, it is verified that every line deleted or anchor inserted in the gold patch is present verbatim in the rendered context for every arm.
- Determinism: Patches are constructed deterministically to avoid confounds from non-reproducible diff generation.
- Pre-registration: Hypotheses and protocols were frozen before data collection.

Representation Arms

The study compares three primary representations for the gold-changed files:

oracle_full: The complete, unmodified source files.
oracle_tier: A structured representation where non-locus code is rendered as UML skeletons (class structures) and method signatures, while the locus of change remains full.
oracle_keep_drop: A binary approach where non-locus code is deleted entirely, leaving only the specific functions/classes being edited.

Behavioral Probes (The "Find/Act" Boundary)

To test if compressed representations carry actionable signal, the authors used behavioral probes:

Setup: 15 held-out classes from seaborn, pylint, and pytest. Three probes (behavioral questions) were minted from test files for each class.
Task: The agent (Sonnet 4.6) reads a representation (Full Source, Frontier Summary, 3B Model Summary, or Signatures) and answers the probe.
Judging: An independent judge (GPT-4o-mini) grades the answers.
Control: A frontier summarizer (Sonnet 4.6) and a small model (Qwen-3B) generated summaries to ensure the "summary" failure wasn't due to poor summarization quality.

Key Results

1. The "Act" Context is Starkly Minimal

The registered hypothesis ( $H_2$ ) predicted that structured tiers (UML skeletons/signatures) would resolve more issues than a simple binary keep/drop approach. This hypothesis failed.

Resolution Rates:
- oracle_keep_drop: 25/70 resolved (35.7%)
- oracle_tier: 23/70 resolved (32.9%)
- oracle_full: 19/70 resolved (27.1%)
Statistical Significance: The difference between structured tiers and keep/drop is null (McNemar $p=0.75$ ). Rendering surrounding code as UML skeletons provides no measurable benefit over deleting it outright when localization is perfect.
Cost Efficiency: Compressed contexts resolve issues at 3–3.7× fewer tokens than full files.
- Cost per resolve: ~19K tokens (keep/drop) vs. ~94K tokens (full files).

2. Natural Language Summaries Fail to Carry Actionable Signal

Behavioral probes revealed a massive gap in actionable information between source code and summaries.

Accuracy: Full source answered 60% (27/45) of behavioral probes. Natural language summaries (both frontier and 3B models) answered only ~9% (4/45).
Representation vs. Summarizer: The gap is inherent to the representation class, not the quality of the summarizer. Frontier summaries performed identically to 3B model summaries.
Signatures: Method signatures + docstrings performed slightly better than prose summaries (6/45) but still failed to match source code.
Conclusion: Natural language summaries do not answer the behavioral questions that source code answers; the "signal" lives in the code structure itself, not in narrative summaries.

3. Run-to-Run Non-Determinism

A critical methodological finding concerns the stability of LLM inference at Temperature 0.

Observation: Re-running the keep/drop arm with byte-identical protocols (same model, same prompts, T=0) resulted in ~9% of per-instance outcomes flipping (25/70 resolved originally vs. 23/70 in the re-run).
Implication: This noise floor (~9%) bounds the interpretation of small effect sizes. Differences of 1–2 instances (like the registered hypothesis $H_2$ ) are statistically indistinguishable from inference noise.

Contributions and Significance

The paper claims four primary contributions:

An Audited Instrument: A resolve@cost framework with per-instance expressibility verification, deterministic patch construction, and pre-registered hypotheses, ensuring every number is traceable to raw artifacts.
A Powered Null Result: Evidence that structured representation tiers (UML/skeletons) carry no additional resolution signal over binary keep/drop at oracle localization. This challenges the assumption that "more structure" equals "better acting."
A De-circularized Find/Act Boundary: Demonstration that source code carries roughly 7× the actionable signal of natural language summaries. Crucially, a frontier summarizer cannot close this gap, indicating the limitation is the representation format, not the summarizer's capability.
Methodological Caution: The quantification of a ~9% noise floor in single-shot, T=0 API inference. This suggests that many small effect sizes reported in SWE-bench literature (including this paper's own null results) may be artifacts of non-determinism rather than true signal.

Scope and Limitations

The authors explicitly state that their conclusions apply to single-shot agents without execution feedback loops. They acknowledge that multi-turn agents might recover from compressed context by reading more, but in the single-shot regime, the "find" step must be perfect (oracle) and the "act" context must be minimal. The study does not claim that summaries are useless for navigation (finding code), only that they are insufficient for acting (editing code) when compared to the source itself.

The paper concludes that for the specific task of editing code given the correct location, the minimal context required is the code being edited itself, and surrounding context (even in structured forms) adds negligible value while incurring significant token costs.