💻 computer science

Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories

This paper presents a controlled ablation study demonstrating that persistent context files (e.g., AGENTS.md) do not measurably improve coding agent correctness because failures stem from implementation skill deficits rather than missing repository knowledge, while also explaining prior contradictory findings through agent-specific task difficulty variations.

Original authors: Prakhar Khatri

Published 2026-07-31

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Prakhar Khatri

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Digital Librarian vs. The Master Builder

Imagine you are trying to teach a super-smart, but brand-new, robot assistant how to fix a complex machine in a giant, messy workshop. You have two main ways to help it. First, you could give it a detailed map of the workshop, a list of rules about how tools should be used, and a guide on how the machine usually behaves. In the world of computer programming, this is called a "context file" (often named AGENTS.md or CLAUDE.md). It's like a sticky note left on the robot's desk saying, "Remember, we always use red screws here!"

Second, you could just let the robot wander into the workshop and figure things out by looking at the machine itself.

For a long time, people building these AI coding assistants have been obsessed with the first option. They spend hours writing these instruction manuals, believing that if the robot just knew the rules better, it would build better code. But here's the big question: Does reading the manual actually make the robot a better builder, or does it just make the robot feel more confident while it still makes the same mistakes? This is the mystery a new study set out to solve, testing whether these digital instruction manuals are the magic keys to better AI coding or just a lot of extra paper.

The Great Manual Test

In this study, researchers acted like strict science teachers. They took two of the smartest AI coding agents available today—one from a company called Anthropic (named Claude) and one from OpenAI (named Codex)—and gave them a series of real-world coding challenges. These weren't fake homework problems; they were actual tasks taken from real software projects, like fixing bugs in a cloud toolkit or tweaking a compiler.

To see if the "manual" helped, they ran the same task three different times for each robot:

The "No Notes" Run: The robot got the task but no manual.
The "Always On" Run: The robot had the full manual shoved into its brain every single step of the way.
The "On-Demand" Run: The robot had a library of smaller, organized notes it could choose to read if it got stuck.

They ran this experiment 288 times in total, checking the final code against a hidden "gold standard" test to see if it actually worked.

The Big Surprise: Manuals Don't Fix Bad Building

The results were a bit of a shock to the people who write these manuals. The study found that having the context file made almost no difference in whether the AI got the job right.

Whether the robot had the manual, the library, or nothing at all, the success rate stayed roughly the same. For the Claude robot, the manual didn't change the outcome by more than 10 percentage points. For the Codex robot, it didn't change it by more than 15 percentage points. In the world of science, this is what we call a "null result"—it means the thing we thought might be important, isn't.

The researchers dug deeper to figure out why the manuals didn't help. They looked at the times the robots almost got the answer right but failed. They found that the robots didn't fail because they were missing a fact from the manual (like "don't use red screws"). Instead, they failed because they lacked building skills.

Think of it like this: If you give a brilliant architect a perfect blueprint of a house, but they don't know how to lay a brick or how to wire a light switch, the blueprint won't save them. The AI agents in the study were great at reading, but they stumbled on the actual doing. They struggled with designing features, picking the right patterns, and connecting the pieces correctly. The manual couldn't teach them how to build; it could only tell them what the building looked like.

The "One Size Doesn't Fit All" Twist

There was one other fascinating discovery. The researchers noticed that a task that was super easy for one robot could be incredibly hard for the other, and vice versa. It's like how some people are great at solving math puzzles but terrible at juggling, while their friend is the opposite.

Because of this, the study suggests that previous research might have gotten confused. If one scientist tested the manuals on a robot that found the tasks easy, and another tested them on a robot that found the tasks hard, they would get different results. The "sweet spot" where a manual might actually help seems to be different for every AI brain.

Did the Manuals Do Anything at All?

So, were the manuals completely useless? Not entirely, but the benefits were very specific and small.

Speed and Cost: On one specific project, when the robot had the manual, it stopped running unnecessary, slow tests. It was like the robot read a sign saying "This test takes 20 minutes" and decided to skip it. This saved time, but it didn't make the code any better.
The "Near-Miss" Test: The researchers tried a special experiment where they took tasks the robots almost solved and gave them the real, high-quality manual to see if it would push them over the edge. It didn't. Even with the best instructions, the robots still couldn't turn a "near miss" into a "success."

The Bottom Line

The study concludes that for the AI coding agents they tested, spending hours writing perfect instruction manuals (AGENTS.md) probably won't make them write better code. The problem isn't that the robots don't know the rules; it's that they need to get better at the actual craft of building software.

While these manuals might help the robots work a little faster or save a tiny bit of money by avoiding bad habits, they aren't the magic wand that fixes broken code. If you want your AI assistant to be a better builder, the study suggests you might get better results by breaking the tasks down into smaller steps or giving it better examples of how to build, rather than just giving it more rules to read.

Technical Summary: Do Context Files Help Coding Agents?

Problem Statement

Persistent context files (e.g., AGENTS.md, CLAUDE.md) have become standard infrastructure for guiding AI coding agents, with platforms like Claude Code automatically loading them into sessions. Practitioners invest significant effort authoring these files to define coding conventions, architectural constraints, and workflow guidance, expecting improved code generation. However, empirical evidence regarding their effectiveness is contradictory. Some studies report efficiency gains (lower runtime, fewer tokens), while others find no significant effect on task completion correctness. Existing literature fails to reconcile these findings because studies often vary in agent family (Codex vs. Claude), evaluation methods, and experimental control, particularly regarding how context is injected.

Methodology

This study presents a controlled ablation of context-injection strategies across two frontier coding agents (Claude Code and Codex) using real-world repositories and tasks.

Experimental Design

Agents: Two distinct agents were evaluated: Claude Code (claude-sonnet-4-6) and Codex CLI (gpt-5.5).
Repositories & Tasks: The study utilized 3 Python repositories (pdm, firebase-admin-python, opshin) with varying AGENTS.md quality and length. Tasks were mined from merged pull requests, requiring the agent to implement changes verified by hidden "gold" tests (SWE-bench Tier-C style). The final set included 17 tasks for Codex and 15 for Claude.
Injection Strategies (Independent Variable): Three distinct strategies were tested:
1. NONE: The AGENTS.md file is removed from the workspace; no context is provided.
2. ALWAYS ON: The full AGENTS.md content is injected into the system prompt every turn.
3. SELECTIVE: Topic-organized wiki files are placed in the workspace, and a system prompt hint directs the agent to retrieve relevant files on demand via its Read tool.
Evaluation: Each task was run under all three strategies with 3 independent repeats (totaling 288 evaluated runs). Success was defined strictly by whether all gold tests passed. Efficiency metrics included tool calls, wall-clock time, and token usage.
Statistical Analysis: The study employed omnibus permutation tests, paired Wilcoxon signed-rank tests for efficiency, and Two One-Sided Tests (TOST) for equivalence testing to bound the effect size of context injection.

Key Contributions

Controlled Two-Agent Ablation: A rigorous study across 288 runs demonstrating that context injection strategy does not measurably move correctness on either agent, with effects bounded to ≤10–15 percentage points (pp) via equivalence testing.
Failure-Mode Triage & Manipulation Probe: Evidence that agents fail primarily on implementation skill (feature design, pattern selection, exact wiring) rather than missing repository knowledge. A specific probe confirmed that real AGENTS.md files never converted a "near-miss" failure to a pass on either agent.
Agent-Specific Borderline Difficulty: Demonstration that task difficulty is agent-specific. The same task can be trivially solvable for one agent while remaining challenging for another (Spearman $\rho=0.75$ correlation in pass rates, but distinct "informative bands"). This offers a candidate explanation for prior contradictory findings in single-agent studies.

Results

Correctness

Null Effect: Neither agent showed a statistically significant improvement in correctness with context injection.
- Claude: Pass rates were 53.3% (NONE), 55.6% (ALWAYS ON), and 55.6% (SELECTIVE).
- Codex: Pass rates were 58.8% (NONE), 56.9% (ALWAYS ON), and 52.9% (SELECTIVE).
Equivalence Bounds: TOST analysis bounded the pairwise strategy differences to <10pp for Claude and <15pp for Codex.
Borderline Tasks: On tasks where the design had statistical power to detect an effect (borderline difficulty), context injection did not improve pass rates. For example, on 4 Codex-borderline tasks, NONE achieved 58% pass rate versus 42% for both ALWAYS ON and SELECTIVE.

Efficiency

Claude: The SELECTIVE strategy significantly reduced cache-creation tokens compared to NONE, attributed to the mechanical difference of injecting a short hint versus the full file every turn.
Opshin Specifics: On the opshin repository, which contained explicit warnings about test suite duration in its context file, Claude exhibited a dose-dependent reduction in blind full-suite test runs and wall-clock time when context was present. This is a process-level effect (how the agent works) rather than an outcome-level effect (correctness).
Codex: All efficiency metrics remained flat across strategies.

Failure Analysis

Inspection of near-miss failures revealed that errors stemmed from engineering precision, architectural pattern choices, and type-system reasoning gaps. None of these failures represented a "missing fact" that a context file could supply. Furthermore, a manipulation-validity probe confirmed that even high-quality AGENTS.md files could not rescue near-miss failures.

Significance and Claims

The paper claims that the widespread practice of authoring generic context files may yield diminishing returns for coding agent correctness. The primary mechanism for failure in real-world coding tasks is a lack of implementation skill, not a lack of repository knowledge.

Reconciliation of Prior Work: The study suggests that contradictory findings in previous literature (e.g., Lulla et al. vs. Gloaguen et al.) likely arise because different agents have different "informative bands" of task difficulty. A task that falls in the borderline range for one agent (where context might theoretically help) may fall into the floor or ceiling range for another, rendering the context effect invisible in single-agent studies.
Practitioner Implications: While generic context documents may not measurably improve correctness, they can influence agent behavior (e.g., reducing wasted test runs if specific warnings are present). However, the authors suggest that effort spent on generic context documents may be better directed toward task decomposition, tooling, or example-driven prompting.
Methodological Contribution: The study highlights the necessity of screening tasks per-agent to ensure an informative dynamic range in future ablation studies and notes that sample sizes of ~120 tasks are required to detect smaller effect sizes (10pp) with adequate power.

The authors release all code, data, and analysis to support future scaling of this design.