🤖 AI

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?

This paper evaluates the effectiveness of repository-level context files (like AGENTS.md) for coding agents and finds that, contrary to common practice, they generally fail to improve task success rates while increasing inference costs by over 20%.

Original authors: Thibaud Gloaguen, Niels Mündler, Mark Müller, Veselin Raychev, Martin Vechev

Published 2026-06-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Thibaud Gloaguen, Niels Mündler, Mark Müller, Veselin Raychev, Martin Vechev

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a very smart, but slightly new, assistant to fix a specific problem in your massive, cluttered workshop. This assistant is an AI coding agent.

To help your assistant, you decide to leave them a "cheat sheet" or a "user manual" (called an AGENTS.md file) on the workbench. This manual is supposed to tell the assistant:

"Here is how this workshop is organized."
"Here are the specific rules we follow for building things."
"Here is the best way to use our special tools."

The big question the paper asks is: Does leaving this cheat sheet actually help the assistant finish the job faster and better, or does it just make them confused and waste time?

The researchers set up a giant experiment to find out. Here is what they discovered, explained simply:

1. The "Cheat Sheet" Paradox

The team tested two types of cheat sheets:

The Human-Written One: Written by the actual workshop owner (the developer).
The AI-Generated One: Written by another AI, just because the agent framework told it to.

The Result: Surprisingly, neither type of cheat sheet made the assistant finish the job more often. In fact, the success rate stayed almost exactly the same whether the sheet was there or not.

2. The "Over-Prepared" Assistant

While the cheat sheets didn't help the assistant get the right answer more often, they did change how the assistant worked.

Without a sheet: The assistant looked around, tried a few things, and fixed the problem.
With a sheet: The assistant read the sheet, took it very seriously, and then went on a "research trip." They checked more files, ran more tests, and double-checked their work.

The Catch: This extra "thoroughness" cost money. Because the assistant spent more time reading and testing, the bill for the AI's time went up by over 20%. It's like hiring a detective who reads the entire encyclopedia before solving a simple missing sock case. They are very careful, but they are expensive and not necessarily faster.

3. The "Map" vs. The "Instructions"

The researchers found a split in what the cheat sheets actually did:

The "Map" (Overview): Most cheat sheets included a big description of the whole workshop ("Here is the tool shed, here is the paint room..."). The assistant read this, but it didn't help them find the right tool. It was like giving someone a detailed map of a city when they just need to know which street corner to turn on.
The "Instructions" (Rules): The cheat sheets also had specific rules ("Always use the red wrench," "Don't paint before sanding"). The assistant followed these rules perfectly. They used the specific tools mentioned and followed the specific steps.

4. The Verdict: Less is More (For Now)

The paper concludes that currently, automatically generating these cheat sheets is a waste of money.

If you use an AI to write the cheat sheet, it just adds fluff (long overviews) that confuses the agent and burns your budget.
If a human writes it, it helps a tiny bit more than the AI version, but it still doesn't significantly improve the success rate.

The Bottom Line:
If you are an AI agent developer, don't just dump a massive, AI-generated manual on your agent. It makes them work harder and cost more without making them smarter.

If you must have a cheat sheet, keep it short. Only include the specific, weird rules that aren't already obvious in the code (like "Always use the red wrench"). Don't include a long history of the workshop; the agent can figure that out on its own.

In short: The cheat sheet makes the agent a more obedient student who studies harder, but it doesn't make them a better student who gets better grades. And studying harder costs more tuition.

Technical Summary: Evaluating AGENTS.md

Problem Statement

The software engineering industry has rapidly adopted coding agents, with a prevailing recommendation from model and agent developers to tailor these agents to specific repositories using context files (e.g., AGENTS.md or CLAUDE.md). These files typically contain repository overviews, tooling instructions, and style guides intended to improve navigation, build execution, and task resolution. Despite widespread adoption—reported in over 60,000 open-source repositories—and strong industry encouragement, there is no rigorous, large-scale investigation into whether these context files actually improve the performance of coding agents on real-world tasks. Previous benchmarks like SWE-bench lack context files, and popular repositories used in such benchmarks often do not represent the codebases where context files are actually deployed.

Methodology

To address this gap, the authors conducted a comprehensive evaluation across two complementary settings:

Novel Benchmark (CTXBENCH): The authors constructed a new benchmark comprising 138 instances derived from real GitHub issues in 12 niche, Python-based repositories that already contain developer-committed context files. Unlike SWE-bench, which relies on popular repositories, CTXBENCH targets repositories where context files are an active practice. The dataset generation involved:
- Filtering repositories with AGENTS.md or CLAUDE.md and at least 400 Pull Requests (PRs).
- Selecting PRs that introduce deterministic, testable behaviors.
- Using LLM agents to generate standardized task descriptions and new unit tests where original PRs lacked them, ensuring the tests pass on the fixed code but fail on the base code.
- Validating that developer-provided context files are distinct from LLM-generated ones (shorter, lower AI-detection scores).
Experimental Setup: The study evaluated four coding agents (CLAUDE CODE, CODEX, QWEN CODE) paired with various models (SONNET-4.5, GPT-5.2, GPT-5.1 Mini, QWEN3-30B-CODER) on two datasets:
- SWE-BENCH Lite: 300 tasks from popular repositories without developer context files.
- CTXBENCH: 138 tasks from repositories with developer context files.
Evaluation Conditions: For each task, agents were tested in three settings:
- None: No context file provided.
- LLM: A context file automatically generated by the agent's recommended initialization command.
- DEV: The actual developer-committed context file (available only for CTXBENCH).
Metrics: Performance was measured by success rate (percentage of tasks where the generated patch passes all tests), the number of interaction steps, and the total inference cost. Trace analysis was performed to understand behavioral changes, including tool usage frequency and reasoning token consumption.

Key Contributions

CTXBENCH: A new benchmark specifically designed to evaluate the impact of actively used, developer-provided context files on coding agents, featuring 138 instances from 12 repositories.
Comprehensive Evaluation: An extensive study across multiple models and agents on both SWE-BENCH and CTXBENCH, revealing that context files do not significantly improve task success rates.
Behavioral Analysis: A detailed investigation of agent traces demonstrating that while agents strictly follow instructions in context files (leading to increased testing and exploration), these files fail to serve as effective repository overviews and increase operational costs.

Results

The study yields several counter-intuitive findings regarding the efficacy of context files:

No Performance Gain: Providing context files does not generally improve task success rates.
- On SWE-BENCH, LLM-generated context files resulted in a marginal, statistically insignificant decrease in success rates (average -0.5%).
- On CTXBENCH, LLM-generated files also showed no significant improvement (average -2%).
- Developer-committed files showed a marginal improvement of 2.4% on average over the "None" setting, but this was not statistically significant ( $p=0.21$ ). However, developer files significantly outperformed LLM-generated files ( $p=0.038$ ).
Increased Cost and Steps: Context files consistently increase the cost and complexity of task resolution.
- LLM-generated files increased inference costs by over 20% on average across both benchmarks.
- Developer-committed files increased costs by up to 19%.
- The number of interaction steps increased significantly in all settings with context files.
Behavioral Mechanisms:
- Instruction Following: Agents strictly follow instructions in context files. When specific tools (e.g., uv, pytest) are mentioned, their usage increases dramatically. This leads to more thorough testing, file exploration, and repository-specific tool usage.
- Ineffective Overviews: Despite recommendations to include codebase overviews, these sections do not help agents locate relevant files faster. In fact, agents often spend extra steps searching for and re-reading context files that are already in their context window.
- Reasoning Overhead: The presence of context files increases the number of reasoning tokens used by adaptive models (e.g., +22% for GPT-5.2 with LLM-generated files), suggesting the additional instructions make the task cognitively harder without providing necessary information.
Redundancy: Context files often act as redundant documentation. When existing documentation (READMEs, docs folders) is removed, LLM-generated context files actually improve performance, suggesting their primary value in standard settings is redundancy rather than unique insight.

Significance and Conclusion

The paper concludes that the current industry practice of automatically generating or broadly including context files like AGENTS.md is not supported by empirical evidence for improving coding agent performance. While agents follow the instructions within these files, leading to more thorough (and costly) exploration, the files generally fail to provide effective repository overviews that accelerate task resolution.

The authors argue that context files should be reserved for specifying non-standard coding practices or specific conventions not already present in the repository's README. They recommend that developers rigorously evaluate any attempt to use context files before deployment, rather than relying on automated generation or broad recommendations. This work highlights a gap between agent developer recommendations and observed outcomes, motivating future research into principled, concise, and task-relevant guidance generation for coding agents.