Code Isn't Memory: A Structural Codebase Index Inside a Coding Agent
This paper demonstrates that integrating a structural codebase index into a fixed coding agent harness significantly improves task localization and resolution rates on SWE-bench benchmarks without incurring additional costs, proving its cost-effectiveness compared to agentic-grep baselines and highlighting its specific value for multi-file change workloads.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a detective trying to solve a complex mystery inside a massive, messy library. The library represents a computer codebase, and the mystery is a bug or a new feature request. You have a super-smart assistant (the AI model) who can read and write, but they can't see the whole library at once. They have to ask for specific books or pages.
This paper asks a simple question: Is it better to give your detective assistant a magical, pre-made map of the library, or should they just wander around shouting "Where is the book about X?" (a method called "agentic grep")?
Here is the breakdown of the study using everyday analogies:
The Three Teams
The researchers set up three different teams to solve the same 91 mysteries (coding tasks) using the exact same super-smart assistant (Claude Opus 4.7).
- Team "Map-On" (SC-ON): The assistant has a Structural Codebase Index. Think of this as a high-tech, pre-built map that knows exactly how every book in the library connects to every other book. It knows that "Chapter A" references "Chapter B," and it can instantly find the right page based on the story's structure, not just keywords.
- Team "Map-Off" (SC-OFF): The assistant has the exact same tools and brain, but the map is removed. They have to find things the old-fashioned way: searching through shelves and shouting keywords.
- Team "Grep" (OpenCode): This is a different, well-known detective agency that only uses the "shouting keywords" method. They don't have a pre-made map at all.
The Experiment
The researchers made sure everything else was identical: the same library, the same detective brain, the same time limits, and even the same "leak-proof" room to ensure no one cheated by peeking at the answers beforehand. They ran the test three times to be sure the results weren't just luck.
The Results: What Happened?
1. The "Map" Team Found Things Much Faster
When the team with the map tried to find the specific file they needed to fix the bug, they succeeded 84.5% of the time. The team without the map only found it 44.3% of the time.
- Analogy: It's like the difference between asking a librarian who knows the whole building layout versus asking someone to just scan the spines of books until they get lucky. The map team knew exactly where to look.
2. The "Map" Team Solved More Problems
Because they found the right files faster, the "Map" team actually fixed the bugs 50.4% of the time. The team without the map only fixed them 41.9% of the time.
- Analogy: When you spend less time looking for the right book, you have more time to actually read it and fix the story.
3. It Wasn't More Expensive
A common fear is that having a fancy map costs too much money or takes too long to build. The study found that the "Map" team didn't cost more per task. In fact, because they solved problems faster, the cost per solved problem was actually lower ($2.30 vs. $2.92 for the keyword team).
- Analogy: Buying a GPS app for your car costs a few dollars, but it saves you gas and time by preventing you from driving in circles. The "Map" team drove straight to the destination; the others took a few wrong turns.
4. Where the Map Shined Brightest
The map was most helpful when the mystery involved multiple files (like a story that spans three different books). In these cases, the "Map" team crushed the others. When the task only involved one single file, the map was still helpful, but the keyword team could sometimes catch up.
- Analogy: If you need to find a single word in one book, shouting the word works fine. But if you need to understand how three different chapters interact to solve a plot hole, a map showing the connections is invaluable.
The Big Conclusion
The paper concludes that for coding agents, a structural map is not too expensive to run. It doesn't slow you down, and it doesn't cost extra money per task.
The real question for companies isn't "Can we afford the map?" but rather, "Do our problems involve complex, multi-file changes where a map would actually help?" If your work involves connecting many different parts of a system, the map pays for itself by helping the AI find the right pieces faster and solve the problem more often.
What the paper does NOT say:
- It does not claim this works for every type of AI or every type of software.
- It does not say this will replace human programmers.
- It does not make claims about medical or clinical uses (this is strictly about coding).
In short: Giving an AI a structural map of the code makes it a better detective, it solves more cases, and it does so without breaking the bank.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.