Exploration Structure in LLM Agents for Multi-File Change Localization
This paper proposes and evaluates a non-linear, domain-scoped parallel agentic exploration framework for localizing multi-file changes in software repositories, demonstrating that it significantly outperforms linear sequential approaches and achieves competitive results against much larger models on benchmarks like SWE-Bench Pro.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a detective trying to fix a broken machine. The machine is a massive software project (like a giant library of code), and someone has reported a bug. Your job is to find exactly which pages in the library need to be rewritten to fix the problem.
This paper is about how different "AI detectives" go about finding those pages. The researchers wanted to see if the way the detective searches matters more than how "smart" the detective is.
Here is the breakdown of their study using simple analogies:
The Problem: The "One-Step-at-a-Time" Trap
Most current AI tools act like a detective who walks through a library one aisle at a time. They pick a shelf, read a book, then move to the next shelf.
- The Flaw: If the bug is actually a mix of problems in three different wings of the library (e.g., the kitchen, the garden, and the attic), a detective who only walks one aisle at a time might get stuck in the kitchen, run out of time (or money), and never check the garden or the attic.
- The Paper's Idea: What if, instead of walking, we sent three different specialists to check the kitchen, garden, and attic at the same time?
The Experiment: The "Ansible" Library
The researchers tested this on a specific software project called Ansible (think of it as a very large, organized library). They created a test where they gave the AI a bug report and asked it to list the files that needed fixing.
They compared four types of detectives:
- The "Bookworm" (Plain LLM): A smart AI that has read many books but has never been inside this specific library. It has to guess based on memory.
- The "Solo Explorer" (RLM): An AI that is given a key to the library and a notebook. It walks in, opens doors, and reads files one by one, writing notes as it goes.
- The "Team of Specialists" (Domain Agents - The New Idea): An AI manager who first maps out the library's sections (Kitchen, Garden, Attic). When a bug comes in, the manager instantly sends a different specialist to each relevant section to work in parallel.
- The "Super-Expert" (Codex): A very large, expensive, and powerful AI detective used as a benchmark.
The Big Findings
1. Teamwork Beats Walking Alone
The "Team of Specialists" approach won by a huge margin, even though they used a smaller, cheaper AI model.
- Analogy: Imagine trying to find a lost key in a stadium. The "Solo Explorer" walks the whole stadium alone and gets tired. The "Team" sends people to the bleachers, the field, and the concession stands simultaneously. They find the key much faster and more accurately.
- Result: The team approach found the correct files much better than the solo walker, even when the solo walker had a bigger brain (a more powerful AI model).
2. Giving a Detective a Key Can Backfire
The researchers thought that giving the AI direct access to the file system (the "Solo Explorer" with a key) would help. Surprisingly, it often made things worse.
- Analogy: If you give a detective a key to a giant warehouse, they might get distracted by looking at thousands of irrelevant boxes (like test files or old drafts) and forget to look for the actual broken part. They get overwhelmed by "noise."
- Result: The AI with direct access often guessed too many wrong files, lowering its accuracy. The "Team" approach was smarter because it knew exactly which sections to look at and ignored the junk.
3. More Agents Don't Always Mean Better Results
They tried forcing the team to consult more specialists than necessary, just to be safe.
- Analogy: It's like calling in the entire fire department to put out a small candle. It doesn't put the fire out any faster; it just costs a lot more money (computer tokens) and creates a lot of confusion.
- Result: Being "aggressive" with calling more agents didn't help find the bug; it just wasted resources.
4. The "Documentation Blind Spot"
No matter how smart the AI was, they all struggled to find documentation files (the instruction manuals).
- Analogy: If a user says, "The red button doesn't work," the AI knows to fix the red button. But the AI rarely realizes that the instruction manual also needs to be updated to say "The red button is now broken." The bug report didn't mention the manual, so the AI ignored it.
- Result: This is a hidden dependency. The AI needs a rule that says, "If you fix a button, you must also check the manual," even if the user didn't ask for it.
The Conclusion
The paper concludes that how an AI explores a codebase is just as important as how smart the AI is.
- A small, well-organized team of specialists (Domain Agents) can beat a giant, powerful AI that wanders aimlessly.
- However, even the best AI teams struggle with "invisible" changes, like updating instruction manuals, because the bug reports don't explicitly ask for them.
In short: Structure beats raw power. Organizing the search process is the key to fixing complex software bugs.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.