Loc2Repair: A Framework for Evaluating the Impact of File-Level Issue Localization in Repo-Level LLM Repair
This paper introduces Loc2Repair, a modular evaluation framework that demonstrates how decoupling file-level issue localization from repair synthesis consistently improves both the effectiveness and efficiency of repository-grounded automated repair systems.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a master mechanic trying to fix a broken car. The car is a complex piece of software, and the "break" is a bug reported by a user.
In the past, we judged AI "mechanics" (Large Language Models, or LLMs) by a single number: Did they fix the car? If the car drove away, great. If it didn’t, we just said, "The AI failed."
But this is like saying a mechanic failed without knowing why. Did they look at the engine when the problem was in the brakes? Did they look at the brakes but use the wrong wrench? Or did they fix the brakes but forget to tighten the final bolt?
This paper introduces Loc2Repair, a new way to test AI mechanics that separates the job into two distinct steps: Finding the problem (Localization) and Fixing the problem (Repair).
The Core Idea: The "Map" Before the "Wrench"
The authors argue that before an AI can fix code, it first needs to know where to look. In a massive software project (a repository), there are thousands of files. If the AI looks at the wrong files, it’s like a mechanic staring at the radio when the engine is smoking.
Loc2Repair is a testing framework that acts like a controlled laboratory. It lets researchers swap out the "finder" and the "fixer" independently to see how they work together.
The Experiment: Three Mechanics, Four Strategies
The researchers took three different AI "mechanics" (called backbones: Gemma4, GLM-4.7, and Qwen3.5) and gave them 500 broken software tasks. They tested each mechanic under four different conditions:
- The "Blind" Approach (Baseline): The mechanic is thrown into the garage with no hints. They have to figure out where the problem is and fix it all at once.
- The "Quick Scout" (Predicted Localization - Qwen4B): A small, fast AI scout quickly scans the garage and points to the likely broken parts. The main mechanic then focuses only on those parts.
- The "Detailed Scout" (Predicted Localization - Gemma4E4B): A slightly slower, more thorough AI scout scans the garage and provides a more detailed map of where the problem might be.
- The "Magic Map" (Gold Localization): The mechanic is given a perfect map showing exactly which files were changed in the original fix. This is the "cheat code" to see the absolute best performance possible if the AI knew exactly where to look.
What They Found
1. Giving the AI a map helps a lot.
When the AI mechanics were given hints about where to look (either from the scouts or the magic map), they fixed more bugs than when they were left to guess.
- Without hints: The AI fixed about 44.7% of the bugs on average.
- With scout hints: The success rate jumped to about 49%.
- With the perfect map: The success rate climbed to 52.4%.
2. It’s often faster, too.
Surprisingly, giving the AI a map didn’t just make them smarter; it often made them faster. By not wasting time searching the wrong files, the AI spent less time overall. On average, the "scout" methods saved over a minute per task compared to the blind approach.
3. Bigger isn’t always better for the scout.
The researchers used two different "scout" AIs. One was faster and cheaper (Qwen4B), and one was slower and more expensive (Gemma4E4B). Interestingly, the faster scout often performed just as well in helping the main mechanic fix the bug, even though it wasn’t the most accurate scout on its own. This suggests you don’t always need the most expensive tool to get a good result.
4. Finding the problem is only half the battle.
Even when the AI was given the perfect map (the "Magic Map" condition), it still failed to fix nearly half of the bugs. This is a crucial finding. It means that just knowing where the bug is isn’t enough. The AI still struggles with the actual fixing part—writing the correct code, debugging, and testing.
The Takeaway
The paper concludes that we need to stop treating AI coding agents as a single black box. By separating "finding" from "fixing," we can:
- Improve performance: Explicitly telling the AI where to look makes it better at its job.
- Save time and money: Focused searches are faster.
- Understand failures: We can now see if an AI failed because it looked in the wrong place or because it couldn’t write the right code.
In short, Loc2Repair proves that giving an AI a good map is a powerful lever for improvement, but it also shows that we still have a long way to go before AI can reliably fix the car once it’s found the broken part.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.