💻 computer science

Exploring Generalizable Automated Program Repair with Large Language Models

This paper presents an extensive empirical evaluation of 13 large language models across four programming languages, revealing that no single model generalizes well across all contexts, that combining models improves unique bug fixes, and that reliance on perfect fault localization significantly overestimates repair accuracy in realistic scenarios.

Original authors: Viola Campos, Ridwan Shariffdeen, Adrian Ulges, Yannic Noller

Published 2026-02-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Viola Campos, Ridwan Shariffdeen, Adrian Ulges, Yannic Noller

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of super-intelligent, multilingual mechanics (these are the Large Language Models, or LLMs). Their job is to fix broken cars (software bugs) without ever seeing the car in person, only by reading a description of the problem and looking at a blueprint.

This paper is a massive road test of 13 of these top-tier mechanics. The researchers wanted to see: Can these AI mechanics fix any car, in any language, just by reading a manual? Or do they have specific strengths and weaknesses?

Here is the breakdown of their findings, translated into everyday terms:

1. The "One-Size-Fits-All" Myth is Dead

You might think, "If I hire the smartest mechanic, they can fix a Ford, a Toyota, and a Ferrari equally well."
The Reality: No single mechanic is the best at everything.

The Analogy: Imagine hiring a mechanic who is a genius at fixing Japanese cars but struggles with German ones. Another mechanic might be a wizard with French engines but gets confused by American trucks.
The Finding: The study found that different AI models excel at different programming languages (Java, Python, JavaScript, PHP). There is no "Super-Model" that dominates all languages. If you want to fix bugs in a multi-language software project, you can't rely on just one AI; you need a committee of experts, picking the right specialist for the specific language.

2. The Power of "Clues" (Test Errors)

When you bring a car to a shop, the mechanic doesn't just guess. They look at the dashboard warning lights and listen to the weird noise the engine makes.

The Analogy: The researchers tested two scenarios:
1. Blind Guessing: Telling the AI, "Here is a broken code, fix it." (Like saying, "My car is making a noise, fix it" with no details).
2. Giving Clues: Telling the AI, "Here is the code, AND here is the exact error message and the test that failed." (Like saying, "My car makes a grinding noise when I brake, and the 'Check Engine' light is flashing red").
The Finding: Giving the AI the error messages and test results was a game-changer. It boosted their success rate dramatically (sometimes by nearly 50%). It's like the difference between a mechanic guessing what's wrong versus looking at the diagnostic computer.

3. The "Perfect Map" vs. The "Fuzzy GPS"

To fix a bug, the AI needs to know where the problem is.

The Analogy:
- Perfect Localization: The researcher points exactly at the broken spark plug and says, "Fix this part."
- Automated Localization: The researcher uses a GPS that says, "The problem is somewhere in the engine block," but it might be pointing at the oil filter instead of the spark plug.
The Finding: When the AI was given the exact location of the bug, it did okay. But when the location was imperfect (like a real-world GPS that sometimes gets it wrong), the AI's performance crashed.
The Lesson: AI is great at fixing things, but it's terrible at finding the problem if it doesn't have a good map. If the "Fault Localization" tool is sloppy, the AI will waste time fixing the wrong part.

4. The "Indentation" Trap (The Python Problem)

Python is a programming language that is very picky about how lines are spaced (indentation). It's like a recipe where if you don't indent the steps correctly, the cake burns.

The Finding: Most AI models struggled with Python because they kept messing up the spacing. They wrote code that was logically correct but syntactically broken because the "tabs" were in the wrong place.
The Exception: One model (Gemini) was surprisingly good at keeping its "tabs" straight, which is why it performed better on Python than the others.
The Takeaway: If you use AI to fix Python code, you need a "spellchecker" (a formatting tool) to clean up the spacing before you run the code.

5. Open Source vs. The "Big Tech" Giants

The study compared Open Models (like Llama or DeepSeek, which anyone can download and run) with Closed Models (like GPT-4 or Claude, which are owned by big companies and you can't see inside).

The Analogy: Think of Closed Models as luxury brand cars (expensive, high performance, but you can't see under the hood). Open Models are like enthusiast builds (you can see everything, modify them, but they used to be less reliable).
The Finding: Historically, the "luxury brands" were faster and smarter. However, the study shows that the "enthusiast builds" (Open Models) are catching up fast. Some of the new open models are now performing just as well as the expensive, closed ones. This is great news for companies who want to avoid being locked into one vendor.

6. Simple vs. Complex Fixes

The Analogy: Fixing a flat tire (single-line bug) vs. rebuilding an entire transmission (multi-part bug).
The Finding: You might expect AI to fail miserably on complex, multi-part bugs. Surprisingly, they handled complex fixes reasonably well. They didn't get as much worse as expected when the problem got harder.

The Bottom Line

This paper tells us that Automated Program Repair is getting real, but it's not magic yet.

Don't rely on one AI: Use a team of different models depending on the programming language.
Give them clues: Always feed the AI the error messages, not just the code.
Fix the map first: If you can't accurately tell the AI where the bug is, it won't matter how smart the AI is; it will fail.
Open is rising: You don't need to pay for the most expensive "black box" models anymore; the open-source alternatives are becoming powerful enough to do the job.

The researchers are essentially saying: "We have the tools, but we need to stop pretending they are perfect and start building systems that use them realistically."

1. Problem Statement

Automated Program Repair (APR) aims to automatically generate bug fixes to assist developers. While Large Language Models (LLMs) have shown promise in this domain, current APR techniques suffer from a lack of generalizability. Existing research often relies on:

Single-model evaluations: Focusing on one specific LLM rather than comparing a diverse set.
Idealized assumptions: Assuming "perfect" Fault Localization (FL) (i.e., the exact buggy line or function is known), which is unrealistic in practice.
Limited scope: Evaluating on a single programming language (typically Java) or specific benchmarks, leading to benchmark overfitting.
Ad-hoc strategies: Using inconsistent prompting methods without systematic comparison of repair ingredients (e.g., test errors, localization hints).

The authors argue that the APR community needs a comprehensive, realistic, and cross-language empirical evaluation to understand the true capabilities and limitations of state-of-the-art LLMs.

2. Methodology

The study conducts an extensive empirical evaluation involving 13 state-of-the-art LLMs (both open and closed-source) across four programming languages (Java, JavaScript, Python, PHP).

Models Evaluated (13 Total)

Closed/Proprietary: Claude 3.7 Sonnet, Claude 3.5 Haiku, Gemini 1.5 Pro, Gemini 2.0 Flash, OpenAI o3-mini, OpenAI GPT-4o.
Open/Weight-Accessible: Llama 3.3 70B, CodeLlama 70B/13B, DeepSeek Coder 32B, DeepSeek R1 (distilled), Qwen 2.5 Coder 32B, CodeGemma 7B.

Benchmarks

The study utilized reproducible subsets of four major benchmarks, filtering for single-function bugs to ensure fair comparison:

Java: Defects4J (264 reproducible single-hunk bugs).
Python: BugsInPy (82 reproducible single-hunk bugs).
JavaScript: BugsJS (72 reproducible single-hunk bugs).
PHP: BugsPHP (129 reproducible single-hunk bugs).
Total: ~195,000 patches generated and evaluated.

Experimental Design & Prompts

The authors tested four distinct prompt configurations to isolate variables:

Base Prompt: Only the buggy function code (simulating zero-shot repair).
Test Prompt: Base + failing test case names, source code, and error messages.
Perfect Line-level Localization (LL): Base + explicit comments highlighting the exact buggy line (simulating perfect FL).
Automated FL: Base + top-3 suspicious functions identified by a spectrum-based tool (FLACOCO), simulating realistic, imperfect localization.

Evaluation Metrics

Pass@k: The probability that at least one of $k$ generated patches passes all test cases. The study reports Pass@1 and Pass@5.
Sampling: To mitigate LLM non-determinism, 15 samples were generated per bug per model ( $n=15$ ), and statistical significance was tested using the Wilcoxon signed-rank test.
Patch Complexity: Bugs were categorized by complexity: single-line, single-hunk, and multi-hunk.

3. Key Contributions

Comprehensive Cross-Language Evaluation: The first large-scale study comparing 13 diverse LLMs across Java, JS, Python, and PHP, revealing that no single model dominates all languages.
Realistic Fault Localization Analysis: A systematic investigation into the impact of imperfect FL, moving beyond the "perfect FL" assumption common in literature.
Ensemble Strategy Validation: Demonstration that combining outputs from multiple models (a "committee" approach) significantly improves repair rates compared to single-model usage.
Open vs. Closed Model Tracking: Evidence that open-weight models are rapidly catching up to, and in some cases surpassing, proprietary models in APR tasks.
Open-Source Artifact: Release of the experimental setup, results, and generated patches to facilitate reproducibility.

4. Key Results

RQ1: Generalization Across Languages

No Universal Solver: Different LLMs perform best on different languages. For example, Claude 3.7 Sonnet excelled in Java, Claude 3.5 Haiku in JavaScript, DeepSeek R1 (dist.) in PHP, and Gemini 2.0 Flash in Python.
Language Specialization: Practitioners should not rely on a single model for a polyglot codebase; a language-specific model selection or committee is required.
Python Indentation Issue: All models struggled significantly with Python due to indentation errors (off-side syntax), with Gemini models performing notably better in this specific regard.

RQ2: Impact of Fault Localization (FL)

Test Information is Critical: Providing failing test cases and error messages yielded the most significant performance boost (up to +47% Pass@1 for some models).
Perfect Localization vs. Test Info: Surprisingly, providing perfect line-level localization hints often provided less value than providing test error messages. In some cases (e.g., PHP), line-level hints even degraded performance.
The "Imperfect FL" Bottleneck: When using automated FL (FLACOCO) instead of perfect localization, performance dropped drastically (e.g., Pass@1 dropped from ~19% to ~2-3% for some models). This indicates that current APR pipelines are highly sensitive to the quality of the localization step.

RQ3: Patch Complexity

Robustness to Complexity: LLMs handled multi-hunk bugs (bugs spanning multiple code locations) surprisingly well. While performance decreased as complexity increased (Single-line > Single-hunk > Multi-hunk), the drop was not as severe as anticipated.
Multi-hunk Success: 9 out of 48 model/language combinations actually performed better on single-hunk than single-line bugs, and 10 performed better on multi-hunk than single-hunk, suggesting LLMs have strong contextual reasoning capabilities.

RQ4: Open vs. Closed Models

Closing the Gap: While closed models generally led initially, open models (specifically DeepSeek R1 and Llama 3.3) have shown rapid improvement.
DeepSeek R1: The distilled DeepSeek R1 model achieved top-tier performance, often matching or exceeding closed models, particularly in PHP and Python.
Trend: The gap is narrowing, suggesting open models are becoming viable alternatives for practitioners seeking digital sovereignty.

Ensemble Results

Committee of Experts: Combining models (e.g., taking 2/3 generations from the best model and 1/3 from a complementary one) consistently improved Pass@5 scores. For instance, combining o3-mini and DeepSeek R1 improved JavaScript repair rates from 68.00% to 71.68%.

5. Significance and Implications

For Practitioners:
- Do not rely on a single "best" model; select models based on the specific programming language.
- Integrate test error messages into prompts; they are more valuable than static code hints.
- Consider ensemble approaches (model committees) to maximize fix rates.
- Be aware that Fault Localization is a major bottleneck; improving FL tools is as critical as improving the repair model itself.
For Researchers:
- Future APR research must move away from "perfect FL" assumptions and evaluate pipelines with realistic, noisy localization tools.
- Benchmarking must be cross-language to avoid overfitting to a single ecosystem (e.g., Java).
- The rapid rise of open models suggests a shift toward democratized, reproducible APR research.

Conclusion: The paper concludes that while LLMs are powerful APR tools, their generalizability is limited by language specificity and the quality of fault localization. A reliable APR system requires a tailored combination of models, robust test information, and realistic evaluation environments.