Technical Summary: Rethinking Automated Program Repair

Problem Statement

Software bugs remain a critical challenge in development, necessitating effective Automated Program Repair (APR) techniques. While Large Language Model (LLM)-based APR systems have demonstrated promise, prior research has primarily focused on overall repair effectiveness using established benchmarks like Defects4J. These studies often rely on single benchmarks that may lack diversity or contain data leakage issues, and they frequently utilize datasets (e.g., QuixBugs, IntroClass) composed of simple, single-hunk bugs that may be insufficiently challenging for modern LLMs.

Furthermore, existing evaluations often overlook critical dimensions: the impact of varying bug complexity, the influence of fault localization (FL) precision (ranging from accurate line-level to vague or absent), and the cost-efficiency of different LLM choices and reasoning settings. The reliance on commercial APIs introduces non-negligible computational costs, particularly for reasoning-intensive settings, yet these costs are rarely standardized or analyzed in relation to repair performance.

Methodology

The authors present a comprehensive empirical analysis of LLM-based APR, addressing these gaps through a multi-dimensional framework:

• Dataset Construction: To mitigate data contamination and ensure diversity, the study constructs a curated dataset from AtCoder, a competitive programming platform. The authors collected 1,914 C++ submissions from October 2025, pairing incorrect programs with their correct patches. After filtering for computational feasibility (limiting fixes to five lines), the final benchmark consists of 640 bugs.
• Complexity Categorization: Bugs are categorized using two metrics:
  1. Hunk-line-level complexity: Based on the number of hunks and fixed lines (Single-Hunk-Single-Line, Single-Hunk-Multi-Line, Multi-Hunk-Single-Line, Multi-Hunk-Multi-Line).
  2. AST operation-type complexity: Based on the diversity of edit operations (insert, delete, relabel, move) required, measured via srcML and GumTreeDiff.
• Experimental Setup:
  • APR Techniques: Two frameworks are evaluated: ChatRepair (iterative generation/refinement) and CodeCorrector (end-to-end analysis and generation).
  • LLMs: Three model families are tested: DeepSeek (V3.2, V4-pro), GPT (GPT-4o-mini, GPT-5), and Llama (3.1-8B, 3.1-405B).
  • Fault Localization (FL) Strategies: Experiments are conducted under three conditions: precise line-level FL, vague hunk-level FL, and no FL.
  • Reasoning Settings: The study evaluates both non-reasoning modes and various reasoning configurations (e.g., Chain-of-Thought, high/max effort) where supported.
• Metrics: Beyond standard repair effectiveness (number of repaired bugs, Top-k, plausible patches), the study introduces standardized cost-efficiency metrics (Repaired Bugs/$ and Attempts/$) to quantify the trade-off between performance and computational cost.

Key Contributions

1. Empirical Analysis of Complexity and Localization: The study systematically analyzes how bug complexity and FL precision impact LLM-based APR performance, moving beyond simple benchmarks to include structurally complex, multi-line bugs.
2. Cost-Efficiency Standardization: The authors propose and apply unit-standardized cost metrics ($1 base) to enable fair comparisons of repair effectiveness against computational costs across different LLMs and reasoning settings.
3. End-to-End Feasibility: By evaluating settings with accurate, vague, and no fault localization, the study demonstrates the potential feasibility of end-to-end LLM-based APR approaches that do not rely on external FL tools.
4. Extensible Infrastructure: The paper provides an experimentation infrastructure and dataset designed to support future replication and systematic evaluation of LLM-based APR.

Key Results

Impact of Bug Complexity (RQ1)

• Complexity Correlation: Repair effectiveness decreases as bug complexity increases. Simplest bugs (Single-Hunk-Single-Line and single-operation types) achieve the highest repair rates (~75-78%).
• Resilience of LLMs: Despite the difficulty, LLM-based APR techniques maintain considerable effectiveness even on structurally complex bugs. Over 50% of moderately complex bugs (e.g., Multi-Hunk-Multi-Line, two-to-three operation types) are successfully repaired.
• Statistical Significance: Chi-square tests confirm that four-operation-type bugs are significantly harder to repair than simpler types, yet repair rates remain non-trivial.

Impact of Fault Localization (RQ2)

• Precision Matters: More precise FL (line-level) generally yields better repair effectiveness and higher patch consistency (Code Consistency Rate).
• Performance Divergence: The performance gap between APR techniques widens significantly as FL becomes less precise. While ChatRepair and CodeCorrector perform similarly under line-level FL, ChatRepair demonstrates superior stability and generalizability under vague (hunk-level) or no FL settings.
• Complexity Interaction: As bug complexity increases, the relative importance of precise FL decreases, suggesting that repairing complex bugs relies more heavily on the model's structural understanding and reasoning capabilities than on precise localization.

Cost-Efficiency and Reasoning Trade-offs (RQ3)

• Non-Linear Gains: Higher-cost LLMs and stronger reasoning settings do not consistently yield proportional improvements in repair effectiveness.
• Model Specificity: The benefit of reasoning strategies varies by model family. GPT-5 achieves the highest overall repair effectiveness in non-reasoning mode (175/211 bugs) but shows minimal gain from enabling reasoning (only 3 additional bugs fixed). Conversely, DeepSeek-V3.2 and DeepSeek-V4-pro benefit significantly from reasoning modes, repairing many additional bugs that non-reasoning configurations miss.
• Cost Disparities: Cost-efficiency is highly dependent on the specific APR framework used. For instance, ChatRepair combined with DeepSeek-V3.2-chat achieves approximately 71.43 repaired bugs/$, whereas the same model with CodeCorrector achieves 47.62 bugs/$. In contrast, GPT-5 (non-reasoning) yields only 2.94 repaired bugs/$.
• Strategic Recommendation: A hybrid approach is suggested: using cost-efficient LLMs (e.g., DeepSeek-V3.2 non-reasoning) for the bulk of repairs, followed by advanced LLMs (e.g., GPT-5 non-reasoning) for remaining unresolved bugs, offers an optimal balance between performance and cost.

Significance and Claims

The paper claims that while structurally complex bugs and imprecise fault localization present challenges, well-designed LLM-based APR techniques retain substantial repair effectiveness. Specifically:

• Over 50% of moderately complex bugs can be repaired by low-cost LLM-based APR techniques.
• The gap in repair effectiveness between different APR techniques becomes more pronounced as fault localization becomes less precise, highlighting the need for robust, generalizable frameworks.
• There is a nontrivial trade-off between repair effectiveness and computational cost; simply using the most expensive model or enabling reasoning does not guarantee the best economic outcome.
• The study provides practical guidance for selecting LLMs and reasoning settings based on the specific constraints of repair performance versus economic cost, advocating for a balanced, multi-stage approach in practical deployment.

The authors conclude that future APR evaluations should move beyond simple benchmarks to include diverse complexity levels and explicitly account for cost-efficiency to better reflect real-world deployment scenarios.