On the Role of Fault Localization Context for LLM-Based Program Repair

1. Problem Statement

Automated Program Repair (APR) using Large Language Models (LLMs) has advanced significantly, yet the role of Fault Localization (FL) context remains underexplored. While traditional APR assumes that narrowing the search space to precise bug locations improves repair accuracy, LLMs operate with large context windows and may benefit from broader semantic information.

The core problem addressed is the lack of understanding regarding:

• How much localization context is needed (file, element, or line level).
• Whether expanding context beyond the ground-truth buggy location improves or degrades performance.
• How different retrieval strategies (rule-based heuristics vs. LLM-based semantic retrieval) compare across these granularities.
• The trade-offs between context quantity, quality, and computational cost.

2. Methodology

The authors conducted a large-scale factorial empirical study to systematically isolate the effects of context granularity and expansion strategies.

• Dataset: 500 instances from SWE-bench Verified, a benchmark of real-world Python software engineering issues.
• Model: GPT-5-mini was used as the repair model to balance reasoning capability with computational cost.
• Experimental Design:
  • Ground-Truth Assumption: To isolate the impact of context expansion from localization accuracy, the study used ground-truth (GT) patches to extract the exact buggy files, elements, and lines. This simulates a "perfect localization" scenario.
  • Factorial Configurations: The study evaluated 61 distinct configurations combining three orthogonal dimensions:
    1. File-Level: No files, Buggy files only, Rule-based expansion (imports/dependents), or LLM-retrieved relevant files.
    2. Element-Level: No elements, Buggy elements only, Call Graph expansion, or LLM-retrieved relevant elements.
    3. Line-Level: No lines, Buggy lines only, Context window (±10 lines), Static code slicing, or LLM-retrieved lines.
  • Evaluation Metric: Resolution rate (percentage of instances where the generated patch passes all tests).
  • Statistical Analysis: Wilcoxon signed-rank tests were used to determine statistical significance between paired configurations.

3. Key Contributions

1. Comprehensive Granularity Analysis: The first study to systematically examine the interaction between file, element, and line-level context in LLM-based APR, moving beyond single-granularity studies.
2. Retrieval Strategy Comparison: A rigorous comparison between structural heuristics (rule-based imports, call graphs, static slicing) and semantic retrieval (LLM-based selection) across all granularities.
3. Counter-Intuitive Findings: Challenging the assumption that "more context is always better," specifically demonstrating that expanding line-level context often harms performance.
4. Cost-Benefit Analysis: Quantifying the token and file count costs of different retrieval strategies, revealing that LLM-based retrieval is often cheaper and more effective than rule-based methods.

4. Key Results & Findings

A. File-Level Context (Dominant Factor)

• Impact: Introducing file-level context yields a massive 15–17× improvement over a no-file baseline (3.6% vs. ~60% resolution).
• Expansion: Expanding beyond the "buggy file" to include relevant files significantly improves performance.
• Strategy: LLM-based file retrieval outperforms rule-based expansion (imports/dependents) in 17/20 configurations.
• Cost: LLM-based retrieval is more cost-effective, retrieving fewer files and using fewer tokens on average than rule-based methods, despite the overhead of an initial LLM call for ranking.

B. Element-Level Context (Conditional Benefits)

• Impact: Adding element context (functions/classes) generally helps, but gains are conditional on file context quality.
• Strategy: LLM-retrieved elements outperform Call Graph-based expansion. Call graphs often introduce irrelevant structural noise.
• Synergy: The best results occur when LLM-retrieved elements are paired with precise line localization.

C. Line-Level Context (Noise Amplification)

• Impact: Unlike files and elements, expanding line-level context frequently degrades performance.
• Findings:
  • Context Windows (±10 lines) and Static Code Slicing often introduce noise (irrelevant code paths, logging) that dilutes the repair signal.
  • Precise "Buggy Lines" (GT) generally outperform expanded line strategies.
  • LLM-retrieved lines perform better than slicing or windows but still rarely beat the precise "buggy line" baseline.
• Conclusion: Line-level context should be used for precision (identifying the edit location), not for understanding (which is handled by file/element context).

D. Optimal Strategy & Interactions

• Best Configuration: The highest resolution rate (63.4%, 317/500 instances) was achieved by combining LLM-retrieved files, LLM-retrieved elements, and precise Buggy lines.
• Complementarity Principle: The most effective strategy combines broad semantic understanding at higher abstraction levels (files/elements) with precise, minimal localization at the line level.
• Uniformity is Suboptimal: Uniformly broad context (expanding everything) or uniformly narrow context (only buggy lines) performs worse than hybrid strategies.

5. Significance and Practical Implications

• Redefining APR Pipelines: The study suggests that APR systems should prioritize LLM-based semantic retrieval for file and element selection over traditional structural heuristics (imports/call graphs).
• Cost Efficiency: LLM-based retrieval reduces token usage and context size while improving accuracy, making it a more scalable solution.
• Context Engineering: Developers should avoid blindly expanding line-level context. Instead, they should focus on retrieving the correct files and functions semantically, while keeping the line context tight to the specific edit location.
• Adaptive Strategies: Since no single configuration works for all bugs, the authors propose adaptive or ensemble strategies that dynamically adjust context granularity based on bug complexity (e.g., broader context for cross-module bugs, narrow context for localized edits).

6. Conclusion

The paper concludes that context quality outweighs quantity in LLM-based APR. While expanding file and element context is crucial for semantic understanding, expanding line-level context often introduces noise. The optimal approach is a hybrid strategy: using LLMs to semantically retrieve broad file and element context, while relying on precise localization for the actual code edits. This challenges the traditional APR dogma that "more precise localization is always better" and "more context is always better," proposing instead a nuanced, multi-granularity approach.