Technical Summary: RGFL: Reasoning Guided Fault Localization for Automated Program Repair Using Large Language Models

Problem Statement

Automated Program Repair (APR) relies heavily on Fault Localization (FL) to identify the specific code regions responsible for a bug before attempting to generate a fix. In modern, large-scale software repositories (often exceeding millions of tokens), providing the entire codebase to Large Language Models (LLMs) is infeasible due to context window limitations. Consequently, APR systems must first isolate a small, relevant subset of code.

Traditional FL techniques (e.g., spectrum-based analysis, static analysis, or information retrieval based on lexical similarity) often fail to capture deep semantic links between bug reports and code, particularly when bugs stem from subtle logic errors or domain-specific behaviors. While recent LLM-based approaches attempt to localize faults by retrieving code based on similarity or asking models to list relevant functions, they often lack explicit reasoning about why a specific code element is relevant. This can lead to the selection of superficially related but causally incorrect elements, hindering the subsequent repair phase.

Methodology

The authors propose RGFL (Reasoning Guided Fault Localization), a novel hierarchical approach that integrates explicit LLM reasoning into the localization pipeline. The method is designed as a modular enhancement to existing pipeline-based APR systems (specifically integrated with Agentless).

Core Components

1. Hierarchical Reasoning Module:
   Instead of asking an LLM to rank a large set of candidates simultaneously (which can dilute attention), RGFL processes candidates individually.

   • File-Level Localization: The system first retrieves a candidate set of files using a combination of LLM prompting and embedding-based search. For each candidate file, the LLM is prompted to generate a natural language explanation of the file's functionality in the context of the specific bug report. These explanations are then used to re-rank the files.
   • Element-Level Localization: Within the top-ranked files, the system extracts classes, functions, and global variables. Similar to the file level, the LLM generates reasoning for each element regarding its relevance to the bug. These reasoning outputs serve as the signal for ranking elements.

2. Two-Stage Ranking Scheme:
   The method employs a two-stage ranking process combining:

   • LLM-based Ranking: The LLM evaluates the generated reasoning against the bug report to produce a ranked list.
   • Embedding-based Ranking: As a comparison and alternative, the generated reasoning and bug reports are embedded (using models like Gemini-embedding-001, Voyage 3.5, or text-embedding-3-small) and ranked via cosine similarity. The study finds that direct LLM ranking outperforms embedding-based ranking of the reasoning.

3. Integration and Scope:

   • RGFL replaces the file and element localization stages of the Agentless pipeline but leaves line-level localization and patch generation unchanged. The authors argue that line-level reasoning is ill-posed for isolated lines and computationally prohibitive for large repositories.
   • The system uses a "Reasoning-first" strategy: given a fixed set of candidates, the LLM explains each one individually before ranking.

Key Contributions

1. Novel Hierarchical Reasoning Module: The design of a system where LLMs generate structured, bug-specific explanations for candidate files and elements, using these explanations as the primary signal for ranking rather than raw code similarity.
2. Systematic Evaluation and Ablation: The authors integrate RGFL into Agentless and conduct a systematic study isolating the impact of file vs. element localization. They introduce a counterfactual upper-bound analysis to quantify how much each localization stage contributes to the final repair success.
3. Cross-Dataset and Cross-Language Validation: Evaluation is performed on SWE-bench Verified, SWE-bench Lite, and SWE-bench Java, covering both Python and Java projects.
4. Fine-Grained Error Taxonomy: The paper categorizes failures into file-level, element-level, line-level, and repair-level misses, using controlled counterfactual injections of ground-truth information to estimate performance upper bounds.

Experimental Results

The evaluation demonstrates consistent improvements over state-of-the-art baselines, including Agentless, OpenHands, and AutoCodeRover.

• File-Level Localization (SWE-bench Verified):

  • Hit@1 improved from 71.4% (Agentless) to 85% (RGFL).
  • MRR improved from 81.8% to 88.8%.
  • Embedding-based re-ranking of reasoning failed to match the performance of direct LLM ranking, highlighting the necessity of natural language reasoning over vector similarity alone.

• Element-Level Localization:

  • Exact Match (within top-3 files) increased from 36% (Agentless) to 69% (RGFL).
  • This represents a 91.67% improvement over the baseline Agentless system.

• End-to-End Repair Success:

  • Integrating RGFL into Agentless increased the Resolved Rate on SWE-bench Verified from 51.6% to 58.2%.
  • This corresponds to a 12.8% absolute improvement in end-to-end repair success.

• Generalizability:

  • RGFL maintained superior performance on SWE-bench Java, outperforming baselines like OpenHands and OrcaLoca, demonstrating robustness across programming languages.

• Counterfactual Analysis:

  • The study revealed that while file localization errors are less frequent, correcting them yields the highest payoff per instance.
  • Interestingly, in some cases where the correct file was identified, providing only the file (without specific element localization) allowed the model to generate correct patches, whereas over-constraining the model to specific elements sometimes led to failure. This suggests that the granularity of localization guidance requires careful balancing.

Significance and Claims

The paper claims that explicit reasoning is a critical, underutilized mechanism in LLM-based APR. By shifting the paradigm from "selecting elements that sound related" to "selecting elements that are causally responsible for the failure," RGFL significantly narrows the search space for repair agents.

The authors assert that their work provides a practical mechanism to adapt LLMs to large, real-world software systems where context limits are a bottleneck. The results suggest that the cost of additional reasoning (approx. 3.7x computational cost per sample) is justified by the substantial gains in localization accuracy and final repair success. The paper concludes that while fault localization is a critical step, the optimal way to present localization information (file vs. element vs. line) to the repair model remains an open area for research, as overly granular constraints can sometimes hinder the model's ability to synthesize a correct patch.