Technical Summary: Characterizing the Failure Modes of LLMs in Resolving Real-World GitHub Issues

Problem Statement

While Large Language Models (LLMs) have shifted the paradigm of automated program repair from isolated code snippet generation to autonomous repository-level issue resolution, their reliability remains significantly below that of human experts. Current benchmarks, such as SWE-bench, indicate that state-of-the-art models resolve only approximately 76.8% of verified issues. A critical gap exists in understanding why these failures occur. Existing evaluation protocols largely rely on binary pass/fail metrics based on final test execution, which obscures the underlying reasoning processes. Furthermore, prior failure analyses often treat LLMs as simple text generators rather than autonomous agents interacting with complex environments, failing to distinguish between general debugging challenges and machine-specific cognitive failure modes. This study addresses the fundamental question: How do LLMs fail differently from human developers in real-world software repair tasks?

Methodology

The authors conducted an empirical study using the SWE-bench Verified dataset, a curated subset of the original benchmark where human engineers have confirmed that issues contain sufficient context for resolution and are evaluated via fair, deterministic harnesses.

• Models Evaluated: Three frontier proprietary models were tested: Claude 4.5 Sonnet, Gemini 3 Pro, and GPT-5.
• Execution Framework: To isolate intrinsic model reasoning from complex agentic scaffolding, the study utilized mini-SWE-agent, a minimalist execution environment. This framework relies on standard bash commands and records complete interaction trajectories (prompts, shell commands, model responses, and environment feedback).
• Experimental Design:
  • A representative subset of 100 tasks was randomly selected from the 500-task benchmark.
  • Each task was executed three times per model, resulting in 900 total repair attempts.
  • All generated patches were evaluated using the official SWE-bench evaluation service, which requires passing all FAIL_TO_PASS tests and maintaining all PASS_TO_PASS tests.
• Failure Analysis: The study focused on the 243 failed attempts (27% of the total). A rigorous manual root-cause analysis was performed by three authors who independently examined execution trajectories, generated patches, reference patches, and evaluation traces. Failures were categorized into a unified taxonomy covering five stages of the repair pipeline.

Key Contributions

1. Unified Failure Taxonomy: The paper constructs a classification framework tailored to repository-level repair, dividing failures into five distinct stages:
   • Problem Understanding
   • Localization
   • Strategy & Logic
   • Implementation & Execution
   • Validation & Harness Constraints
2. Characterization of Divergent Patterns: The study systematically identifies where LLM behavior diverges from human reasoning, revealing that models excel at mechanical tasks (like fault localization) but struggle with high-level cognitive tasks (like strategy synthesis).
3. Actionable Insights for Agent Architecture: The authors derive practical implications for future agent designs, emphasizing that providing missing contextual information and hidden task constraints is often more critical than simply increasing raw model intelligence.
4. Replication Package: The authors release a comprehensive dataset including 900 execution traces and 243 expert-annotated failure reports to support future investigations.

Key Results and Findings

1. Failure Distribution and Cognitive Bottlenecks (RQ1)

Contrary to the assumption that fault localization is the primary hurdle, the study found that cognitive reasoning is the dominant source of failure.

• Strategy & Logic (37%): This is the most error-prone stage. Models frequently produce partial fixes (S1) or incorrect strategies with side effects (S2), failing to synthesize comprehensive repair logic even when the fault is correctly identified.
• Problem Understanding (29.2%): Failures often stem from misinterpreting issue descriptions or, notably, anchoring on misleading hints (P2). Models exhibit "alignment-induced sycophancy," prioritizing user-provided speculative hints over system invariants.
• Localization (7%): Surprisingly, pure localization failures are rare. Models are proficient at navigating large codebases and identifying primary fault locations.
• Validation & Harness Constraints (16.5%): A significant portion of failures arises because rigid test harnesses reject semantically correct patches due to superficial discrepancies (e.g., strict type assertions or formatting mismatches) rather than functional incorrectness.

2. Model Behavioral Patterns (RQ2)

• Distinct Bottlenecks: While all models share vulnerabilities, their error distributions differ. Claude shows a balanced split between understanding and strategy errors. Gemini and GPT concentrate failures heavily in the Strategy & Logic stage.
• Execution Stability: Gemini demonstrated the highest execution stability (69/100 tasks consistently solved across 3 trials), whereas Claude and GPT exhibited higher stochastic variance, suggesting that single-run evaluations may underestimate their peak capabilities.
• Reasoning Inertia: During failed trajectories, all models showed severe inefficiency. Instead of early stopping, they entered repetitive, unproductive reasoning loops, with GPT showing a significant increase in token usage during failures without resolving the underlying logical flaws.

3. Task Features and Interventions (RQ3)

The study linked specific failure modes to task characteristics:

• Implicit Rules (P1): Failures occur when tasks require domain-specific knowledge (e.g., cryptographic protocols) absent from the prompt.
• Textual Distractors (P2): Models are easily misled by TODO comments or speculative hints in issue reports.
• Structural Dispersal (L1): Models struggle to propagate fixes across architectural boundaries (e.g., missing subclasses in different modules).
• Intervention Efficacy: Targeted prompt interventions that explicitly expose hidden constraints (e.g., warning models about incomplete issue descriptions or strict type requirements) significantly improved success rates for Validation & Harness failures (V1, V2). However, these interventions were less effective for deep domain knowledge gaps (P1), suggesting a need for active knowledge retrieval rather than just better prompting.

Significance and Claims

The paper claims that the primary bottleneck in autonomous software repair has shifted from mechanical tool execution and fault localization to high-order cognitive reasoning.

• Re-evaluating Benchmarks: The study highlights a "specification-oracle gap," where current benchmarks may systematically underestimate model capabilities by penalizing functionally correct patches that fail rigid, undocumented test assertions.
• Future Directions: The authors argue that achieving human-level proficiency requires a transition from retrieval-centric architectures to logic-driven agents. Future systems must prioritize systematic planning, critical skepticism (to resist misleading hints), and active knowledge elicitation (e.g., retrieving external documentation) to bridge the expertise gap between AI and human developers.
• Robustness: The findings suggest that robustness in real-world deployment requires addressing not just model intelligence, but also the stability of execution trajectories and the alignment of evaluation harnesses with semantic correctness.