Technical Summary: SWE-RPG

Problem Statement

Large language model (LLM)-powered coding agents are increasingly deployed to modify existing code repositories for tasks such as bug fixing and feature implementation. However, current repository-level benchmarks (e.g., SWE-bench, RACE-bench) primarily evaluate success based on a binary "pass/fail" outcome of the final code patch against executable tests. This outcome-centric approach fails to diagnose why an agent fails. Resolving a repository-level issue requires a complex chain of interdependent reasoning: recovering explicit and implicit requirements, formulating a repository-grounded implementation plan, and translating that plan into correct code. A final verdict cannot distinguish whether a failure stemmed from misunderstanding implicit constraints, generating an insufficient plan, or writing incorrect code. Consequently, existing benchmarks lack the granularity to identify specific bottlenecks or provide clear directions for improving agent architectures.

Methodology

To address these limitations, the authors introduce SWE-RPG (Software Engineering - Requirement, Planning, and Generation), a unified benchmark designed for stage-aware evaluation.

Benchmark Construction

SWE-RPG comprises 163 tasks derived from 31 mature Python and Java repositories, including 113 bug fixes and 50 feature additions. The construction process involved rigorous filtering from over 2,000 initial issue-PR pairs to ensure reproducible environments and stable tests.

The core innovation of SWE-RPG is the construction of validated Ground-Truth (GT) references for intermediate stages, not just the final patch:

1. Requirement Clarification GT: Recognizing that initial issue descriptions often omit critical implementation details, the authors developed a practitioner-guided taxonomy of six categories (Functional Intent, Business Semantics, Technical Context, Interface/Protocol Specifications, Code Structure/Conventions, and Data-Structure Semantics). Using synthesis agents and validation against repository evidence (issues, PRs, patches), they generated "Clarification GTs" in the form of QA pairs that make implicit requirements explicit.
2. Implementation Planning GT: The authors synthesized actionable, step-by-step implementation plans. Crucially, these plans were validated for reproducibility: a coding agent was tasked with generating code from each plan step to verify it produced a functionally equivalent sub-patch to the gold reference. This ensures the plans are sufficient to generate correct code.
3. Code Generation GT: The final gold patches were sourced directly from merged Pull Requests, with task-relevant tests serving as the executable oracle.

Evaluation Protocol

The benchmark evaluates agents across three stages:

• Requirement Clarification: Measuring the agent's ability to recover implicit requirements against the Clarification GT.
• Planning: Assessing whether the agent's generated plan covers the necessary steps and constraints defined in the Planning GT.
• Code Generation: Evaluating the final patch against executable tests.

For unresolved runs, an LLM judge attributes the failure to the earliest deviating stage (Requirement, Planning, Code Generation, or Others), enabling a retrospective diagnosis of the agent's trajectory.

Key Contributions

1. SWE-RPG Benchmark: The first repository-level benchmark to jointly evaluate agents' capabilities in requirement clarification, implementation planning, and code generation, supported by validated intermediate ground truths.
2. Stage-Aware Diagnosis: A protocol that moves beyond binary success metrics to attribute failures to specific reasoning stages, revealing distinct capability bottlenecks.
3. Comprehensive Evaluation: An evaluation of three popular coding agent frameworks (Claude Code, Codex, OpenCode) paired with six recent LLM backends (including Claude-Sonnet-5, GPT-5.6-Terra, and MoonshotAI-Kimi-K3).

Results

The evaluation of 18 agent-LLM configurations yielded the following findings:

• Low Overall Resolution: The average resolved rate across all configurations was only 31.5%, indicating significant challenges in repository-level issue resolution even for state-of-the-art models.
• Implicit Requirement Recovery as the Primary Bottleneck: Failure attribution analysis revealed that 46.7% of all agent runs failed primarily during requirement clarification or implementation planning. Specifically, implicit requirement recovery accounted for 24.5%–46.0% of agent runs, making it the most common failure point.
• Stage-Specific Failure Profiles: Different agent-LLM pairings exhibited distinct failure modes. For instance, while some configurations had similar overall resolution rates, their underlying failure distributions varied significantly (e.g., high requirement failure vs. high code generation failure).
• Coverage Gaps: Analysis of agent trajectories against GTs showed that while agents reliably covered high-level intent, their coverage dropped significantly for implementation-facing categories such as interface contracts, code structure, and data semantics. Similarly, planning coverage decreased from identifying target locations to specifying implementation approaches and validation strategies.

Significance and Claims

The paper claims that SWE-RPG provides a necessary shift from outcome-based evaluation to process-aware diagnosis. By providing validated references for intermediate reasoning steps, the benchmark allows researchers to:

• Identify that the primary barrier to success is not necessarily code generation, but the recovery of implicit requirements and the formulation of sufficient implementation plans.
• Move beyond "black box" failure analysis to pinpoint specific stages where agents diverge from correct trajectories.
• Suggest that future improvements in coding agents should prioritize implicit-constraint recovery and plan-to-code reproducibility rather than solely focusing on final patch generation.

The authors maintain a modest stance, acknowledging limitations such as the current focus on Python and Java, the lack of human-in-the-loop interaction assessment, and the reliance on LLM-assisted construction of ground truths (though manually validated). They position SWE-RPG as a foundational step toward more interpretable and effective repository-level coding agents.