Technical Summary: SWE-Bench ProMax

Problem Statement

As AI coding agents advance, existing software engineering benchmarks are rapidly saturating, raising concerns about whether rising scores reflect genuine capability improvements or merely the limitations of current benchmark designs. A recent audit of SWE-bench Verified revealed that nearly 60% of unsolved instances contained flawed test suites—either overly narrow (rejecting correct solutions) or overly broad (checking unstated requirements)—and that frontier models could reproduce gold patches from training data. Furthermore, current benchmarks primarily focus on isolated bug fixes or feature implementations, often involving single-file modifications. This fails to capture the complexity of code refactoring, a critical, long-horizon task in professional development that requires coordinated, behavior-preserving changes across many files. Existing refactoring benchmarks are either limited in scale, restricted to a single programming language, or lack rigorous human verification of test quality.

Methodology

The authors introduce SWE-Bench ProMax, an expert-curated, multilingual benchmark designed to address these gaps through a rigorous three-stage construction pipeline:

1. Data Collection: The authors identified candidate repositories on GitHub meeting specific criteria (≥500 stars, approved open-source license, primary language among Python, Java, TypeScript, Go, C, C++, or Rust). They extracted commits submitted after January 2025 containing the keyword "refactor" (excluding "bug fix") that modified both test and non-test files. This yielded an initial pool of 29,782 candidates.
2. Environment Construction: For each candidate, an isolated Docker environment was constructed containing the repository at the pre-refactoring state with all dependencies installed. The gold patch (source and test changes) was applied, and the test suite was executed. Instances where the environment failed to build or the gold patch did not pass all tests were discarded.
3. Filtering and Problem Rewriting: This stage involved human experts assisted by LLMs to ensure high quality:
   • Commit Analysis: Experts analyzed diffs to understand scope and intent.
   • Quality Filtering: Tasks with insufficient complexity (e.g., single-file changes, trivial line counts) were removed. Crucially, test suites were manually reviewed to eliminate overly narrow tests (enforcing specific implementation details) and overly broad tests (checking behavior outside the task scope).
   • Problem Statement Rewriting: Original commit messages were rewritten from scratch to provide precise, unambiguous specifications. Each description was verified to be a necessary and sufficient condition for the gold patch.
   • Human Verification: A final review ensured alignment between the issue description, test suite, and gold patch.

The final benchmark consists of 170 instances drawn from 70 distinct repositories across seven programming languages.

Key Contributions

1. High-Quality, Expert-Curated Benchmark: Unlike automated benchmarks, SWE-Bench ProMax undergoes multi-stage expert curation. Issue descriptions are rewritten to be precise specifications, and test suites are manually vetted to remove defects that plagued prior benchmarks.
2. Unprecedented Scale and Complexity: The benchmark targets large-scale refactoring, with instances averaging 11.4 modified files and 261.6 lines of code per task. This significantly exceeds existing benchmarks (e.g., 86% of SWE-bench Verified instances modify only a single file).
3. Multilingual Coverage: The dataset spans seven diverse languages (Python, Java, TypeScript, Go, C, C++, Rust), enabling analysis of how language-specific paradigms (e.g., ownership models in Rust, manual memory management in C) affect agent performance.
4. Rigorous Evaluation Protocol: The benchmark uses an outcome-driven evaluation where an instance is resolved only if the agent's modifications pass every test in the suite, without relying on intermediate steps or specific commands.

Results and Analysis

The authors evaluated six frontier models (three proprietary, three open-weight) under two agent scaffolds (mini-swe-agent and OpenHands).

• Unsaturated Challenge: The best-performing model, GPT-5.2, achieved a 41.2% resolve rate on SWE-Bench ProMax. This is substantially lower than the >75% resolve rates seen on SWE-bench Verified, confirming that large-scale, multi-file refactoring remains a significant unsolved challenge.
• Cost-Efficiency of Open-Weight Models: Open-weight models demonstrated competitiveness with proprietary models at a fraction of the cost. For instance, GLM-5 achieved a 36.5% resolve rate at $0.24 per instance, compared to Claude Sonnet 4.6 at 38.8% for $4.77.
• Language Variance: No single model dominated all languages. Performance varied significantly by language (e.g., Claude Sonnet 4.6 excelled in Rust and TypeScript, while GPT-5.2 performed best in Python and C), suggesting that model strengths are tied to specific language training data and architectures.
• Failure Mode Analysis: Trajectory analysis revealed two primary failure patterns:
  1. Incomplete Refactoring: Agents consistently modified fewer files than required by the gold patch. While they often identified core files, they failed to propagate changes to peripheral call sites, documentation, and configuration files, leading to test failures due to inconsistent downstream dependencies.
  2. Unproductive Exploration: Failed attempts consumed significantly more interaction rounds than successful ones, often entering repetitive edit-revert cycles without expanding the scope of modifications.

Significance

The paper claims that SWE-Bench ProMax provides a meaningful, unsaturated benchmark for evaluating AI coding agents on realistic, large-scale software engineering tasks. By addressing the quality defects of prior benchmarks (flawed tests, ambiguous descriptions) and scaling up the complexity to match production-level refactoring, it offers a more rigorous test of an agent's ability to maintain coherent plans across multiple files. The results suggest that while frontier models are improving, the capacity for sustained cross-file coordination remains a fundamental bottleneck. Furthermore, the benchmark highlights that open-weight models can approach frontier performance at a significantly lower cost, challenging the assumption that higher API costs guarantee proportionally better results in complex refactoring scenarios.