SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale

1. Problem Statement

Software Engineering (SWE) agents, particularly those driven by Reinforcement Learning (RL), are rapidly improving but face a critical bottleneck: the scarcity of large-scale, executable task collections with reproducible environments and reliable test suites.

• Limitations of Current Benchmarks: Existing datasets (e.g., SWE-bench) are often limited in scale, diversity, or language coverage (heavily skewed toward Python). They frequently rely on manual verification, which limits the volume of data available for training.
• The "Language-Agnostic" Gap: While some multilingual benchmarks exist, they often lack the robustness to handle "long-tail" languages with heterogeneous build systems, dependency managers, and test runners.
• Training vs. Evaluation: Most released resources are designed for evaluation (static benchmarks) rather than interactive training (which requires pre-built environments, fine-grained diagnostics, and massive scale).

2. Methodology: The SWE-rebench V2 Pipeline

The authors introduce a fully automated, language-agnostic pipeline designed to harvest real-world SWE tasks from GitHub and construct executable training environments at scale. The pipeline consists of five distinct stages:

A. Preliminary Data Collection

• Source: Aggregates data from GitHub Archive, cloning repositories in distributed map-reduce jobs to bypass API rate limits.
• Filtering: Links issues to Pull Requests (PRs) that reference them. Filters for permissive licenses, merged PRs, and the presence of test modifications.
• Strategy: Uses strict thresholds for high-resource languages (e.g., Python, Java) to optimize compute-to-yield ratios, while relaxing thresholds for long-tail languages to preserve diversity.

B. Setup Synthesis (Interactive Agent)

• Goal: Construct a Docker container with the correct runtime, dependencies, and test execution scripts for each repository.
• Mechanism: Deploys an interactive setup agent (based on mini-SWE-agent and Qwen3-Coder-480B) that operates in a closed-loop debugging cycle.
  • The agent inspects the codebase, attempts dependency installation, and iteratively refines scripts based on build errors and test failures.
  • It synthesizes install_config.json containing shell commands for installation and a verbose test_cmd.
• Innovation: Unlike non-interactive pipelines that fail on complex ecosystems, the interactive agent can handle heterogeneous build systems (e.g., C/C++ recompilation, JVM test ordering issues) by retrying and adapting.

C. Execution-Based Validation

• Process: Runs the full project test suite in two passes:
  1. Pre-fix: Validates the baseline environment.
  2. Post-fix: Applies the solution patch and re-runs the suite.
• Oracle: Only instances with at least one "fail-to-pass" (F2P) test are retained, ensuring a non-trivial learning signal.
• Output: Generates containerized images with pre-built dependencies to ensure reproducible execution without network-dependent installs at runtime.

D. Filtering by Issue Clarity

• Automated Quality Control: Uses an ensemble of three LLM judges (gpt-oss-120b, GLM-4.7, DeepSeek-V3.2) to score issue descriptions.
• Criteria: Tasks are filtered if the problem statement is underspecified, ambiguous, or lacks clear acceptance criteria. The system uses a "Verified-E" prompt configuration (incorporating the patch and test patch) to maximize precision.

E. Metadata Enrichment & PR-Based Expansion

• Diagnostic Metadata: Automatically tags tasks with failure modes (e.g., "Test Suite Coupling," "Implicit Naming," "External Dependencies") to allow researchers to filter or curate datasets for specific training goals (e.g., curriculum learning).
• PR-Derived Tasks: To scale beyond issue-linked data, the pipeline generates synthetic problem statements from PR descriptions and patches for PRs not linked to issues, creating a massive auxiliary dataset.

3. Key Contributions

1. Language-Agnostic Pipeline: A unified workflow that applies across 20 programming languages, relying on reusable language-specific templates (base images, parsers) rather than manual engineering for each language.
2. Large-Scale Datasets:
   • SWE-rebench V2 (Issue-Linked): 32,000+ executable tasks from 3,600+ repositories across 20 languages, complete with pre-built Docker images.
   • SWE-rebench V2-PRs (PR-Derived): 120,000+ tasks with installation instructions and metadata, generated from PR descriptions to expand the training corpus significantly.
3. Diagnostic Framework: Instance-level metadata that flags confounders (e.g., flaky tests, external dependencies), enabling controlled training and stratified evaluation.
4. Comprehensive Ablations: Rigorous experiments comparing interactive vs. non-interactive setup agents, different LLM models, and filtering strategies.

4. Results and Experiments

• Setup Synthesis Performance:
  • Interactivity is Critical: Interactive agents significantly outperform non-interactive pipelines. An interactive agent using Qwen3-30B achieved a 36.9% pass@5 success rate, compared to 14.9% for a non-interactive pipeline using the much larger Qwen3-480B.
  • Retry Budget: Increasing setup attempts from 1 to 10 nearly doubled the success rate for difficult repositories.
• Filtering Performance:
  • The ensemble of three models (gpt-oss-120b, GLM-4.7, DeepSeek-V3.2) using an averaging strategy provided the best balance of precision and recall for filtering underspecified issues.
• Model Performance Baselines:
  • Evaluated 7 frontier models on a subset of 300 tasks (5 languages).
  • Top Performer: Claude Opus-4.5 achieved the highest pass rates (e.g., ~36% on Python, ~29% on Rust).
  • Language Variance: Performance varied significantly by language, with Python and Go showing higher success rates than Scala or Rust, highlighting the difficulty of long-tail ecosystems.
• Diagnostic Insights:
  • Identified common failure modes: Test Suite Coupling (fixing one issue breaks unrelated tests), Implicit Naming (tests require specific variable names not in the prompt), and External Dependencies (links to inaccessible docs).

5. Significance

• Enabling RL at Scale: By providing pre-built, reproducible environments and massive task volumes, SWE-rebench V2 removes the infrastructure bottleneck that has hindered the training of SWE agents via Reinforcement Learning.
• Democratizing Multilingual Training: It shifts the focus from Python-centric benchmarks to a truly multilingual ecosystem, allowing agents to learn generalizable software engineering skills across diverse toolchains.
• Configurable Training Substrate: The inclusion of diagnostic metadata allows researchers to design curricula (e.g., starting with "clean" tasks and progressing to "noisy" real-world scenarios) and isolate specific agent capabilities.
• Open Science: The authors release the datasets, collection code, and execution artifacts, fostering reproducibility and further research in autonomous software engineering.

In summary, SWE-rebench V2 represents a paradigm shift from static, evaluation-only benchmarks to dynamic, large-scale, language-agnostic training ecosystems, providing the necessary infrastructure to train the next generation of robust, general-purpose software engineering agents.