Technical Summary: SWE-MERA

Problem Statement

The rapid advancement of Large Language Models (LLMs) in software engineering has exposed critical limitations in existing evaluation benchmarks, most notably SWE-bench. The authors identify two primary failure modes in static benchmarks:

1. Data Contamination: A significant portion of successful patches in SWE-bench involve direct solution leakage (32.67%) or pass due to inadequate test cases (31.08%). As models are repeatedly tested on fixed datasets, they inadvertently memorize solutions or overfit to outdated examples.
2. Benchmark Saturation: Static datasets lose effectiveness over time as state-of-the-art models achieve near-perfect scores, making it difficult to distinguish meaningful progress.

Furthermore, existing dynamic benchmarks like LiveCodeBench primarily target algorithmic problems and fail to capture the repository-level complexity essential for realistic software engineering assessment.

Methodology

SWE-MERA introduces a dynamic, continuously updated benchmark designed to mitigate contamination and ensure relevance. The core of the methodology is a seven-stage automated pipeline that collects, validates, and curates real-world GitHub issues and pull requests (PRs).

The Collection Pipeline

The pipeline executes on a monthly (planned quarterly) basis to ensure the dataset reflects the latest software engineering challenges. The process involves:

1. Repository Selection: Filters GitHub repositories based on criteria including a minimum of 10 stars/forks, recent activity (current year), Python as the primary language, and an open-source license.
2. PR-Issue Mapping Construction: Establishes strict one-to-one mappings between closed issues and merged PRs. The PR merge date must be later than the first day of the previous month to ensure temporal freshness.
3. Metadata Extraction and Filtering: Downloads and parses issue/PR metadata. Pairs are discarded if the combined title and body length is less than 25 characters.
4. Patch Extraction and Validation: Generates git diffs. Only PRs modifying both source code and test files are retained. PRs modifying more than 15 source files are excluded to manage complexity.
5. Repository Build Validation: Tasks are executed in a Docker container. A task is considered valid only if pytest returns at least one passed test after reverting the repository to the pre-fix state and applying the future test cases.
6. End-to-End Task Execution: Verifies reproducibility and correctness in a controlled environment.
7. LLM-based Pipeline Evaluation: Uses the Qwen3-32B model to evaluate candidate tasks on four criteria: task correctness, test correctness, test completeness, and complexity. Tasks falling in the bottom 25% of scores for correctness or completeness are filtered out. Complexity is not used for filtering to retain a mix of difficulty levels.

Evaluation Framework

To assess model performance, the authors employ the Aider coding agent framework.

• Mechanism: Aider provides LLMs with up to six independent attempts ("tries") to fix an issue. Each attempt allows up to four reflections to lint or test output.
• Metrics: The evaluation reports pass@1 (success on the first attempt) and pass@6 (success within six attempts).
• Infrastructure: Experiments were conducted on NVIDIA H100 GPUs, with token usage ranging from 60–140M prompt tokens and 3–20M completion tokens per model evaluation.

Key Contributions

1. Dynamic Benchmarking Pipeline: A robust, seven-stage pipeline capable of collecting approximately 10,000 potential tasks, with 728 samples currently available for the 2025 subset. This pipeline minimizes contamination risks through rigorous automated validation and temporal filtering.
2. Automated Scoring and Leaderboard: Implementation of an automated scoring system based on the Aider agent and a dynamic user leaderboard that updates monthly.
3. Open Infrastructure: The entire pipeline is released as a Python package, and the dataset is available via Hugging Face, facilitating reproducibility and extensibility for future research.
4. Comprehensive Evaluation: A detailed performance analysis of over a dozen recent LLMs (including Qwen3, DeepSeek-R1, and Devstral) on tasks collected between September 2024 and June 2025.

Results

The evaluation of state-of-the-art models on the SWE-MERA 2025 subset yielded the following insights:

• Discriminative Power: SWE-MERA effectively ranks models according to their capabilities. For instance, it correctly orders the Qwen2.5-Coder series by parameter size (7B < 14B < 32B).
• Model Performance:
  • DeepSeek-R1-0528 achieved the highest pass@6 (40.2%) and pass@1 (27.8%).
  • Devstral-Small-2505 demonstrated superior performance relative to its size, outperforming larger models like Qwen2.5-Coder-32B.
  • Qwen3-32B slightly outperformed QwQ-32B, consistent with its specifications.
• Temporal Anomalies: An interesting observation noted that DeepSeek-R1 variants performed better on 2024 tasks compared to 2025 tasks, a behavior not observed in other investigated models.
• Failure Modes: The study tracked specific failure points, such as "localize files" (identifying correct files), "generate patch" (creating valid patches), and "regression tests" (passing original repository tests). Token limit hits were minimal (0.0%–5.7%).

Significance and Claims

The paper posits that SWE-MERA addresses the fundamental challenges of data contamination and benchmark saturation that plague static evaluations. By introducing dynamic updates and rigorous quality validation, the benchmark ensures:

1. Real-world Relevance: Tasks reflect the latest challenges in software development.
2. Fair Evaluation: Models are tested on fresh problems, minimizing the risk of memorization.
3. Continuous Improvement: The benchmark evolves alongside AI and software engineering practices.

The authors claim that SWE-MERA establishes reliable performance baselines free from the contamination issues found in traditional static benchmarks. They acknowledge limitations, including the potential for dynamically collected tasks to lack the nuance of human-curated problems, challenges in grading open-ended solutions, and the fact that the current framework focuses primarily on programming correctness rather than code readability, maintainability, or security.

The framework is designed to be extensible to other languages (Java, JavaScript, TypeScript, Go, C++) and serves as a community-driven resource to advance AI research in software engineering.