What's in a Benchmark? The Case of SWE-Bench in Automated Program Repair

Technical Summary: What's in a Benchmark? The Case of SWE-Bench in Automated Program Repair

Problem Statement
The field of Automated Program Repair (APR) has rapidly evolved with the integration of Large Language Models (LLMs) and agent-based systems. While benchmarks like Defects4J have historically served as common grounds for evaluation, SWE-Bench has emerged as a dominant benchmark designed to reflect real-world scenarios by mining issues from popular open-source Python repositories without pre-existing test cases for validation. Despite its rapid adoption and the existence of public leaderboards (SWE-Bench Lite and SWE-Bench Verified), there has been a lack of systematic analysis regarding the ecosystem surrounding these benchmarks. Specifically, the dynamics of who is submitting solutions, the nature of the products and models employed, and the openness of these approaches remain under-investigated. This paper addresses the need to understand the composition of the SWE-Bench ecosystem to guide future research toward greater transparency and diversity.

Methodology
The authors conducted the first comprehensive study of the SWE-Bench Lite and SWE-Bench Verified leaderboards. The study analyzed a total of 212 entries: 79 from the Lite leaderboard and 133 from the Verified leaderboard.

• Data Collection: The researchers manually inspected each leaderboard entry, retrieving metadata from the submission pull requests, associated websites, and repository files (e.g., README.md, metadata.yaml). They supplemented this with targeted Google searches (using the entry name + "SWE-Bench") and examination of LinkedIn profiles to identify submitter affiliations and product details.
• Exclusion Criteria: The "Full" and "Multimodal" leaderboards were excluded because solutions in "Full" are already represented in the other leaderboards, and the study focused strictly on language-based agents.
• Annotation: Using a combination of deductive and inductive coding, the authors categorized data across five dimensions corresponding to their Research Questions (RQs):
  1. Evolution: Tracking submission counts and precision (% Resolved) over time.
  2. Submitter Profile: Classifying submitters as Academia, Industry (further segmented by size and public status), Academia-Industry collaborations, Open-Source Community, Single Developers, or Unknown.
  3. Products & Availability: Categorizing the software artifacts (e.g., IDE plugins, cloud platforms, CLI tools) and their availability modes (Publicly Available Product, Upon Request, B2B, Non-Commercial Solution, Unavailable).
  4. Openness: Distinguishing between open-source and closed-source solution code.
  5. LLMs Employed: Identifying the specific LLMs used, including combinations and whether they are proprietary or open-source.

Key Contributions and Results

• RQ1 (Evolution): Submissions have grown steadily, with activity shifting significantly toward the Verified leaderboard since its official release in August 2024. While Lite submissions have slowed in 2025, Verified continues to grow. The Verified leaderboard demonstrates higher precision, with a median of 46.6% and a maximum of 76.8%, compared to Lite's median of 32% and maximum of 60.3%. This difference is attributed to Verified's construction, which filters out instances deemed too hard or unsolvable.
• RQ2 (Submitter Profile): The majority of submissions originate from industry (58% of total, rising to 79% when including collaborations). Small companies and large publicly traded corporations (e.g., Amazon, IBM, Google, EPAM) are the most active contributors. While academic contributions remain competitive, they constitute a smaller share of the total entries, particularly in Verified. Individual developers and open-source communities are present but less dominant in the Verified leaderboard.
• RQ3 (Products & Availability): Entries are associated with diverse product forms, including Coding Assistants (often IDE plugins), Development Platforms (cloud-based), and Issue Resolution tools. Publicly Available Products (PAP) and Non-Commercial Solutions (NCS) are the most common availability modes. Notably, entries associated with PAPs and "Upon Request" solutions tend to achieve higher precision, correlating with higher industry involvement.
• RQ4 (Openness): There is a distinct divide in openness. Academia drives the open-source ecosystem; when academic institutions are involved (solely or in collaboration), solutions are predominantly open-source. In contrast, industry submissions are largely closed-source. While closed-source solutions generally achieve higher median precision, several open-source approaches have reached state-of-the-art performance, particularly in 2025.
• RQ5 (LLMs): Proprietary LLMs dominate the top results. The Claude family (specifically Claude 4 Sonnet) is the most prevalent model for achieving state-of-the-art results on both leaderboards. The top result on Verified (76.8%) was achieved using Claude 4 Sonnet. While open-source models (e.g., Qwen2.5, Llama) are used, often in combination with closed models or fine-tuned, they generally yield lower precision when used exclusively. Combinations of multiple LLMs are common among top performers.

Significance and Implications
The paper argues that the SWE-Bench ecosystem is currently characterized by a strong industry presence and a reliance on proprietary, high-cost LLMs to achieve top performance. The authors highlight several critical implications:

• For Researchers: There is a misalignment between the Software Engineering (SE) and AI communities regarding patch validation. SE researchers typically distinguish between correct and overfitting patches, whereas many AI-focused submissions report only test-passing results. The paper calls for greater dissemination of overfitting risks and encourages the production of open-source artifacts.
• For Practitioners: Leaderboard scores should be interpreted with caution due to risks of patch overfitting and data contamination (memorization of training data). Performance on SWE-Bench may not generalize to diverse, real-world scenarios.
• For Benchmark Builders: There is a trade-off between transparency and participation. While detailed metadata improves understanding, strict requirements may discourage industry participation. The authors suggest balancing lightweight metadata requirements with the need for traceability. They also call for mechanisms to track benchmark versions, handle emerging subsets (e.g., SWE-bench-Lite-S), and mitigate data contamination through evolving infrastructure like SWE-bench-Live.
• Cost and Accessibility: The reliance on expensive, cloud-based APIs for top-tier performance creates a barrier to entry for academic researchers and smaller entities. The paper suggests that future evaluations should consider cost-performance trade-offs and encourage the reporting of results from open-source models to provide diverse baselines.

In conclusion, the study provides a foundational understanding of the current state of APR research as reflected in SWE-Bench, revealing a landscape driven by industry and proprietary models, while identifying specific areas where the community must evolve to ensure robustness, transparency, and inclusivity in future benchmark-driven research.