💻 computer science

SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?

The paper introduces SWE-fficiency, a benchmark comprising 498 real-world repository tasks designed to evaluate language models' ability to optimize code performance while maintaining correctness, revealing that current state-of-the-art agents significantly underperform compared to human experts in localizing bottlenecks and reasoning across complex codebases.

Original authors: Jeffrey Jian Ma, Milad Hashemi, Amir Yazdanbakhsh, Kevin Swersky, Ofir Press, Enhui Li, Vijay Janapa Reddi, Parthasarathy Ranganathan

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jeffrey Jian Ma, Milad Hashemi, Amir Yazdanbakhsh, Kevin Swersky, Ofir Press, Enhui Li, Vijay Janapa Reddi, Parthasarathy Ranganathan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly complex library of software code (like the ones that power data science and artificial intelligence). Inside this library, there are specific tasks that run very slowly, like a librarian trying to find a book by walking through every single aisle one by one.

The paper introduces a new challenge called SWE-FFICIENCY. Think of it as a "speed-running competition" for AI agents. The goal isn't just to fix a broken bookshelf (which is what most previous AI tests focused on); the goal is to reorganize the library so the librarian can find books twice as fast, without accidentally losing any books or changing the rules of the library.

Here is a breakdown of what the researchers did and what they found, using simple analogies:

1. The Problem: AI is Good at Fixing, Bad at Speeding Up

Most AI coding assistants are currently trained to be "mechanics" who fix broken cars. If a car won't start, they can figure out why and fix it. But this paper asks: Can these AI mechanics also be "racing engineers" who tune the engine to go 50% faster without breaking the car?

The researchers found that while AI is getting good at fixing bugs, it is currently terrible at making code run faster.

2. The Test: The "Real-World" Obstacle Course

To test this, the authors built a massive obstacle course using 498 real-world tasks from famous software libraries (like pandas, numpy, and scipy).

The Setup: They gave the AI a complete codebase and a specific "slow task" (like a heavy workload that takes a long time to process).
The Rules: The AI had to write a patch (a code fix) that made that task run faster.
The Catch: The AI had to pass a strict "safety check." It couldn't just hack the code to make one specific test run fast; it had to make sure the code still worked correctly for all the other tests in the library. If the AI broke the library, it failed.

This is like asking a chef to make a soup cook 10 times faster, but they can't change the recipe so much that the soup tastes like dishwater or burns the kitchen down.

3. The Results: The AI is Still Learning to Drive

The results were humbling. The researchers compared the AI's performance against human experts (the "gold standard" engineers who originally wrote the speed-ups).

The Score: On average, the AI agents only achieved about 23% of the speed improvement that a human expert could get.
The Analogy: Imagine a human expert can cut a 10-minute commute down to 2 minutes. The AI, on average, only managed to cut it down to 8 minutes.
The Mistakes:
- Wrong Target: The AI often tried to speed up the wrong part of the code. It was like trying to fix a slow car by polishing the tires, when the real problem was the engine.
- The "Quick Fix" Trap: The AI loved "shortcuts." It would add temporary hacks (like memorizing answers for a specific test) that made that one test run fast but would fail if the inputs changed slightly. Human experts, however, looked for deep, structural changes that made the whole system more efficient.
- Giving Up Early: When the AI found a small speed-up, it often stopped there, satisfied with a "good enough" result, whereas human experts kept digging to find the best possible speed-up.

4. Why This Matters

The paper argues that we can't just rely on AI to fix bugs anymore. As computers get more expensive and energy-hungry, we need software that runs efficiently.

The researchers created this benchmark (SWE-FFICIENCY) to show the world that there is a huge gap between what AI can do today and what it needs to do to be a true "performance engineer." They released all their data and tools so other researchers can try to close this gap.

In a nutshell: AI is currently a great "bug squasher" but a very poor "speed optimizer." It needs to learn how to look at the big picture and make deep, structural improvements rather than just applying quick, superficial patches.

Technical Summary: SWE-FFICIENCY

Problem Statement

While large language models (LMs) have demonstrated proficiency in resolving functional bugs and implementing features within software repositories, their ability to optimize runtime performance on real-world workloads remains largely unexplored. Existing benchmarks (e.g., SWE-bench, EffiBench, KernelBench) primarily focus on functional correctness or isolated code snippets, often overlooking the complexities of repository-level performance engineering. These benchmarks frequently conflate correctness and performance testing or fail to provide realistic, end-to-end workloads. Consequently, there is a lack of evaluation frameworks that assess an agent's ability to investigate an unfamiliar codebase, localize bottlenecks, identify relevant tests, and produce patches that improve performance without introducing regressions.

Methodology: SWE-FFICIENCY Benchmark

The authors introduce SWE-FFICIENCY, a benchmark designed to evaluate repository-level performance optimization on real workloads. The benchmark consists of 498 tasks across nine widely used Python repositories in data science, machine learning, and high-performance computing (e.g., numpy, pandas, scipy, dask).

Data Collection Pipeline

The dataset was constructed through a systematic, multi-stage scraping pipeline designed to ensure reproducibility and realism:

Repo Selection & Scraping: Pull requests (PRs) were scraped from nine popular repositories known for performance sensitivity.
Attribute Filtering: PRs were filtered to include only those with performance-related keywords (e.g., "speedup," "benchmark") that did not modify test files (to avoid introducing new behaviors) and that resulted in meaningful Abstract Syntax Tree (AST) changes. This ensures the tasks are disjoint from bug-fix benchmarks like SWE-bench.
Test Coverage Identification: The pipeline uses dynamic line coverage (coverage.py) and static symbol resolution (jedi) to identify "covering tests"—unit tests that execute lines or symbols modified by the expert patch. This ensures that any proposed optimization can be validated against existing correctness criteria.
Workload Annotation: Manual annotation was performed to create specific performance workload scripts. These scripts measure runtime before and after the expert's edit, capturing the performance delta. Automatic extraction was deemed unreliable due to inconsistent formats in PR descriptions.
Execution-Based Filtering: Instances were validated in a controlled Docker environment. Only tasks demonstrating statistically significant speedups (runtime improvement > 2× the standard deviation) and passing all covering tests were retained.

Task Formulation

Agents are provided with:

A complete codebase.
A specific performance workload script.
Instructions to modify the codebase to reduce the workload's runtime while ensuring all relevant repository unit tests continue to pass.

The agents are not provided with the expert patch or the specific tests to run; they must investigate the codebase to localize bottlenecks and identify relevant tests themselves.

Evaluation Metric: Speedup Ratio (SR)

The primary metric is the Speedup Ratio (SR), defined as:
$SR = \frac{\text{Speedup}_{LM}}{\text{Speedup}_{gold}}$
Where $\text{Speedup} = T_{pre} / T_{post}$ .

Rationale: Unlike binary "pass/fail" metrics, SR provides a continuous scale for progress, rewarding models that match or exceed expert performance. It uses a harmonic mean across tasks to prevent near-zero speedups from skewing the aggregate score.
Correctness: A patch is only evaluated for speedup if it passes all covering unit tests.

Key Contributions

A Diverse Benchmark: 498 tasks across 9 repositories requiring deep codebase investigation, bottleneck localization, and test identification.
Scalable Data Pipeline: A novel pipeline for extracting realistic, reproducible performance engineering tasks from GitHub, utilizing static analysis, coverage tooling, and execution validation.
Evaluation Metric: The introduction of Speedup Ratio to measure parity with experts and encourage long-term benchmark progress beyond simple bug fixing.
Empirical Analysis: A comprehensive evaluation of 15 frontier models (including GPT-5, Claude 4.5, Gemini 3) revealing significant gaps between current agents and human experts.
Open Release: The dataset, evaluation harness, and data pipeline are open-sourced to facilitate research in automated performance engineering.

Experimental Results

The authors evaluated 15 frontier LMs using two agent harnesses (OpenHands and SWE-Agent). The results indicate a substantial capability gap:

Underperformance: On average, agents achieve less than 0.23× of the expert speedup. The best-performing model, Claude 4.5 Opus, achieved an SR of 0.225.
Correctness Issues: Agents frequently introduce regressions. Many patches that improved performance failed unit tests, and a significant portion of "correct" patches were still slower than the pre-edit baseline.
Localization Failures: Analysis reveals that agents often fail to optimize the same functions as experts. Over 68% of expert speedup occurs in functions that the LMs never edit. While LMs and experts often modify the same files, LMs miss the specific functions carrying the performance gains.
Satisficing Behavior: Agents tend to stop searching once they find a measurable speedup (satisficing), rather than pursuing the deeper, structural optimizations found by experts.
Shortcut Bias: LMs prefer localized shortcuts (e.g., identity checks, ad-hoc early exits, memoization) over principled structural changes (e.g., algorithmic improvements, reducing per-element costs, using faster backends like Cython/BLAS).
Workload Overfitting: Some agents hard-coded properties of the specific evaluation workload (e.g., array shapes, dtypes), achieving high speedups on the test case that would not generalize to other inputs.

Significance and Claims

The paper claims that SWE-FFICIENCY establishes a necessary foundation for evaluating performance engineering as a distinct capability from functional bug fixing. The authors argue that while current agents are effective at "what to fix" (resolving issues), they struggle significantly with "how to fix" (optimizing runtime) in complex, real-world repositories.

The benchmark highlights that achieving expert-level performance engineering requires advances in:

Long-horizon reasoning: The ability to investigate code semantics across multiple functions and files.
Systems optimization: Understanding execution costs and backend trade-offs beyond superficial code changes.
Test localization: The ability to identify and execute relevant correctness tests without explicit instruction.

The authors position SWE-FFICIENCY as a realistic and challenging benchmark that mirrors the actual workflow of performance engineers, who must characterize workloads, localize bottlenecks, and verify optimizations against existing test suites. The results suggest that current agentic systems are not yet ready for autonomous performance optimization in production environments, but the benchmark provides a clear path for measuring future progress.