💬 NLP

Evaluating LLMs on Real-World Software Performance Optimization

This paper introduces SWE-Pro, a rigorous repository-level benchmark derived from 102 expert optimizations that reveals current Large Language Models significantly fail to match human-level performance in real-world software optimization, achieving negligible gains compared to the substantial speedups and memory reductions delivered by expert engineers.

Original authors: Ezgi Sarıkayak, Wenchao Gu, Hesham Ghonim, Chunyang Chen

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ezgi Sarıkayak, Wenchao Gu, Hesham Ghonim, Chunyang Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Speedy Chef" vs. The "Master Chef"

Imagine you have a massive, busy restaurant kitchen (a real-world software codebase). You hire a team of AI Chefs (Large Language Models or LLMs) to help the Master Chefs (human experts) cook faster and use less energy.

The Master Chefs know exactly how to chop vegetables efficiently, organize the pantry to save steps, and adjust the stove temperature to save fuel. They have years of experience.

The AI Chefs are incredibly smart. They can read the recipe, understand the ingredients, and even write new recipes that look perfect on paper. But the big question this paper asks is: Can these AI Chefs actually make the kitchen run faster and use less energy in the real world, or do they just make things look good without actually improving the speed?

The Problem: The Old Tests Were Too Simple

Previously, researchers tested AI Chefs by giving them a single, tiny task: "Chop this one carrot."

The Flaw: In the real kitchen, you don't just chop one carrot. You chop 10, 1,000, or 100,000. Sometimes the carrots are wet; sometimes they are frozen.
The Result: The old tests were like judging a chef on how well they chop a single carrot in a quiet room. They didn't test if the chef could handle a rush hour with 1,000 orders, or if the chef accidentally used too much water (memory) while trying to be fast.

The paper argues that these old tests were too easy and didn't catch the "noise" (random fluctuations) that happens in real computers.

The Solution: SWE-Pro (The "Real Kitchen" Simulator)

The authors built a new, much harder test called SWE-Pro. Think of it as a high-tech simulation of a chaotic, real-world kitchen.

Real Recipes: They didn't make up fake tasks. They looked at 102 real examples where human experts had successfully optimized code in famous open-source projects (like pandas, scikit-learn, and xarray). These are the "Gold Standard" recipes.
The Stress Test: Instead of testing the AI on one input, SWE-Pro tests them on many different scenarios.
- Analogy: It's not just "chop 10 carrots." It's "chop 10 carrots," then "chop 100," then "chop 10,000," and do it with wet carrots, frozen carrots, and carrots of different sizes.
The Noise Filter: Computers are messy. Sometimes a program runs slow just because the computer was thinking about something else (like a waiter dropping a tray). SWE-Pro runs the tests over and over again, using a special statistical "noise filter" to make sure that if the AI says "I'm faster," it's actually true and not just a lucky fluke.
Two Metrics: They measure two things:
- Speed: How fast the food is served (Runtime).
- Memory: How much counter space and storage the chef uses (Peak Memory and Time-Weighted Memory Usage).

The Results: The AI Chefs Struggle

When they put the top AI models (like GPT-5.2, Claude Sonnet 4.6, etc.) through this rigorous "Real Kitchen" test, the results were surprising and disappointing.

The Human Experts (Gold Standard): When humans optimized the code, they achieved massive improvements.
- Analogy: The Master Chef cut the cooking time by 15 times and reduced the counter space needed by 171 times. They found clever ways to rearrange the kitchen that saved huge amounts of resources.
The AI Chefs:
- Speed: The AI models barely made any difference. In most cases, their "improvements" were so small they couldn't be distinguished from random noise.
- Memory: The AI models almost never managed to save memory. They were nearly non-existent in this area.
- The "No Signal" Problem: Even when the AI wrote code that passed the basic tests (the food tasted right), it rarely produced a measurable speedup. It was like a chef who chopped the carrots neatly but took the exact same amount of time as before.
- The Regression: Sometimes, the AI actually made things slower. One model (GPT-5.2) actually made the code 30% slower on average!

The Key Takeaway

The paper concludes that while AI is great at writing code that looks correct and follows the rules, it is currently very bad at the deep, complex engineering required to make software run significantly faster or use less memory in real-world scenarios.

The Gap: There is a huge gap between what AI can do today and what expert human engineers can do.
The Bottleneck: The problem isn't that the AI can't make a big improvement if it gets lucky (it can, in rare cases). The problem is that the AI cannot reliably find the opportunities to make those improvements in the first place.

In short: If you ask an AI to "make this software faster," it might write a nice-looking patch, but don't expect it to actually speed up your computer or save your memory. For now, that job still belongs to the human experts.

Technical Summary: SWE-Pro – Evaluating LLMs on Real-World Software Performance Optimization

1. Problem Statement

Software performance optimization is a complex, manual engineering task characterized by multidimensional trade-offs (e.g., runtime vs. memory) and high sensitivity to execution environments and input data. While Large Language Models (LLMs) have demonstrated proficiency in code generation and bug fixing, their ability to perform repository-level performance optimization remains under-evaluated.

Existing benchmarks suffer from critical limitations:

Oversimplification: Many focus on isolated functions (e.g., EffiBench) or single fixed workloads, failing to capture the complexity of real-world repositories.
Metric Deficiencies: Current frameworks often evaluate only runtime, neglecting memory usage and the critical time-memory trade-offs inherent in optimization.
Measurement Noise: Benchmarks frequently rely on fixed-count profiling or coarse-grained aggregation, which obscures input-specific performance fluctuations and fails to distinguish genuine gains from environmental noise (e.g., scheduler jitter, cache effects).
Lack of Statistical Rigor: There is a deficit in benchmarks that employ adaptive measurement and statistical convergence to ensure observed improvements are reproducible and significant.

Consequently, it remains unclear whether current LLMs can navigate the complexities of real-world software performance to deliver robust, expert-level optimizations.

2. Methodology: The SWE-Pro Benchmark

To address these gaps, the authors introduce SWE-Pro, a repository-level benchmark derived from 102 expert-written optimizations across three major open-source Python data libraries: pandas, scikit-learn, and xarray.

2.1 Dataset Construction

Source: Merged Pull Requests (PRs) explicitly labeled as performance-related, with developer acknowledgment in changelogs or documentation.
Validation: Manual inspection ensures tasks are reproducible, involve Python source modifications, and represent genuine optimization opportunities.
Gold Standard: Each task includes the original codebase, the expert-authored patch (Gold Solution), and a set of parameterized performance tests.

2.2 Parameterized Performance Testing

Unlike benchmarks using fixed inputs, SWE-Pro employs structured parameter sweeps to evaluate performance across diverse workloads.

Dimensions: Tests vary input size, data distribution (sparsity, duplication), data type, and function-level configurations.
Workload Generation: Parameters are combined via Cartesian products to generate execution configurations (median 6 per task, up to 24), ensuring the evaluation captures how optimizations generalize or degrade under different conditions.

2.3 Reliable Measurement Framework

SWE-Pro implements a hierarchical, noise-aware measurement pipeline to ensure statistical significance:

Adaptive Iterations: Instead of fixed counts, the system dynamically determines the number of measurement iterations based on the convergence of the 90% Relative Confidence Interval Width (RCIW).
Hierarchical Noise Modeling: The framework accounts for both within-container noise (scheduler jitter, interpreter behavior) and across-container noise (CPU state, memory layout) by executing workloads in fresh Docker instances.
Signal-to-Noise Ratio (SNR): Only performance effects exceeding a noise-aware threshold (SNR ≥ 2) are accepted as valid improvements.
Multi-Metric Evaluation: The benchmark tracks three distinct metrics:
- Runtime: Execution time.
- Peak Memory: Maximum memory footprint above baseline.
- Time-Weighted Memory Usage (TWMU): A normalized metric integrating memory pressure over time, calculated as $\bar{m}_{twmu} = \frac{1}{T} \int_0^T \max(0, m(t) - m_{base}) dt$ .

2.4 Experimental Setup

Models: Six state-of-the-art LLMs were evaluated (GPT-5.2, Claude Sonnet 4.6, Kimi K2.5, Gemini 3.1 Flash-Lite, GLM-5.1, MiniMax M2.7).
Context Strategies: Models were tested under two retrieval conditions:
- Oracle: Ground-truth relevant files provided directly.
- BM25: Context retrieved via lexical similarity from the repository state.
Evaluation Metrics: Patch success (applicability), Functional Correctness (test pass rate), and Performance Improvement Factor (IF) calculated via harmonic mean across workloads.

3. Key Results

The evaluation reveals a substantial gap between current LLM capabilities and expert-level engineering.

3.1 Patch Generation and Correctness

Applicability: Under Oracle retrieval, patch application rates ranged from 30.4% to 97.1%.
Correctness: Even when patches applied successfully, functional correctness rates were significantly lower (18.6% to 79.4%), indicating LLMs frequently produce syntactically valid but semantically incorrect code.
Retrieval Impact: Performance dropped further under BM25 retrieval, highlighting the sensitivity of optimization tasks to context quality.

3.2 Performance Optimization Capabilities

Runtime: LLMs struggle to produce measurable runtime gains.
- Only a small fraction of tasks passed the SNR filter (e.g., Gemini 3.1 Flash achieved 12.7% pass rate under Oracle).
- The magnitude of improvement was negligible (e.g., IF of 1.27× for the best model) compared to the Gold Solution's 15.48× speedup.
- Some models (e.g., GPT-5.2) produced patches that degraded performance (IF < 1).
Memory Optimization: LLMs demonstrated nearly non-existent capabilities in memory optimization.
- Peak memory and TWMU improvements were largely absent across all models.
- The only notable exception was Claude Sonnet 4.6 on a single instance (PR #7578), which achieved high IFs (16.61× peak memory, 28.82× TWMU under Oracle; 3556× TWMU under BM25). However, these gains did not generalize across tasks.
- In contrast, Gold Solutions achieved a 171.31× peak memory reduction and 619.22× TWMU reduction across 55.9%–70.6% of tasks.

3.3 Workload Sensitivity

The study highlights that optimization effectiveness is highly input-dependent. For example, in PR #7578, runtime improvements scaled significantly with input size (up to 100× for large workloads), while memory improvements remained stable. This underscores the necessity of parameterized testing, as single-input benchmarks would miss these critical behaviors.

4. Significance and Contributions

The paper claims the following contributions:

SWE-Pro Benchmark: A comprehensive, repository-level benchmark comprising 102 expert-validated tasks with parameterized tests, designed to shift evaluation toward robust, workload-aware analysis.
Noise-Aware Framework: A novel measurement pipeline utilizing adaptive sampling, hierarchical noise modeling, and SNR filtering to distinguish genuine optimization from environmental variance.
Multi-Metric Insight: The first systematic study to jointly evaluate runtime, peak memory, and TWMU, revealing that current LLMs fail to navigate time-memory trade-offs effectively.
Empirical Evidence of Limitations: The findings expose a "substantial gap" where current LLMs, despite excelling at local logic refinements, fail to deliver measurable efficiency gains in real-world scenarios. The primary bottleneck is identified not as the magnitude of potential improvements, but the reliable identification of optimization opportunities themselves.

The authors conclude that while LLMs show promise in specific, isolated cases, they currently lack the systematic reasoning required for expert-level performance engineering, necessitating future research into models capable of navigating complex, multi-metric optimization landscapes. The SWE-Pro dataset and evaluation pipeline are open-sourced to support further community research.