💻 computer science

Rethinking Code Performance Benchmarks for LLMs

This paper reveals that existing LLM code performance benchmarks are largely insufficient due to inadequate test suites, and proposes a novel multi-agent framework that generates more rigorous, performance-oriented tests to effectively expose significant runtime improvements in LLM-generated code.

Original authors: Nhat Minh Le (Peter), Yisen Xu (Peter), Zhijie Wang (Peter), Tse-Hsun (Peter), Chen

Published 2026-07-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nhat Minh Le (Peter), Yisen Xu (Peter), Zhijie Wang (Peter), Tse-Hsun (Peter), Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge at a cooking competition. The goal isn't just to see if the chefs can make a dish that tastes good (functional correctness); you also want to know if they can make it faster than the recipe book's standard version (performance efficiency).

This paper is like a group of food critics who decided to re-examine the rules of this cooking competition. They looked at four popular "cookbooks" (benchmarks) used to test AI chefs (Large Language Models) and found some serious flaws in how the competition was being scored.

Here is the breakdown of their findings using simple analogies:

1. The Problem: The Stopwatch Was Broken (and the Race Was Too Short)

The authors found that the current competitions were using two bad methods to measure speed:

The "One-and-Done" Stopwatch: Most competitions timed the chefs' dishes only once. In the real world, if you run a race once, you might trip on a pebble, or the wind might be in your favor. You need to run the race many times (they ran it 30 times) to get a true average.
The "Toy" Race Track: The test cases (the inputs) were like running a race on a tiny, 10-meter track. On such a short track, a professional sprinter and a casual walker might finish in the exact same time. The test cases were too small to reveal the true speed difference between a slow algorithm and a fast one.

The Result: When the authors re-ran the tests with a proper stopwatch and a longer track, they discovered that 94% of the time, the "faster" recipes provided by the competition organizers weren't actually faster at all. They were just as slow as the standard ones. This meant the competition couldn't tell if an AI was actually writing efficient code or just writing code that looked different.

2. Why Were the Tests Failing?

The authors looked closely at the "faster" recipes and found two main reasons they failed the speed test:

The "Cosmetic" Change: Some recipes just changed the font or rearranged the ingredients list (refactoring). They looked different on paper, but the cooking time was identical.
The "Hidden" Speed: Some recipes did use a better technique (like switching from a slow spoon to a high-speed blender). However, because the test track was so short, the blender didn't have enough time to show its advantage. The test cases were too weak to expose the real speed difference.

3. The Solution: The "Super-Tester" AI

To fix this, the authors built a new tool: a Multi-Agent AI Framework. Think of this as a team of three expert inspectors working together:

The Generator: Creates new, tougher test cases (longer race tracks, heavier loads).
The Diagnostician: If a test fails, this agent figures out why (e.g., "The test asked for a cake but the oven was off").
The Repairer: Fixes the test so it works correctly but still pushes the code to its limits.

This team generated new, tougher tests that forced the code to run under heavy pressure.

4. The New Results

When they used these new, tougher tests:

For the "Faster" Recipes: Suddenly, 24% to 25% of the "faster" recipes that previously looked identical to the slow ones were revealed to be genuinely faster. The new tests finally exposed the hidden speed.
For the AI Chefs: When they tested actual AI-generated code with these new, tough tests, they found that the AI was actually writing efficient code in about 22% of the cases. Under the old, weak tests, these successes were invisible.

The Bottom Line

The paper concludes that we have been judging AI chefs with a broken stopwatch and a toy race track. We thought the AI wasn't very good at writing fast code, but that was mostly because the tests weren't good enough to see the speed.

To truly know if AI can write efficient code, we need to:

Run the tests many times (to avoid bad luck).
Use much larger, more challenging inputs (to force the code to show its true speed).
Stop relying on single-run results.

Until we fix the tests, we can't be sure if the AI is slow or if we just haven't given it a real challenge yet.

Technical Summary: Rethinking Code Performance Benchmarks for LLMs

Problem Statement

Existing function-level performance benchmarks for Large Language Models (LLMs), such as EffiBench, Enamel, EvalPerf, and Mercury, aim to evaluate whether LLMs can generate computationally efficient code. However, empirical observations suggest that LLM-generated implementations often show little to no execution-time difference compared to canonical (baseline) solutions. This paper posits that such results may not solely reflect the limitations of LLMs but rather fundamental flaws in the benchmark evaluation settings. Specifically, the authors identify two primary challenges:

Runtime Measurement Noise: Many benchmarks rely on single-run or few-run executions without statistical significance testing, making it difficult to distinguish genuine performance gains from system noise (e.g., OS scheduling, CPU caching).
Insufficient Test Inputs: Derived largely from correctness-oriented benchmarks (e.g., HumanEval, MBPP), existing test suites often use small inputs that fail to stress performance-critical paths. Consequently, algorithmic improvements (e.g., $O(n^3)$ to $O(n^2)$ ) remain invisible because the input sizes do not trigger the computational load necessary to expose the difference.

The core research question is whether current benchmarks can reliably distinguish genuinely efficient implementations from merely functionally correct ones, and whether benchmark-provided "performant" implementations are actually distinguishable from canonical solutions under rigorous testing.

Methodology

The authors conducted a comprehensive re-evaluation of 1,538 tasks across four popular Python function-level benchmarks (EffiBench, Enamel, EvalPerf, and Mercury). The study is structured around four research questions (RQs):

1. Rigorous Re-evaluation (RQ1)

To address measurement noise, the authors executed both the canonical solution and the benchmark-provided performant implementation 30 times per task under controlled conditions. They applied the Mann-Whitney U test (a non-parametric test) to determine statistical significance ( $p < 0.05$ ) and calculated Cliff's delta to measure effect size. This approach contrasts with the single-run or low-repetition methods used by the original benchmarks.

2. Analysis of Code Changes (RQ2)

For the tasks where no significant performance difference was found, the authors analyzed the nature of the code changes between canonical and performant implementations.

Manual Analysis: A subset of 308 non-significant tasks was manually analyzed by two authors using open coding, achieving a Cohen's Kappa of 0.93.
LLM-as-a-Judge: To scale this analysis to the remaining 1,136 tasks, the authors employed DeepSeek-V3.1 and GPT-4o as judges. The models were prompted to categorize code changes into "no meaningful performance change" (e.g., refactoring, built-in substitution) or "potential performance impact" (e.g., algorithm strategy changes, data structure replacements).

3. Multi-Agent Test Generation (RQ3)

To address the issue of insufficient test inputs, the authors developed a multi-agent framework consisting of three agents:

Test Generation Agent: Generates deterministic, performance-oriented test inputs designed to stress critical paths.
Test Diagnosis Agent: Analyzes failure logs to identify root causes (error type, line number) without generating full code diffs.
Test Repair Agent: Fixes failed tests based on the diagnostic output.
This framework iteratively generates, diagnoses, and repairs test suites (up to 3 refinement iterations per batch) to produce validated, executable tests that preserve functional correctness while increasing computational load. The generated suites were compared against the state-of-the-art method COFFE.

4. Impact on LLM Evaluation (RQ4)

The authors re-evaluated code generated by three modern LLMs (GPT-4o-mini, Claude-Sonnet-4.5, and Gemini-2.5-Flash) using both the original benchmark test suites and the newly generated performance-oriented suites. They measured the shift in the proportion of tasks showing statistically significant performance improvements.

Key Results

RQ1: Reliability of Existing Benchmarks

Under the rigorous 30-run statistical testing protocol, 93.89% (1,444 out of 1,538) of the benchmark-provided performant implementations showed no statistically significant performance difference compared to their canonical solutions. Only 6.11% of tasks exhibited significant improvements, with most of these showing large effect sizes. This suggests that the original benchmarks fail to expose intended optimizations.

RQ2: Nature of Non-Significant Changes

Manual and LLM-as-a-Judge analyses revealed that the lack of significant differences stems from two factors:

No Meaningful Change: Approximately 21.47% of tasks involved changes that did not impact performance (e.g., refactoring, variable renaming, or identical code).
Hidden Potential: Approximately 72.62% of tasks involved substantive algorithmic or data-structure changes (e.g., replacing $O(n^3)$ with $O(n^2)$ , using hash maps instead of arrays) that should improve performance. However, the original test inputs were too small to make these differences observable.

RQ3: Effectiveness of Generated Tests

The multi-agent framework successfully generated test suites that exposed hidden performance differences:

On the 209 manually verified tasks with potential performance impact, the framework revealed significant improvements in 44.98% (DeepSeek-v3.1) and 48.80% (GPT-4o) of cases, vastly outperforming the SOTA method COFFE (which achieved only ~2%).
Across the full set of 1,345 previously non-significant tasks, the framework identified significant improvements in 24.01% and 25.43% of tasks, respectively.
All detected significant cases exhibited large effect sizes ( $\delta \ge 0.474$ ).

RQ4: Impact on LLM Evaluation

When evaluating LLM-generated code with the new, stronger test suites:

The proportion of tasks showing statistically significant performance improvements increased substantially.
For Claude-Sonnet-4.5, significant improvements were detected in 44.21% of tasks (vs. 9.75% with original tests).
For GPT-4o-mini and Gemini-2.5-Flash, the rates increased to 25.75% and 17.82%, respectively.
Overall, the enhanced test suites confirmed statistically distinguishable performance improvements in an average of 22.19% of evaluated cases that were previously indistinguishable.

Significance and Claims

The paper claims that current conclusions regarding the efficiency of LLM-generated code, drawn from existing function-level Python benchmarks, must be interpreted with caution. The primary contributions are:

Revealing Benchmark Limitations: The study demonstrates that the lack of performance gains in LLM outputs is often an artifact of insufficient test suites and non-rigorous measurement, rather than a fundamental inability of LLMs to optimize code.
Methodological Rigor: It establishes that reliable performance evaluation requires repeated execution (e.g., 30 runs) and statistical significance testing, moving beyond single-run comparisons.
Test Generation Framework: The proposed multi-agent framework effectively generates performance-oriented tests that stress critical paths, revealing optimizations that standard benchmarks miss.
Future Directions: The authors argue that future benchmarks must move beyond simple functional correctness tasks. They should explicitly design inputs to scale and expose algorithmic differences, incorporate statistical rigor, and eventually extend evaluation from isolated functions to class-level or repository-level settings where performance bottlenecks arise from component interactions.

The authors conclude that without addressing these limitations, benchmarks risk misrepresenting the true performance capabilities of LLMs, potentially leading to the dismissal of genuine optimizations or the false attribution of inefficiency to the models.