Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?
This paper audits three prominent repository-level performance-optimization benchmarks (GSO, SWE-Perf, and SWE-fficiency) and reveals that their leaderboard scores are unreliable due to significant reference patch fragility across different machines, scoring rules that distort rankings, and the fact that most tasks are already solvable by existing public submissions, thereby masking true performance gaps.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a judge at a cooking competition. The goal isn't just to make a dish that tastes good; it's to make it cook faster than the original recipe while keeping the flavor exactly the same.
In the world of computer science, "coding agents" (AI programs) are entering this competition. They are given real software projects and asked to write code patches that make the software run faster. To see who wins, researchers created "leaderboards" (like a scoreboard) based on three specific contests: GSO, SWE-Perf, and SWE-fficiency.
This paper is like a group of food critics who decided to audit the competition itself. They asked: "Are these scores actually telling us who is the best chef, or are the rules of the game misleading us?"
Here is what they found, explained simply:
1. The "Weather" Problem: The Recipe Changes When You Move Kitchens
The Issue: The competition assumes that if a code patch makes a program faster on one computer, it will be faster on any computer. It's like assuming a cake will bake in exactly 20 minutes whether you are in a kitchen in New York or Tokyo.
The Audit: The researchers took the "official winning recipes" (reference patches) and tried to bake them on four different types of computers (different "kitchens").
- The Result: In many cases, the recipe that was supposed to be the "fastest" actually became slower or just barely the same speed on a different computer.
- The Metaphor: Imagine a runner who is the fastest on a track in the morning but gets tired and slows down in the afternoon. The "official record" says they are the fastest, but that record only holds up under specific conditions.
- The Takeaway: For one of the contests (SWE-Perf), the "speed gains" were so tiny (like 0.03%) that the computer's internal noise (like a fan spinning or a background process) made the results flip-flop. The "winning" patch wasn't actually winning; it was just lucky with the weather.
2. The "Scorecard" Problem: One Bad Dish Can Ruin Your Whole Meal
The Issue: How do you calculate the final score?
- Contest A (GSO): It's a simple pass/fail. Did you beat the reference speed? Yes = 1 point. No = 0 points.
- Contest B (SWE-fficiency): This uses a complex math formula (harmonic mean) that punishes you heavily if you have one really bad result.
The Audit: The researchers looked at the same set of AI chefs and scored them using both rules.
- The Result: The rankings changed completely! An AI that was #1 in Contest A dropped to #7 in Contest B.
- The Metaphor: Imagine a student taking a test.
- Rule A: You get 1 point for every correct answer.
- Rule B: You get 1 point for every correct answer, BUT if you get one question wrong, your entire score is divided by 1,000.
- Under Rule B, a student who got 99% of questions right but missed one tiny detail could end up with a lower score than a student who got 80% right but never missed a "catastrophic" error.
- The Takeaway: The leaderboard doesn't just show who is the best coder; it shows who is best at avoiding the specific "traps" built into the scoring math. In SWE-fficiency, the worst 10 tasks (the "bad dishes") carried 58% to 82% of the weight for the entire score. One bad patch could ruin a submission's reputation.
3. The "Already Solved" Problem: The Finish Line is Moving
The Issue: The benchmarks are supposed to show us how much progress AI is making. But are the tasks still hard?
The Audit: The researchers looked at the top 10 public submissions for each task. They asked: "Has anyone already solved this?"
- The Result: For almost every task (99.8%), at least one AI had already written a patch that worked and was faster than the original code. For most tasks (85%), at least one AI had already matched or beaten the "official reference" speed.
- The Metaphor: Imagine a race where the goal is to run a mile in under 6 minutes. The researchers checked the results and found that almost everyone had already run it in under 6 minutes. The only people left in the race were trying to shave off the last few seconds to match the "world record" time.
- The Takeaway: The "hard part" of finding any solution is mostly over. The remaining challenge is just fine-tuning to match the specific speed of the reference patch. The benchmarks are no longer testing if AI can "fix" things; they are testing if AI can "perfect" things.
Summary: What Should We Believe?
The paper concludes that leaderboard scores alone are not enough to judge coding agents.
- Don't trust the "Official Record" blindly: A patch might look like a winner on one computer but fail on another.
- Check the Math: A ranking might look high just because the scoring rule penalized a competitor for one tiny mistake, not because they were generally worse.
- Look at the Details: Most tasks are already "solved" by someone. The real gap isn't about finding a solution; it's about getting the last few percentage points of speed.
In short, the scoreboard is real, but the game is more complex than the numbers suggest. To truly understand who is winning, you have to look under the hood at how the score was calculated and where the AI actually succeeded.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.