Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering
The paper argues that current coding benchmarks are misaligned with agentic software engineering because they conflate model performance with system harness components, penalize valid alternative solutions by relying on single reference answers, and lack the granular feedback necessary for iterative system improvement.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The Big Idea: We Are Grading the Wrong Thing
Imagine you are trying to judge how good a chef is at cooking a complex meal.
Currently, the way we test coding "agents" (AI systems that write software) is like this: You give the chef a single, pre-written recipe (the "reference solution"). The chef tries to cook the dish. If the final plate looks exactly like the picture in the recipe book, the chef gets a perfect score. If the chef cooks a delicious, healthy, and creative version of the dish that tastes better but looks slightly different, they get a low score.
The authors of this paper argue that this is a broken system. They say we aren't just testing the chef (the AI model); we are testing the entire kitchen setup (the "system harness"), which includes the stove, the tools, the ingredients, and the instructions.
The Core Problem: The "Kitchen" vs. The "Chef"
The paper makes a crucial distinction:
- The Model (The Chef): This is the AI brain that generates the code.
- The System Harness (The Kitchen): This is the complex environment where the AI lives. It includes the tools it uses, the context it reads, the rules it follows, and the feedback loops that tell it if it made a mistake.
The Analogy:
Think of a coding agent like a Formula 1 race car.
- The Model is the engine.
- The System Harness is the chassis, the tires, the aerodynamics, the pit crew, and the driver's strategy.
Current benchmarks are like a race where you only look at the final time and say, "This engine is fast!" But the paper points out that if you change the tires or the driver's strategy (the harness), the car can go 20% faster or slower, even with the exact same engine.
The authors show that in real-world testing, changing the "kitchen setup" (the harness) changes the results just as much as upgrading the "chef" (the AI model) to a newer version. Yet, our current tests treat the result as if it's only about the chef's talent.
Three Symptoms of the Broken System
The paper identifies three specific ways our current testing methods are misaligned with reality:
1. Blurring the Lines (Conflation)
The Analogy: Imagine a student takes a math test. They get a score of 80%. We assume the student is smart. But what if the student had a calculator, a cheat sheet, and a tutor whispering answers? If we don't report how they got the 80%, we can't tell if the student is actually smart or if the tools did the work.
The Paper's Claim: Current benchmarks report a single score (e.g., "Model X is 65% accurate"). They don't tell you what tools or environment were used. This makes it impossible to know if the AI is actually getting smarter, or if the "kitchen" just got better.
2. The "One Right Answer" Trap (Single Reference)
The Analogy: Imagine you ask a carpenter to build a table. You have a photo of a specific table you want.
- Scenario A: The carpenter builds a table that is sturdy, beautiful, and functional, but uses a slightly different wood grain than your photo.
- Scenario B: The carpenter builds a table that looks exactly like your photo but is wobbly and falls apart.
Current benchmarks would give Scenario B a perfect score and Scenario A a failing score because it didn't match the photo exactly.
The Paper's Claim: Real software engineering isn't about copying a single solution. It's about solving a problem in the best way possible. By forcing AI to match a single "gold standard" code snippet, we punish creative, valid, and often better solutions. We are testing if the AI can mimic a specific patch, not if it can solve the problem.
3. The Black Box (No Component Signal)
The Analogy: Imagine your car breaks down. You take it to a mechanic who says, "The car is broken." That's an "end-to-end" score. It tells you something is wrong, but it doesn't tell you what. Is it the battery? The tires? The engine?
The Paper's Claim: When a coding agent fails a test, current benchmarks just say "Failed." They don't tell you why. Did the AI misunderstand the instructions? Did the tools fail? Did the environment crash? Without knowing which part of the "kitchen" failed, developers can't fix the system. They are left guessing.
What Should We Do Instead?
The authors propose three changes to fix this:
- Report the Full Recipe: When publishing test results, we must list every tool, environment, and setting used. We need to know if the score came from a genius AI or a super-powered kitchen.
- Grade on Behavior, Not Appearance: Instead of checking if the code looks like the "gold standard," we should check if the code works and follows the rules (like safety checks or design patterns). There should be many ways to solve a problem, and the test should accept any valid solution.
- Test the Parts, Not Just the Whole: We need to break the system down. Test the AI's ability to read instructions separately from its ability to use tools. This helps us fix the specific broken part rather than guessing.
The Bottom Line
The paper argues that we are trying to measure the future of software engineering (complex, autonomous AI systems) with tools designed for the past (simple, one-shot code generation).
To move forward, we need to stop treating the AI model as the only thing that matters. We need to start measuring the entire system—the tools, the rules, and the feedback loops—because in the real world, that is what actually gets the job done. Until we do that, our rankings and scores will be misleading.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.