💻 computer science

SpecBench: Evaluating Specification-Level Reasoning for Software Engineering LLM Agents

This paper introduces SpecBench, a new benchmark that evaluates the ability of software engineering agents to identify flaws and improve incomplete or ambiguous system specifications through expert-level reasoning, addressing a critical gap left by existing code-generation-focused benchmarks.

Original authors: Grant Hamblin, Kevin Song, Zhanda Zhu, Anand Jayarajan, Sihang Liu, Nandita Vijaykumar, Gennady Pekhimenko

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Grant Hamblin, Kevin Song, Zhanda Zhu, Anand Jayarajan, Sihang Liu, Nandita Vijaykumar, Gennady Pekhimenko

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: From "Building" to "Blueprinting"

Imagine you are hiring a robot to build a house.

Old Benchmarks (like SWE-Bench): These tests give the robot a perfect, detailed blueprint and ask, "Can you build the wall exactly as drawn?" The robot just needs to lay the bricks. If the blueprint says "red brick," the robot lays a red brick.
The Real World Problem: In real life, the blueprint you start with is often messy. It might say "put a door here" without specifying if it's a front or back door, or it might forget to mention that the foundation needs to be deeper because of the soil type. If the robot just starts building based on that messy blueprint, the house might collapse later.
The New Benchmark (SpecBench): This paper introduces a test that doesn't ask the robot to build the house. Instead, it asks the robot to read the messy blueprint and point out the mistakes before construction begins. It tests the robot's ability to say, "Hey, this plan is missing a door," or "This wall will hit a tree," or "You didn't say what kind of wood to use."

What is SpecBench?

SpecBench is a new test designed to evaluate how well AI agents (smart computer programs) can reason about software specifications.

In software engineering, before anyone writes code, they write a "specification" (a plan). In big projects like Linux, Kubernetes, or React, these plans go through a process called an RFC (Request for Comments). This is like a town hall meeting where experts argue, critique, and refine the plan until it's perfect.

SpecBench simulates this town hall meeting. It gives an AI:

The initial, messy plan (the RFC).
The history of how the project has worked in the past.
The current code of the project.

The AI's job is to act like a senior engineer and find the flaws in the plan. It needs to find things that are:

Missing: "You forgot to say what happens if the internet cuts out."
Confusing: "You said 'fast,' but did you mean 1 second or 1 minute?"
Contradictory: "You said this feature is safe, but it breaks that other rule."
Wrong: "This idea conflicts with how we've always done things."

How Did They Build the Test?

The researchers looked at five real-world software giants: Kubernetes, React, Rust, TVM, and vLLM.

They took real historical documents where people proposed new features. They then looked at the actual discussions where human experts tore those proposals apart and found the holes. These "holes" became the Golden Set (the correct answers).

The Challenge of "Human Variance":
Sometimes, one expert cares about speed, while another cares about security. To handle this, the researchers used a panel of AI judges to vote on which criticisms were the most important. They separated the flaws into two groups:

Core Flaws: The big, obvious mistakes that almost everyone agrees on (like a missing foundation).
Extended Flaws: Smaller, more nuanced issues that some experts might catch and others might miss.

The "Open World" Problem:
In a coding test, if the robot writes the wrong code, it fails. But in a planning test, the robot might find a new flaw that the original humans missed. The researchers decided: "If the robot finds a flaw that isn't in our answer key, we can't say it's wrong, but we can't give it credit either." So, they gave the robot a limited number of guesses (a budget) and only scored it on how many of its guesses matched the known "Golden" flaws.

How Did the AI Do?

The researchers tested the smartest AI agents available (like GPT-5.4, Claude, and Codex).

The Score: The best AI got about 44.4% accuracy.
What this means: Even the smartest AI is still missing more than half of the critical flaws in complex software plans. They are getting better at writing code, but they are still not very good at planning code.
The Gap: The AI was much better at finding the "Core" flaws (the big, obvious ones) than the "Extended" flaws (the subtle, tricky ones).

Why Does This Matter?

Currently, we have AI that is great at following instructions (Implementation). We don't yet have AI that is great at designing the instructions (Specification).

This paper shows that while AI is getting good at being a "bricklayer," it is still struggling to be the "architect." If we want AI to run entire software projects, it needs to learn how to spot the holes in the plan before the first line of code is written.

In short: SpecBench is a report card showing that our AI architects are still learning how to read the blueprints before they start building.

Technical Summary: SpecBench

Problem Statement

Current software engineering (SWE) benchmarks, such as SWE-Bench, primarily evaluate implementation-level reasoning: the ability of an agent to generate code that passes tests given a fixed, precise, and assumed-correct specification. This formulation overlooks a critical phase in real-world software development: specification design. In complex systems, initial design proposals are often incomplete, ambiguous, or flawed, requiring extensive expert review and revision before implementation can proceed. Existing benchmarks do not measure an agent's ability to identify these deficiencies or reason about system design without execution feedback.

Methodology

The authors introduce SpecBench, a benchmark designed to evaluate specification-level reasoning. The core capability measured is the ability to generate complete, unambiguous, consistent, and correct system specifications by identifying deficiencies in initial design proposals.

Dataset Construction

Source: Tasks are derived from the Request for Comments (RFC) processes of five diverse open-source repositories: Kubernetes, React, Rust, TVM, and vLLM.
Input: For each task, an agent receives:
1. The initial RFC design proposal.
2. The project codebase (at the commit prior to the RFC proposal).
3. The history of prior RFC discussions and decisions.
- Agents are restricted from accessing internet resources or Git history after the proposal date.
Task: The agent must identify specification deficiencies (omissions, ambiguities, inconsistencies, or incorrect assumptions) in the initial proposal.
Deficiency Classification: Deficiencies are categorized based on IEEE Std. 1028-1997 and prior work into four classes:
- Omission: Missing necessary information.
- Ambiguous: Information with multiple interpretations.
- Inconsistent: Contradictions within the proposal or with the existing system.
- Incorrect: Contradictions with preceding documents or information.

Benchmark Construction & Evaluation

Golden Set: The ground truth consists of deficiencies raised by original contributors during historical RFC discussions.
Handling Expert Variance: To address noise from differing expert priorities, the authors use an LLM-based expert panel to rate extracted deficiencies on a 5-point Likert scale. Deficiencies are labeled "core" (mean score ≥ 3.0 and ≥ 2/3 expert endorsement) or "extended." Core items receive double the weight in scoring.
Open-World Validation: Unlike deterministic test suites, the space of valid specification gaps is open-ended. The authors adopt Information Retrieval (IR) methodology:
- Predictions matching the golden set are verifiably valid.
- Predictions not in the golden set are treated as unjudged (not necessarily incorrect) rather than false positives.
- A bounded prediction budget is imposed ( $N = \lceil 1.25 \times |G| \rceil$ ) to prevent agents from flooding the evaluation with low-quality guesses.
Judging Mechanism: To handle semantic matching between free-text predictions and golden sets, a two-stage process is used:
1. Structured Decomposition: Deficiencies are decomposed into Subject (system component), Predicate (nature of deficiency), and Impact (consequence).
2. Ensemble Judging: Two independent LLM judges (GPT-5.4 and Sonnet 4.6) perform four trials per task. A match is accepted if it appears in at least 3 of 4 trials.

Key Contributions

SpecBench Benchmark: The first benchmark specifically targeting specification-level reasoning, decoupled from code implementation.
Novel Evaluation Protocol: A methodology for evaluating open-ended specification critiques using a bounded prediction budget, weighted scoring based on expert consensus, and structured semantic matching (SPI triples) to handle linguistic variance.
Empirical Baselines: Evaluation of state-of-the-art SWE agents (including Codex, Claude, and Gemini variants) on this new task, revealing significant gaps in current capabilities.

Results

Overall Performance: The best-performing agent, GPT-5.4 (referred to as Codex-5.4 in the results section), achieved an accuracy of 44.4%. All evaluated agents remained below 45%, indicating substantial room for improvement.
Core vs. Extended: Agents performed significantly better on core deficiencies (high expert consensus) than on extended ones, aligning with the construction of the dataset.
Repository Variance: Performance varied across repositories, with the best agent showing the largest lead in React and vLLM tasks.
Judge Reliability: Inter-judge agreement was measured using pairwise Jaccard similarity, with plans to increase judge diversity in future versions.

Significance and Claims

The paper claims that SpecBench addresses a critical gap in SWE agent evaluation by shifting focus from executing clear requirements to designing them. The authors argue that:

Real-world software engineering requires agents to understand community values (e.g., Rust's memory safety vs. React's API stability) and historical design decisions, not just code generation.
Current benchmarks assume "perfect specifications," which does not reflect the reality of complex systems where initial proposals are often flawed.
SpecBench isolates specification-level reasoning capabilities, distinct from the implementation-level reasoning emphasized in prior work like SWE-Bench.

The authors note that this is an early version of the benchmark focusing on identifying deficiencies. Future work will extend SpecBench to include revision tasks (correcting the specifications) and incorporate human expert validation to establish human performance baselines and measure human-LLM agreement.