💬 NLP

Asuka-Bench: Benchmarking Code Agents on Underspecified User Intent and Multi-Round Refinement

The paper introduces Asuka-Bench, a novel benchmark designed to evaluate code agents on web development tasks by simulating real-world multi-round refinement cycles where agents iteratively improve underspecified projects based on automated UI testing and natural language feedback, revealing significant performance gaps among current models.

Original authors: Xin Wang, Liangtai Sun, Yaoming Zhu, Shuang Zhou, Jiaxing Liu, Fengjiao Chen, Lin Qiu, Xuezhi Cao, Xunliang Cai, Licheng Zhang, Zhendong Mao

Published 2026-06-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xin Wang, Liangtai Sun, Yaoming Zhu, Shuang Zhou, Jiaxing Liu, Fengjiao Chen, Lin Qiu, Xuezhi Cao, Xunliang Cai, Licheng Zhang, Zhendong Mao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a brilliant but slightly literal-minded architect to build a house.

The Old Way (Existing Benchmarks)
In the past, testing these architects was like giving them a perfect, 50-page blueprint that listed every single nail, wire, and paint color. You asked, "Build this," and they handed you a finished house. If the house matched the blueprint, they got an A. If it didn't, they got an F.

The problem is that real life doesn't work like that. Real clients rarely have a perfect 50-page blueprint. They usually say, "I want a house with a kitchen and a place to sleep," and then, once they see the first draft, they realize, "Oh, I actually wanted the kitchen to be bigger," or "Wait, the door opens the wrong way."

The New Way (Asuka-Bench)
The paper introduces Asuka-Bench, a new way to test "Code Agents" (AI programs that write software). Instead of giving the AI a perfect blueprint, the researchers give it a vague, messy request, like: "Make a shopping website with a list of products and a cart."

Then, they don't just grade the first result. They set up a three-person team to act out a real-world development cycle:

The Builder (Code Agent): This is the AI trying to build the website based on the vague request.
The Inspector (UI Agent): This is a robot that actually visits the website in a web browser. It doesn't read the code; it acts like a human user. It clicks buttons, tries to buy things, and checks if the pages load. It's like a quality control inspector walking through the house to see if the doors open.
The Client (User LLM): This is another AI that watches the Inspector. If the Inspector finds a problem (e.g., "The 'Buy' button doesn't work"), the Client translates that into a polite note for the Builder: "Hey, the button is broken. Please fix it."

The Builder then fixes the website, and the cycle repeats. This happens for up to three rounds.

The "DAG" Analogy
The researchers also invented a smart way to give feedback called a DAG (Directed Acyclic Graph). Think of this like a recipe.

If you are trying to bake a cake, you can't frost it before you bake it.
In the old testing methods, if the cake was burnt, the inspector might also complain that the frosting is missing, even though you couldn't frost a burnt cake.
In Asuka-Bench, the system knows the order. If the "baking" step fails, it stops the inspector from checking the "frosting" step. It only tells the Builder, "You didn't bake the cake." This prevents the Builder from getting confused by complaints about things that haven't happened yet.

What They Found
The researchers tested 8 different AI models using this method. Here is what they discovered:

Some AIs are better at fixing than others: Just because an AI is good at building the first draft doesn't mean it's good at fixing mistakes. Some models built a great first version but couldn't understand the "Client's" notes to fix the errors. Others started messy but got better with every round of feedback.
The gap is huge: The best models could complete about 52% of the projects perfectly after three rounds of fixing. The worst models only completed about 8%. That's a massive difference.
It's still hard: Even the smartest AI couldn't finish every single project perfectly. This shows that while AI is getting good, it still struggles with the messy, back-and-forth nature of real human requests.

In Summary
Asuka-Bench is a new "driving test" for AI coders. Instead of asking them to drive a car on a perfectly straight, empty track (a perfect blueprint), they are asked to drive in city traffic, get directions from a passenger when they take a wrong turn, and correct their course. It turns out that being able to listen and fix mistakes is a completely different skill than just knowing how to drive straight.

Technical Summary: Asuka-Bench

Problem Statement

Existing code-generation benchmarks predominantly rely on a "single-turn, one-shot" paradigm where an agent receives a complete, unambiguous specification and is evaluated on its ability to produce a correct output in a single pass. This approach fundamentally mismatches real-world software development, particularly in web development. In practice, users rarely articulate full specifications upfront; requirements often emerge and clarify only after inspecting intermediate results and providing corrective feedback. Current benchmarks fail to evaluate an agent's capacity to interpret underspecified intents, incorporate execution feedback, and iteratively refine its work toward a correct implementation. Furthermore, many existing web benchmarks focus on visual fidelity or assume fully specified inputs, neglecting the functional correctness and robustness derived from user interaction loops.

Methodology

The authors introduce Asuka-Bench, a benchmark designed to evaluate code agents under underspecified user intent through multi-round iterative refinement, grounded in browser-observable behavior.

Evaluation Framework

The benchmark operates via a closed-loop mechanism involving three distinct agents:

Code Agent: Receives a deliberately underspecified user query (e.g., "Create a shopping website...") and generates an initial web project. It iteratively refines this project based on feedback.
UI Agent: Deploys the generated project in a browser environment and executes predefined evaluation criteria as autonomous, interactive test cases. Unlike static code analysis, the UI Agent judges outcomes based on rendered behavior (e.g., clicking elements, verifying data updates) rather than source code inspection.
User LLM: Synthesizes the pass/fail results from the UI Agent into structured natural-language feedback, which is returned to the Code Agent for the next refinement round.

Dataset Construction

Sources: The dataset comprises 50 web development tasks derived from three sources: de-identified real user requests from a production service, formalized queries from open-source GitHub repositories, and summaries of established feature-rich websites.
Task Formalization: Each task is paired with a Clarified Product Requirements Document (PRD) that serves as the ground truth. From this PRD, the benchmark derives:
- An underspecified query (the input to the Code Agent).
- Simulated data (mock APIs, user profiles) to enable self-contained frontend execution.
- Evaluation Criteria: A structured set of natural-language test cases organized into a Directed Acyclic Graph (DAG).
Criteria Types: Tasks are categorized into Presence (element existence), Functionality (interactive behavior), and Robustness (edge-case handling).
DAG-Aware Protocol: Evaluation tasks are executed in topological order. A task is only evaluated if its prerequisite dependencies have passed. This prevents "cascading failures" where a root cause blocks downstream evaluation, ensuring feedback focuses on root causes. A "soft-satisfaction" threshold ( $T=0.5$ ) is used, where a parent task is considered satisfied if at least 50% of its sub-criteria pass, minimizing evaluation noise.

Experimental Setup

The authors evaluated 8 state-of-the-art Large Language Models (LLMs) across two agent frameworks: OpenHands (sandbox-based) and Claude Code (CLI-based). The maximum number of refinement rounds was set to three. Metrics included Project Completion Rate (PCR), cumulative weighted Task Pass Rate, and cumulative weighted Criteria Pass Rate.

Key Contributions

Asuka-Bench Benchmark: The first web generation benchmark to systematically evaluate code agents under underspecified requests with multi-round interactive refinement, addressing the gap between static instruction benchmarks and iterative real-world development.
User-Feedback-Centered Evaluation Protocol:
- Interaction-as-Feedback: Evaluation is grounded in browser-rendered behavior rather than source code.
- DAG-Aware Iterative Evaluation: By running criteria in topological order and stopping early on failed prerequisites, the protocol yields an average 10.5 percentage point (pp) higher task fix rate while reducing evaluation token costs by 23–26%.
Comprehensive Empirical Analysis: The study provides insights into how intent ambiguity and iterative feedback affect performance, demonstrating that the ability to repair from feedback is a distinct capability axis from initial generation quality.

Results

Model Differentiation: Asuka-Bench effectively separates model performance. The weighted Task Pass Rate varies by 38 percentage points across models after three rounds.
Iterative Improvement: While the strongest models achieve high Task Pass Rates (~~90%), the Project Completion Rate (requiring all tasks to pass) remains lower (~~52%), highlighting the difficulty of achieving full correctness.
Repair vs. Generation: The ability to repair errors from feedback is largely decoupled from first-round generation quality. For instance, Kimi-K2.6 started with a mediocre first round but improved significantly in subsequent rounds, outperforming models with stronger initial generations in the final cumulative score.
Saturation: The benchmark is far from saturated; even the strongest model (Claude-4.6-Sonnet) completes only 52% of projects after three rounds. However, extending the strongest model to 8 rounds resulted in 100% completion, confirming the tasks are solvable and the feedback mechanism is effective.
Human Alignment: The automated evaluation agent achieved 93.35% accuracy and 96.52% F1 score against human expert judgments, validating the reliability of the closed-loop framework.

Significance

The paper claims that Asuka-Bench reveals a critical blind spot in existing benchmarks: the inability to assess an agent's capacity to converge under a feedback-driven loop. By shifting from static, one-shot evaluation to iterative refinement based on underspecified intents, the benchmark provides a more realistic measure of an agent's utility in real-world deployment. The results suggest that "repair-from-feedback" is a separable dimension of model capability that current one-shot benchmarks fail to capture. The authors position Asuka-Bench as a step toward benchmarks that reflect the iterative, dialogue-driven nature of actual software engineering, encouraging the community to move beyond static evaluation paradigms.