💻 computer science

MT-Web2Code: Benchmarking Coding Agents on Multi-Turn Regional Reconstruction and Localized Modification

This paper introduces MT-Web2Code, the first multimodal benchmark designed to evaluate coding agents on realistic multi-turn web UI tasks involving regional reconstruction and localized modifications, revealing significant limitations in current agents' ability to maintain fidelity and prevent error accumulation across iterative turns.

Original authors: Qiming Li, Shujie Hu, Haohan Liu, Xiaocheng Feng, Songxiang Liu, Guanglu Wan

Published 2026-08-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qiming Li, Shujie Hu, Haohan Liu, Xiaocheng Feng, Songxiang Liu, Guanglu Wan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to be a master web designer. In the past, we tested these robots by handing them a blank canvas and a picture of a website, asking them to build the whole thing from scratch in one go. It was like asking a chef to cook a full five-course meal instantly. But in the real world, web design is rarely that dramatic. Usually, a developer opens an existing website, finds a broken button, fixes a weird font, or adds a missing photo, all while making sure they don't accidentally delete the rest of the page. It's more like a game of "spot the difference" and "fix the glitch" played over and over again. This is the world of "iterative coding," where the robot has to remember what it did five minutes ago and not mess it up while fixing the new problem. The big question researchers are asking is: Can our smartest AI robots handle this messy, step-by-step reality, or do they only work well when given a clean slate?

This is exactly what the paper "MT-Web2Code" investigates. The authors created a brand-new, super-challenging test called MT-Web2Code to see how good AI coding agents really are at this "fix-it-as-you-go" style of work. Instead of asking the AI to build a whole page, they gave it 102 different web pages from 16 different categories (like shopping, news, and government sites) and asked it to perform two specific types of tasks over multiple turns. The first task is "Regional Reconstruction," which is like the robot being told, "Hey, this whole section of the page is missing; here's a picture of what it should look like, please rebuild just that part without touching the rest." The second is "Localized Modification," which is even trickier: "Here are three tiny things that look wrong (maybe a button is the wrong color or a line is too thick); please fix only those tiny spots."

To make this test fair and impossible to bypass, the researchers invented a clever "Reverse-Corruption" engine. Imagine they took a perfect, golden website and then deliberately broke it in specific ways—hiding a whole section or messing up a few fonts. Then, they recorded exactly how they broke it. The AI's job is to reverse that process: look at the broken page, figure out what's wrong, and fix it back to the perfect state. Because the researchers know exactly how the page was broken, they can measure the AI's success with extreme precision. They found that while these AI robots are getting better, they still struggle mightily with this multi-turn game.

The results were a bit of a reality check. When the AI tried to rebuild a missing region, it often did a decent job on the new part but accidentally messed up the surrounding content, like changing the background color of the whole page when it was only supposed to fix a menu. When asked to make tiny, precise edits, the robots often failed to align the code perfectly with the visual fix, showing they lack fine-grained control. Perhaps most worryingly, the paper found an "error snowball" effect. If the AI made a small mistake in the first turn, that mistake got carried over and amplified in the next turns, making the final result much worse than if it had started fresh each time. Interestingly, the researchers also discovered that giving the AI a written description of what the broken part looked like didn't always help; sometimes, it actually confused the robot, making it rely too much on the words and ignore the actual picture.

In short, this paper suggests that while AI is great at building websites from scratch, it is still learning how to be a careful, iterative editor. It shows that being good at one doesn't automatically mean you're good at the other. The authors hope that by using their new, super-precise scoring system—which checks both the fixed part and the untouched parts—they can help train future AI to become more reliable, step-by-step web designers who don't break the rest of the house while fixing a single window.

Technical Summary: MT-Web2Code

Problem Statement

Recent advances in Large Vision-Language Models (LVLMs) have demonstrated significant capabilities in generating web UIs from scratch. However, existing benchmarks predominantly focus on single-turn, full-page generation, treating each task as an independent input-output problem. This formulation fails to capture the reality of frontend engineering, where developers engage in iterative workflows: reconstructing missing regions and modifying localized elements within existing codebases while preserving unaffected content. Current benchmarks cannot evaluate whether a model can reliably edit an evolving page, maintain consistency across turns, or prevent error propagation. There is a critical gap between static UI generation benchmarks and the dynamic, multi-turn requirements of real-world UI coding.

Methodology

1. Benchmark Construction: MT-Web2Code

The authors introduce MT-Web2Code, the first multimodal coding benchmark designed for multi-turn interactions. It encompasses 102 tasks spanning 16 vertical domains (e.g., CRM, ERP, Retail, Government). The benchmark defines two complementary task granularities:

Macro-Level Regional Reconstruction: The agent must reconstruct an entire missing semantic region from a visual reference and integrate it into the surrounding page structure without altering the rest of the page.
Micro-Level Localized Modification: The agent must repair fine-grained structural and stylistic defects (e.g., color, spacing, text alignment) within specific bounding boxes while strictly preserving the surrounding layout.

2. Data Generation: Reverse-Corruption Trajectory Engine

To construct high-quality, deterministic multi-turn repair trajectories without costly turn-level human annotation, the authors developed a Reverse-Corruption Trajectory Engine:

Process: Starting from a "Golden Page" ( $H_0$ ), the engine iteratively injects deterministic structural and stylistic defects to create a sequence of corrupted states ( $H_1, \dots, H_T$ ). The agent's task is to reverse this trajectory autoregressively.
Element Fingerprinting: To track elements across sequential modifications despite DOM reflows, the system assigns position-agnostic Element Fingerprints (hashes of tag, text, and attributes) to HTML nodes. These identifiers allow the system to deterministically track target regions and their nearest surviving siblings, ensuring the "ground truth" for each turn is uniquely defined.
Defect Taxonomy: Micro-level defects are generated using 9 atomic operators across 5 perceptual dimensions (Layout, Elements, Text, Color, Spacing), ensuring topological independence between targeted nodes to prevent cascading repair failures.

3. Evaluation Protocol: Dual-Axis Metrics

The paper proposes a dual-axis evaluation protocol that separately measures target-region fidelity (in-box) and preservation of unaffected content (out-of-box):

Macro-Level (Regional): Uses a 5-dimensional VLM-based rubric (Layout, Elements, Text, Color, Spacing) to score the reconstructed region perceptually. The surrounding page is evaluated for collateral reflow or restyling.
Micro-Level (Localized): Uses deterministic pixel-grounded alignment. The repair footprint is defined as the pixel-difference mask between the golden and corrupted pages. Accuracy is measured via Structural Similarity (SSIM) within the footprint, while preservation is measured by SSIM outside the footprint, complemented by a color-sensitive penalty.
Unified Scoring: The final score combines in-box and out-of-box metrics with a weighting of $0.8 \times S_{inbox} + 0.2 \times S_{outbox}$ .

Key Results

The authors evaluated 13 frontier coding agents (including Gemini, Claude, GPT, Kimi, GLM, Qwen, and Doubao series) on MT-Web2Code. The experiments reveal three consistent limitations:

Struggle with Faithful Regional Reconstruction: While agents show promise, they often fail to reconstruct target regions faithfully while maintaining consistency outside the target area. Larger models sometimes introduce unnecessary global layout changes even when explicitly instructed to preserve unaffected regions.
Lack of Fine-Grained Visual-Code Alignment: In micro-level tasks, agents generally achieve high scores for preserving surrounding content (often >0.95) but exhibit substantial variance in repairing the specific defects. This indicates a weakness in precise visual grounding and code alignment.
Error Snowballing: In multi-turn settings, errors occurring in early turns are inherited and amplified by subsequent edits. When models are forced to edit based on their own previous (potentially flawed) outputs, performance degrades significantly compared to scenarios where they edit from a correct golden state.

Notable Findings:

Task Specificity: Strong performance in macro-reconstruction does not guarantee strong performance in micro-modification (and vice versa). For instance, Gemini-3.5-Flash led in Macro-Level (65.5), while Doubao-Seed-2.0-Pro led in Micro-Level (83.5).
Caption Impact: Providing textual captions for visual references yielded inconsistent results; it improved some models (e.g., Gemini-3.5-Flash) but degraded others (e.g., Kimi-K2.6), suggesting that textual priors can sometimes conflict with visual evidence.
Judge Reliability: High-performing coding agents are not necessarily reliable judges; some top coding models assigned overly optimistic scores compared to human-aligned evaluations.

Significance and Claims

The paper claims that MT-Web2Code bridges the gap between static UI generation benchmarks and the iterative nature of real-world frontend development. Its primary contributions are:

First Multi-Turn Benchmark: It is the first benchmark dedicated to iterative web UI coding, covering both regional reconstruction and localized modification.
Scalable Data Construction: The Reverse-Corruption Trajectory Engine enables the creation of deterministic repair trajectories without expensive human annotation, providing reproducible ground truths.
Granular Evaluation: The dual-axis protocol provides fine-grained feedback signals that distinguish between successful repairs and collateral damage, a capability missing in existing global metrics.
Foundation for Training: Beyond benchmarking, the deterministic, pixel-grounded, and rubric-based metrics offer a natural foundation for verifiable reward signals in Reinforcement Learning (RL), potentially facilitating the training of future iterative UI coding agents without relying on human preference data.

The authors conclude that current agents, despite strong visual understanding, lack the consistency and fine-grained alignment required for reliable long-horizon multi-turn UI coding, highlighting a clear direction for future research.