Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification

The paper introduces Vision2Web, a hierarchical benchmark comprising 193 real-world tasks and a novel dual-component agent verification system to systematically evaluate and reveal significant performance gaps in large language model-based agents across the spectrum of visual website development, from UI generation to full-stack deployment.

The Gist

Imagine you want to build a house. You have a beautiful architectural drawing (the visual prototype) and a list of requirements (the text instructions).

In the past, AI coding assistants were like apprentices who could build a single brick wall perfectly if you pointed at a photo of it. But if you asked them to build the whole house, connect the plumbing, install the electricity, and make sure the front door opens when you knock, they often got lost, forgot the blueprints, or built a wall that looked nothing like the drawing.

"Vision2Web" is a new, giant training ground and testing center designed to see if AI can actually build these "digital houses" (websites) from scratch, just by looking at pictures and reading instructions.

Here is a simple breakdown of how it works:

1. The Three Levels of Difficulty (The "Ladder")

The researchers didn't just throw the AI into the deep end. They created a ladder with three rungs, getting harder as you climb up:

• Level 1: The Static Snapshot (The "Photo Booth")
  • The Task: The AI sees a picture of a webpage on a phone, a tablet, and a computer. It must build a webpage that looks exactly like the picture on all three screens.
  • The Challenge: It's like a painter trying to copy a photo perfectly. If the AI gets the colors or the spacing wrong, it fails.
• Level 2: The Interactive Tour (The "Theme Park")
  • The Task: Now, the AI has to build a website with multiple pages. If you click a button, it should take you to a new page. If you click "Back," it should go back.
  • The Challenge: This is like building a theme park where the rides (pages) are connected by roads (navigation). The AI has to remember how the whole park fits together, not just one ride.
• Level 3: The Full-Stack Empire (The "Smart City")
  • The Task: The AI must build a complete system. It needs a front end (what you see), a back end (the brain that processes data), and a database (the memory). It has to handle logins, shopping carts, and saving user data.
  • The Challenge: This is like building a whole city with traffic lights, power grids, and police stations. If one part breaks, the whole city stops working.

2. The "Referee" System (How they grade the AI)

How do you know if the AI actually built a working website? You can't just ask the AI, "Did you do a good job?" because it might lie.

The researchers created a two-part referee team to grade the work:

• Referee A: The "Robot Butler" (GUI Agent Verifier)
  • This is a robot that acts like a human user. It follows a strict checklist: "Click the login button," "Type 'password'," "Check if the dashboard appears."
  • If the robot can't click the button or the page crashes, the AI fails this part. It checks if the functionality works.
• Referee B: The "Art Critic" (VLM Judge)
  • This is a super-smart AI that looks at the finished website and compares it side-by-side with the original drawing.
  • It asks: "Is the button the right color? Is the font the right size? Does it look like the blueprint?" It checks the visual beauty.

3. What Did They Find? (The Results)

The researchers tested the world's smartest AI models on this benchmark. Here is the verdict:

• The "Good News": The AIs are getting really good at Level 1. They can copy a single webpage picture very well.
• The "Bad News": As soon as you ask them to do Level 2 or Level 3, they start to stumble.
  • The "Amnesia" Problem: When building complex sites, the AI often forgets what it did five minutes ago. It might build a login page but forget to connect it to the database.
  • The "Small Screen" Struggle: The AIs are great at building for big computer screens but often mess up when trying to make the site look good on a tiny phone screen.
  • The "Complexity" Gap: The more complex the website (like a SaaS platform or a shopping site), the worse the AI performs. They struggle to keep all the moving parts working together.

4. Why Does This Matter?

Think of Vision2Web as a driver's license test for AI.

Before this, we only tested if AI could park a car (write a small snippet of code). Now, we are testing if they can drive across the country, navigate traffic, and get to the destination without crashing (build a full website).

The paper tells us that while AI is a brilliant apprentice, it is not yet a master architect. It needs help to plan the big picture and keep track of all the details when the project gets huge. This benchmark helps researchers figure out exactly where the AI is failing so they can teach it better.

In short: Vision2Web is the ultimate "show me, don't just tell me" test for AI, proving that while they can draw a pretty picture, building the whole house is still a work in progress.

Technical Summary

1. Problem Statement

While Large Language Models (LLMs) have significantly advanced autonomous coding agents, systematic evaluation of complex, end-to-end visual website development remains limited. Existing benchmarks suffer from three primary gaps:

• Limited Task Formulation: Current benchmarks (e.g., SWE-Bench) focus on incremental, issue-driven code edits rather than holistic, end-to-end software engineering.
• Misaligned Multimodal Coverage: Existing multimodal benchmarks are often restricted to static "Design-to-Code" tasks (single-page UI reproduction) and lack evaluation of interactive, multi-page, or full-stack scenarios.
• Insufficient Verification: Reliable assessment of complex interactions and long-horizon system outcomes is challenging due to underspecified task definitions and a lack of constrained, reproducible verification procedures.

2. Methodology: Vision2Web Benchmark

Vision2Web is a hierarchical benchmark designed to evaluate multimodal coding agents across the full software development lifecycle.

A. Hierarchical Task Design

The benchmark organizes 193 tasks into three progressively difficult levels to disentangle agent capabilities:

1. Level 1: Static Webpage: Agents must generate device-responsive static code (desktop, tablet, mobile) from prototype images, focusing on layout, visual content, and styling fidelity.
2. Level 2: Interactive Frontend: Agents generate multi-page, interactive frontends based on multiple prototypes and text descriptions, requiring reasoning across pages and coherent navigation flows.
3. Level 3: Full-Stack Website: Agents interpret structured requirement documents and prototypes to build cohesive full-stack systems, managing application states, backend logic, and integrated debugging.

B. Dataset Construction

• Source: Tasks are curated from the C4 validation set to ensure contamination-free, real-world data.
• Pipeline: A three-stage filtering process (Structural Assessment, Content Screening via VLM, and Manual Review) ensures high-quality, diverse, and well-defined tasks.
• Scale: The dataset comprises 193 tasks across 16 subcategories (4 major categories), containing 918 prototype images and 1,255 test cases.

C. Workflow-Based Agent Verification Paradigm

To address the unreliability of open-ended agent evaluation, Vision2Web introduces a structured verification framework:

• Workflow Abstraction: End-to-end testing is formalized as a directed dependency graph where nodes represent self-contained verification sub-procedures (functional or visual).
• Dual-Verifier System:
  1. GUI Agent Verifier (Functional Score): A GUI agent executes expert-designed test workflows to verify functional correctness (e.g., login, navigation, state management). It uses a 3-tuple definition (Objective, Guided Actions, Validation Criteria) to constrain agent behavior and ensure reproducibility.
  2. VLM-Based Judge (Visual Score): A Vision-Language Model acts as a judge to compare rendered pages against reference prototypes, assigning component-level fidelity scores based on visual rubrics.
• Scoring: Performance is measured by the Functional Score (FS) (proportion of passed functional nodes) and Visual Score (VS) (average fidelity across prototypes).

3. Key Contributions

1. Hierarchical Task Design: The first benchmark to systematically evaluate agents from static UI generation to full-stack development, enabling fine-grained failure attribution.
2. Realistic Multimodal Data: A large-scale dataset grounded in real-world websites with explicit specifications, covering diverse categories (Content, Transaction, SaaS, Public Service).
3. Workflow-Based Agent Verification: A reproducible, implementation-agnostic evaluation paradigm that combines structured workflows with autonomous agents to objectively measure both functional correctness and visual fidelity.

4. Experimental Results

The authors evaluated 8 state-of-the-art multimodal models (including Claude-Opus-4.5, GPT-5, Gemini-3-Pro, and Qwen3-VL) integrated into two frameworks (OpenHands and Claude Code).

Key Findings:

• Performance Degradation with Complexity: All agents show significant performance drops as task complexity increases. For example, Gemini-3-Pro-Preview achieved strong static webpage scores (~63.3 on desktop) but dropped sharply on full-stack tasks (Visual Score: 11.7, Functional Score: 22.6).
• Device and Visual Complexity: Performance degrades on smaller devices (tablet/mobile) and with denser, more complex prototype images.
• Model Leadership: Claude-Opus-4.5 consistently outperformed other models across all levels and frameworks, though it still struggled with full-stack development compared to static tasks.
• Framework Impact: The choice of coding agent framework (OpenHands vs. Claude Code) significantly influenced performance, suggesting a need for joint optimization of models and frameworks.
• Category-Specific Weaknesses: Agents performed best on "Public Service" sites (simple structures) and worst on "SaaS Platforms" (complex multi-page navigation).
• Specific Failure Modes:
  • Fine-Grained Visual Alignment: Misaligned layouts and asset handling issues.
  • Cross-Module Understanding: Loss of visual fidelity and broken links on subsequent pages.
  • System-Level Planning: Failure in long-horizon planning, state management, and autonomous verification, leading to non-functional full-stack systems.

5. Significance

Vision2Web provides a critical foundation for advancing autonomous software engineering by:

• Highlighting Limitations: Demonstrating that current SOTA models, while capable of isolated coding tasks, lack the robustness for holistic, end-to-end system construction.
• Standardizing Evaluation: Offering a rigorous, reproducible framework that moves beyond simple code generation metrics to assess functional and visual fidelity in realistic scenarios.
• Guiding Future Research: Identifying specific gaps in cross-modal reasoning, long-horizon planning, and state management, directing future research toward hierarchical task designs and principled autonomous evaluation.

The benchmark is constructed entirely from public resources, ensuring no ethical or privacy concerns, and is intended to drive the field toward more reliable and capable multimodal coding agents.