💻 computer science

WebVR: Benchmarking Multimodal LLMs for WebPage Recreation from Videos via Human-Aligned Visual Rubrics

This paper introduces WebVR, a novel benchmark and human-aligned visual rubric designed to evaluate the capability of Multimodal LLMs to faithfully recreate webpages from demonstration videos, addressing the gap in video-conditioned web generation research through a synthetically curated dataset of 175 diverse pages.

Original authors: Yuhong Dai, Yanlin Lai, Mitt Huang, Hangyu Guo, Dingming Li, Hongbo Peng, Haodong Li, Yingxiu Zhao, Haoran Lyu, Zheng Ge, Xiangyu Zhang, Daxin Jiang

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuhong Dai, Yanlin Lai, Mitt Huang, Hangyu Guo, Dingming Li, Hongbo Peng, Haodong Li, Yingxiu Zhao, Haoran Lyu, Zheng Ge, Xiangyu Zhang, Daxin Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

🎬 The Big Idea: Teaching AI to "Watch and Do"

Imagine you are a master carpenter. Usually, if you want a client to build a specific chair, you might give them a blueprint (text) or a photo (static image). But what if you gave them a video of you building the chair?

In that video, they wouldn't just see the final shape; they would see how you move your hands, when you sand the wood, and how the chair swivels.

WebVR is a new test for Artificial Intelligence (specifically Multimodal LLMs) that asks: "Can you watch a video of a website being used, and then build an exact, working copy of it?"

Current AI is great at looking at a photo and drawing a picture. But this paper argues that to build a real website, the AI needs to understand the movement, the timing, and the "feel" that only a video can show.

🏗️ The Problem: The "Static Photo" Trap

Before WebVR, researchers tested AI by showing it a screenshot of a website and asking it to write the code.

The Flaw: A photo is like a frozen moment in time. It doesn't tell you what happens when you hover your mouse over a button, how a menu slides in, or how a page scrolls.
The Result: The AI builds a website that looks right in a picture but feels dead and broken when you try to use it. It's like building a car that looks great in a showroom but has no engine.

🧪 The Solution: The WebVR Benchmark

The researchers created WebVR, a giant "driving test" for AI. Here is how it works, broken down into four simple steps:

1. The "Fake" Factory (Data Synthesis)

To make sure the AI hasn't just memorized the answers from the internet, the researchers didn't just download real websites. Instead, they built a "factory."

They took real videos of websites.
They used AI to rewrite the "story" of the website (e.g., changing a "Nike Shoe Store" into a "Fictional Space Station Shop") while keeping the exact same layout and animations.
Analogy: It's like taking a recipe for a chocolate cake, changing the name to "Moon Rock Cake," but keeping the exact same baking steps. This ensures the AI has to learn the structure, not just copy the ingredients.

2. The "Magic Sandbox" (Execution)

Once the AI writes the code, they don't just read it. They put it in a standardized sandbox (a safe, controlled digital room).

The sandbox runs the code, clicks the buttons, scrolls the page, and records a new video of the result.
Analogy: It's like hiring a chef to cook a meal. You don't just read their recipe; you watch them cook, taste the food, and record the meal being served to see if it actually works.

3. The "Human-Aligned Rubric" (The Grading Sheet)

This is the paper's biggest innovation. Instead of a computer comparing pixels (which is too strict and misses the point), they created a detailed checklist based on human feelings.

The checklist asks specific questions like: "Did the button turn blue when the mouse hovered over it?" or "Did the text fade in smoothly?"
Analogy: Instead of a teacher saying "This essay is 8/10," they give a rubric: "Did you use a thesis statement? Yes/No. Did you use 3 paragraphs? Yes/No." This makes grading fair and consistent.

4. The "Judge" (Evaluation)

A super-smart AI acts as the judge. It watches the Reference Video (the original) and the Generated Video (the AI's creation) side-by-side. It uses the checklist to score them.

The Secret Sauce: The judge looks at both the code and the video. The code tells the judge what was intended, and the video tells the judge what actually happened.

📉 What They Found: The "Motion Gap"

The researchers tested 19 of the smartest AI models in the world. Here is what they discovered:

The "Look" is Easy: The AIs are getting really good at copying the colors, fonts, and layout. (Score: ~80/100).
The "Feel" is Hard: The AIs struggle terribly with animations and interactions. They often forget to make buttons clickable, or the animations are jerky and wrong. (Score: ~60/100).
The "Thinking" Helps: Models that have a "Thinking" mode (where they pause to plan before answering) did better, but they still couldn't fully master the complex movements.

The Analogy: Imagine a dance instructor. The AI is great at standing in the right pose (static layout), but when the music starts and the dance begins (interaction/motion), it trips over its own feet.

🏆 Why This Matters

For Developers: It shows us exactly where AI needs to improve. We can't just make AI "smarter"; we need to teach it how to understand time and motion.
For the Future: If we can fix this, you could record a video of your dream website on your phone, and an AI could build the entire working site for you in seconds, complete with all the cool animations.
For Science: They proved that using a "Video + Code + Checklist" approach is the only way to truly test if an AI is building a real website or just a fake picture.

🚀 In a Nutshell

WebVR is a new ruler for measuring AI. It says: "Don't just show us a picture of what you built. Show us the video of it working. If the buttons don't click and the animations don't flow, you haven't passed the test."

It's a wake-up call: The future of web design isn't just about static images; it's about bringing the web to life, and right now, AI is still learning how to dance.

1. Problem Statement

Current benchmarks for web generation primarily rely on text prompts or static screenshots as input. This approach has two critical limitations:

Loss of Dynamic Information: Static inputs fail to capture essential temporal signals such as interaction flows, transition timing, motion continuity, and animation easing, which are crucial for faithful webpage recreation.
Evaluation Gaps: Existing evaluation protocols often use coarse-grained metrics or static structural checks, lacking a principled framework to assess fine-grained visual fidelity and dynamic interaction correctness.

Consequently, the capability of Multimodal Large Language Models (MLLMs) to translate video demonstrations into executable, high-fidelity front-end code remains largely unexplored and unquantified.

2. Methodology: The WebVR Framework

The authors introduce WebVR, the first benchmark specifically designed for video-to-webpage generation. The framework consists of a controlled data synthesis pipeline and a novel evaluation protocol.

A. Data Synthesis Pipeline

To prevent data contamination (since MLLMs are pre-trained on existing web data), WebVR constructs 175 unique webpages through a four-stage synthesis process:

Seed Data Preparation: Real-world showcase videos are collected from design galleries. An MLLM generates a structured caption abstracting the design (aesthetics, layout, motion).
Semantic Re-theming: The caption is rewritten into a fictional specification with a different domain (e.g., changing a "SaaS Dashboard" to an "Art Gallery") while preserving the exact layout, interaction logic, and motion patterns. This ensures the models cannot rely on memorized content.
Visual Asset Retrieval: To reduce ambiguity, the system retrieves specific high-resolution images from Unsplash based on keywords extracted from the re-themed caption, providing grounded visual assets for the models.
Candidate Generation & Selection: Multiple MLLMs generate HTML code based on the re-themed caption and assets. These candidates are rendered in a standardized sandbox, and the best-performing execution video is selected as the Reference Video ( $V_{ref}$ ) for the benchmark.

B. Evaluation Protocol

WebVR evaluates models by comparing their generated output against the reference video using a Human-Aligned Visual Rubric.

Execution Sandbox: Generated HTML is executed in a standardized browser environment (Chromium, 2560x1440, 30 FPS) with fixed interaction scripts to produce an Execution Video ( $V_{gen}$ ).
Visual Rubrics: Instead of pixel-matching, the system uses a set of fine-grained, atomic binary criteria ( $R$ $R$ ) derived from the design specification. These criteria cover four orthogonal dimensions:
1. Global Aesthetics (GA): Color mood, typography, coherence.
2. Navigation and Footer (NF): Persistent UI components.
3. Section-Specific Layouts (SSL): Grid structure, hierarchy, spacing.
4. Interaction and Motion (IM): Hover effects, scroll animations, state transitions.
Automated Judge: An MLLM-based judge ( $J$ ) evaluates the pair $(V_{gen}, C_{gen})$ against the rubric $R$ . It outputs a binary satisfaction status for each criterion, and the final score is the average satisfaction rate.

3. Key Contributions

WebVR Benchmark: The first dataset of 175 diverse, video-conditioned webpage tasks. It uses a controlled synthesis pipeline to ensure data freshness and includes paired visual assets for fair evaluation.
Video-Based Evaluation Protocol: A novel approach that renders generated code into videos for comparison, enabling the assessment of dynamic behaviors (animations, transitions) that static screenshots miss.
Human-Aligned Visual Rubric: A fine-grained evaluation framework that guides an MLLM judge to score based on atomic visual checks. This method achieves 96% agreement with human expert preferences, significantly outperforming rubric-free evaluation.

4. Experimental Results

The authors evaluated 19 state-of-the-art MLLMs (including Qwen, Kimi, GPT, Gemini, and Claude series).

Overall Performance: The best-performing model, Kimi-K2.5, achieved an overall score of 79.14, followed closely by Claude-Sonnet-4.6 (78.49) and GPT-5.2-Thinking (77.93). Notably, top open-source models are competitive with closed-source counterparts.
Dimensional Analysis:
- Strengths: Models perform well on Global Aesthetics (GA) and Navigation/Footer (NF), with top scores exceeding 89%. This indicates strong capability in capturing static style and layout.
- Weaknesses: Interaction and Motion (IM) remains a significant bottleneck. Even the best models only score around 60 on IM. There is a steep performance drop (approx. 27+ points) between static aesthetics and dynamic interactions.
Scaling Effects: Larger models and "Thinking" (reasoning-enhanced) variants show consistent improvements, particularly in GA and NF. However, reasoning helps less with the IM dimension, suggesting that current scaling laws do not fully solve the temporal logic problem.
Evaluation Impact: The study confirms that using the visual rubric is critical. Without it, automated judges show low agreement with humans (59–66%) and produce inflated, uninformative scores. With the rubric, agreement rises to 76–96%.

5. Failure Analysis

Qualitative analysis identified three distinct failure modes:

Structural Collapse: Weaker models fail to render valid HTML, resulting in blank pages or broken layouts.
Semantic Divergence: Mid-tier models generate valid but semantically unrelated pages (e.g., hallucinating a generic template instead of the specific design).
Asset Misplacement: Stronger models capture the layout skeleton but fail at fine-grained spatial grounding (e.g., placing images in wrong locations or referencing wrong assets).

6. Significance

WebVR establishes a rigorous standard for evaluating the next generation of front-end AI agents. It highlights that while MLLMs have largely mastered static visual cloning, faithfully recreating dynamic, interactive web experiences from video remains a major challenge. The benchmark and the rubric-based evaluation toolkit provide a necessary foundation for developing models capable of understanding temporal cues and translating them into executable interaction logic, pushing the field toward truly "interaction-responsive" web synthesis.