Technical Summary: Video2Code

Problem Formulation

The paper addresses the challenge of generating executable front-end code (HTML, CSS, JavaScript) from UI interaction videos. While Vision-Language Models (VLMs) have advanced in static UI-to-code tasks (reconstructing layout from screenshots), they struggle with dynamic behavior. The authors identify a critical failure mode in existing video-capable VLMs termed state-transition misalignment.

Current models typically rely on sparse frame sampling or compressed temporal representations to manage context budgets. While effective for natural videos, this approach is fragile for UI interactions where code-relevant evidence is concentrated around short action boundaries (e.g., a click triggering a dropdown). Sparse sampling often misses these brief transitions, breaking the state-action-state sequence required to implement correct webpage behavior. Consequently, models may reconstruct the visual appearance but fail to recover the executable logic triggered by user actions.

The authors formulate the task not merely as visual reconstruction, but as executable state-transition recovery. The goal is to preserve the complete transition evidence: the UI state before an action, the action execution itself, and the resulting UI state.

Methodology: Video2Code

To address state-transition misalignment, the authors propose Video2Code, an action-aware video-to-code approach. Instead of allocating visual budget uniformly across the video, Video2Code employs a Coarse-to-Fine strategy centered on Action-Aware Revisit.

1. Action-Aware Revisit Process

The method operates in two rounds:

• Coarse Video Reading (Round 1): The model performs an initial pass over the full UI interaction video at a coarse temporal resolution. Its objective is to infer the global interaction flow and predict action-critical temporal segments. These segments are defined as intervals covering three elements: the pre-action state, the action execution, and the post-action state.
• Temporal Clipping Tool: Based on the predicted segments, the model invokes a tool (extract_multi_segment) to retrieve high-resolution clips of these specific intervals.
• Focused Segment Revisit (Round 2): The retrieved high-resolution clips are reintroduced into the model's context. The model combines the global context from the coarse pass with the local, high-fidelity evidence from the revisited clips to generate the final executable code.

2. Definition of Action-Critical Segments

The paper distinguishes between two types of critical segments to ensure complete transition evidence:

• Pure Action Segments: Contain a single operation (click, text input, selection, scroll) including pre-action, execution, and post-action states.
• Action-Scroll Segments: Group an action with a subsequent scroll if the scroll is necessary to reveal the result of the action (e.g., clicking a link loads content that only becomes visible after scrolling). This prevents the model from missing contextual dependencies.

3. Action-Aligned Supervision and Training

To train the model, the authors constructed WebVidCoding, a dataset of 12,000 action-aligned video-code pairs.

• Data Collection: An automated browser exploration agent interacts with webpages, recording UI interaction videos. A lightweight visual marker (a 12-pixel bar) is injected to provide explicit temporal anchors for action boundaries.
• Supervision Structure: The training follows the same two-round inference structure:
  • Round 1: Supervises the model to predict temporal windows for the clipping tool based on the action timeline.
  • Round 2: Supervises the generation of executable code using the global video context and the specific clips returned from Round 1.
• Base Model: The approach is instantiated by fine-tuning GLM-4.6V-100B on this dataset using supervised fine-tuning.

Key Contributions

1. Problem Formulation: The authors define UI video-to-code generation as executable state-transition recovery and identify state-transition misalignment as the primary bottleneck in budget-limited video observation.
2. Methodological Innovation: They propose Video2Code, which utilizes a Temporal Clipping Tool to revisit action-critical regions at higher temporal resolution. This allows the model to recover local transition evidence without densifying the entire video.
3. Dataset and Supervision: They introduce WebVidCoding, a dataset with action-aligned video-code supervision, enabling the model to learn both revisit planning and code generation from recovered transition evidence.

Experimental Results

The authors evaluate Video2Code on WebVideo2Code-Bench, comprising real-world and synthetic webpage scenarios. Evaluation metrics include Visual Similarity (matching rendered states) and Functional Correctness (reproducing state changes after executing actions via browser-based verification).

• Performance Gains: On the real-world subset (WebVideo2Code-Real), Video2Code significantly outperforms the base GLM-4.6V model.
  • Visual Similarity: Improved from 70.94% to 82.06%.
  • Functional Correctness: Improved from 29.85% to 61.99%.
• Comparison with Baselines: While closed-source models (e.g., Kimi-K2.6, Gemini-3.1-Pro) achieve strong results, open-source models using direct video observation struggle with functional correctness. Video2Code substantially strengthens the underlying open-source model, particularly on dense multi-step interactions.
• Ablation Studies: Removing the "Revisit" mechanism (disabling the temporal clipping tool) causes a significant drop in performance, especially in functional correctness (dropping from 61.99% to 47.84%), confirming that focused revisit is crucial for recovering executable transition evidence.
• Robustness: The model maintains relatively stable performance across varying interaction counts (1–16 steps) and video lengths, though functional correctness decreases slightly with higher interaction-type diversity.

Significance and Claims

The paper claims that Video2Code demonstrates that Action-Aware Revisit is a necessary mechanism for behavior-faithful UI video-to-code generation. The authors argue that simply increasing the visual budget uniformly is insufficient; instead, the model must dynamically allocate resources to recover the state-action-state transitions that define interactive behavior.

The work positions itself as a step toward generating webpages that are not just visually accurate but functionally executable, bridging the gap between static UI reconstruction and dynamic interaction modeling. The authors modestly note that their current scope is limited to common interactions (clicks, scrolls, inputs) and does not yet cover complex scenarios like drag-and-drop, file uploads, or highly asynchronous backend-dependent behaviors.