Technical Summary: UI2App Benchmarking Visual Interaction Inference

Problem Statement

While Large Language Models (LLMs) and Vision-Language Models (VLMs) have demonstrated growing competence in generating web pages from natural language or single screenshots, existing approaches suffer from significant limitations. Text-driven paradigms rely on complex prompts that struggle to precisely specify detailed visual layouts and cross-page visual coherence. Conversely, image-driven paradigms (screenshot-to-code) have largely focused on visual fidelity—measuring how closely the rendered output matches a reference image—while neglecting interaction inference.

Current benchmarks often produce artifacts that are visually correct but behaviorally inert "frozen façades." They fail to evaluate whether the generated application can recover dynamic behaviors, state management, and cross-page interactions solely from static visual evidence. There is a critical gap in measuring a model's ability to infer missing interaction logic (e.g., state persistence across routes, data CRUD operations) without explicit textual or demonstration-based specifications.

Methodology

The UI2App Benchmark

The authors introduce UI2App, the first benchmark specifically targeting interaction inference: the ability to recover application behavior from screenshots alone, without any textual or behavioral guidance.

• Dataset Construction: The benchmark comprises 327 screenshots organized into 45 state-coherent screenshot sets. These sets represent runnable, multi-route web applications across four categories: Content, Admin, Transaction, and Specialty.
• Source: Screenshots are curated from open-source GitHub projects, filtered for permissive licensing, structural validity, and buildability.
• Task Definition: Models are given a set of $M$ screenshots (4–14 per task) capturing distinct routes or visual states. No action traces or interaction descriptions are provided. The model must generate source code (React + TypeScript) for a runnable application that reproduces the screenshots and their implied behaviors.

Evaluation Protocol

The evaluation is an end-to-end pipeline scoring artifacts along four dimensions:

1. Executability (EXEC): Measures if the code builds and renders.
   • EXEC@1: One-shot pass rate.
   • EXEC@3: Pass rate after up to three rounds of self-debugging using build error feedback.
2. Navigation Reachability (NRS): The fraction of input screenshot routes that are reachable from the home route via visible navigation in the generated app.
3. Visual Fidelity (VFS): A judge-free, block-level matching metric comparing the rendered DOM to the reference. It averages four sub-metrics: Size, Text, Position, and Color.
4. Interaction Inference Score (IIS): The core contribution. IIS quantifies how well a model infers and realizes screenshot-implied interactions.
   • Taxonomy: Interactions are categorized into seven types (Toggle, Expand/Collapse, List Operations, Data CRUD, Form Validation, Notification, Cross-Route State).
   • Scope: Interactions are weighted by state-management complexity: S1 (UI-state), S2 (Data-state), and S3 (Cross-route persistence).
   • Scoring: Unlike previous benchmarks that compare against a single reference trace, IIS uses a rubric-based protocol. It evaluates whether the generated application realizes the functional criteria of the interaction (trigger, state transition, result), accommodating diverse but semantically equivalent implementations.

Key Contributions

1. UI2App Benchmark: A dataset of 45 multi-route, state-coherent application sets designed to test interaction inference from image-only inputs, moving beyond static visual reconstruction.
2. Interaction Inference Score (IIS): A novel metric grounded in an interaction taxonomy and scope-based weighting. It assesses inferred behaviors based on functional correctness and state-management complexity rather than matching a single ground-truth execution trace.
3. Comprehensive Evaluation: The paper establishes the first baselines for image-only interaction inference by evaluating six frontier VLMs (including Claude Sonnet 4.6, Gemini 3.1 Pro, GPT-5.4, Kimi K2.5, Qwen3.5, and GLM-4.6V) and a scaling ladder of Qwen2.5-VL.

Experimental Results

The evaluation of six frontier models reveals a marked capability mismatch between visual reconstruction and interaction realization:

• Visual Fidelity $\neq$ Interaction Inference: The model leading in Visual Fidelity (Gemini 3.1 Pro Preview, VFS=78.1) ranks fourth in IIS (7.5). Conversely, the IIS leader (Claude Sonnet 4.6, IIS=39.3) ranks second in VFS (75.7). The IIS leader is 5.2× ahead of the VFS leader in interaction inference.
• Cross-Route State Bottleneck: The most significant challenge is S3 (cross-route state persistence). Three of the six models scored exactly zero on this dimension. Even the best model (Claude) achieved only 21.6/100 on S3 interactions.
• Admin Applications: Admin-style apps proved to be a consistent bottleneck, yielding the lowest mean IIS (9.4) across all models.
• Scaling Behavior: Within the Qwen2.5-VL family, a "phase transition" was observed between 32B and 72B parameters, where executability and visual fidelity jumped sharply. However, parameter scaling alone did not saturate the interaction inference requirement.
• Failure Modes: Analysis showed that while some models (e.g., GPT-5.4) improved significantly via self-debugging for build errors, this did not translate to better interaction inference. Failures in interaction often stemmed from missing state containers (e.g., CartContext) or dead code where UI elements were rendered but handlers were not wired.

Significance and Claims

The paper claims that inferring complete interaction behavior from static screenshots remains a key frontier challenge for vision-language models.

• Decoupling of Capabilities: The results demonstrate that visual fidelity is a poor proxy for interaction realization. A model can produce a pixel-perfect replica that is functionally inert.
• Beyond Specification-Following: Unlike benchmarks where interaction is explicitly specified via text or video, UI2App requires the model to perform interaction inference—recovering missing behavior from visual cues alone. This represents a shift from "specification-following" to "latent-affordance inference."
• Future Direction: The authors posit that UI2App provides a necessary testbed to drive models toward stronger inference capabilities, specifically addressing the gap in cross-route state persistence and complex state management that current models fail to infer.

The paper concludes that while models have advanced in code generation, the ability to synthesize runnable, interactive applications from static visual evidence without explicit behavioral guidance is not yet solved, with cross-route state management being the most pervasive bottleneck.