Technical Summary: FIGMATRACE

Problem Statement

Vision Language Models (VLMs) have demonstrated significant proficiency in objective, verifiable domains such as object detection and document understanding. However, they continue to underperform on subjective and creative design tasks. The authors identify a primary bottleneck: a lack of high-quality human workflow data that captures the diverse preferences, decisions, and "taste" that distinguish human experts. While prior works have attempted to generate design data through automated annotation of existing Figma files or by converting HTML to JSON, these approaches suffer from ambiguous validation, a lack of exhaustive quality guidelines, and an inability to capture the nuanced decision-making processes of human designers. Furthermore, existing datasets often focus on the final artifact rather than the trajectory of decisions required to create it.

Methodology

1. Skill Taxonomy and Task Curation

To address the data gap, the authors first defined a unique, expert-curated taxonomy of 10 high-level design skills (e.g., Visual Perception, Figma Structural Craft, Accessibility Expertise, Vector & Asset Craft). Based on this taxonomy, they constructed 126 open-ended, subjective, long-horizon tasks. These tasks were categorized into:

• Verifiable Tasks: Pixel-perfect replication, flaw repair, and accessibility remediation.
• Non-Verifiable Tasks: Platform adaptation, theming, sketch-to-Figma conversion, and prototype wiring, which rely on expert judgment and bias.

2. Data Collection and Processing (FIGMATRACE)

The dataset, FIGMATRACE, was built by capturing over 200 hours of OS-level expert screen recordings and actions while designers completed these tasks. The raw data underwent a rigorous multi-stage processing pipeline:

• Action Filtering: Random mouse movements and hovers were filtered out, retaining only meaningful interactions (clicks, typing, scrolling) mapped to the Playwright MCP toolset.
• Frame Extraction: A two-pass extraction method was employed to identify "settled" states after actions, ensuring that frames were captured only when the UI had stabilized, rather than at arbitrary time intervals.
• Effect Filtering: Changes in pixel values were analyzed to distinguish between actions that altered the artifact and those that did not.
• Phase-Based Segmentation: A novel contribution of this work is the conversion of video data into trajectories based on design phases (e.g., "Componentising," "Refinement Polish," "Reference Gathering") rather than simple maximum context-length sharding. The authors utilized Gemini-3.6-Flash to categorize video spans into 11 distinct phases. This approach was chosen to teach VLMs specific skills and intent boundaries, avoiding the noise introduced by random pauses in video.

3. Training and Evaluation

The authors trained four VLMs (QWEN3.6-35BA3B, QWEN3.8-27B, GEMMA-4-31B, and MUSE-GLIMMER-30B) using the ms-swift framework on 92,472 actions sampled from 35 sessions. The models were evaluated on four out-of-distribution (OOD) agentic GUI environments:

• GUI-Odyssey: Multi-app, multi-viewport navigation.
• AndroidControl: Instruction granularity and mobile navigation.
• Mind2Web: Instruction grounding on open web data.
• VideoGUI: Planning and action narration.

Key Results

Performance Improvements

Training on FIGMATRACE yielded significant performance gains across all evaluated benchmarks:

• Benchmark Dominance: The fine-tuned QWEN3.8-27B model outperformed frontier closed-source models like CLAUDE-OPUS-5 and GPT-5.6-SOL on specific tasks. Notably, it achieved a 6.4% absolute increase on GUI-Odyssey and an 11.8% absolute increase on AndroidControl compared to the closed baselines.
• In-Domain Gains: On the ScreenSpot-Pro Creative split, the fine-tuned QWEN3.8-27B showed a 7.4% absolute increase over its base model (36.7% vs. 29.3%).
• Generalization: The improvements generalized across different model architectures, with all four tested models showing positive gains.

Ablation Study: Phase-Based vs. Length-Based

The authors conducted a critical ablation study comparing their phase-based trajectory curation against standard maximum context-length sharding.

• Result: The phase-based approach outperformed the length-based approach by 7.3 absolute points on average.
• Analysis: Qualitative analysis revealed that length-based sharding often cuts tasks mid-action, obscuring task intent. In contrast, phase-based segmentation preserved skill-based grounding, allowing models to better understand "Continue the work" instructions and undirected references.

Qualitative Analysis

Human evaluation of the best-performing model (QWEN3.8-27B) identified three key patterns driving success:

1. Element Selection Accuracy: The base model frequently selected incorrect UI elements (median error ~457 px), whereas the fine-tuned model landed within ~15 px of the target.
2. Coordinate Understanding: The base model often emitted raw pixel coordinates that exceeded normalized grid limits (e.g., $y > 1000$), while the fine-tuned model adhered to the norm-1000 coordinate system.
3. Decisiveness: The fine-tuned model consistently provided coordinates even in ambiguous scenarios where the base model failed to act.

However, the authors also noted failure modes introduced by the dataset, including repetition (predicting the same pixel twice) and a screen-center focus bias, likely artifacts of noise in the raw video processing.

Significance and Claims

The paper claims that FIGMATRACE is the first dataset to provide pairs of gold action sequences and intent-segmented recordings of expert Figma workflows. Its primary significance lies in demonstrating that:

1. Data Quality over Quantity: Capturing high-quality, expert-curated human workflows with a focus on design phases is more effective for training creative agents than simply scaling up raw data or using automated annotation.
2. Phase-Based Curation: Structuring training data around design phases (skills) rather than arbitrary time windows significantly improves a model's ability to understand long-horizon task intents.
3. Generalization: Training on specific design workflows (Figma) transfers effectively to broader agentic navigation tasks (Android, Web), suggesting that the underlying skills of visual analysis and structural reasoning are transferable.

The authors have open-sourced the dataset and the best-performing model (QWEN3.8-27B) to facilitate further research in creative AI agents.