💬 NLP

Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining

The paper introduces Video2GUI, an automated framework that synthesizes a large-scale dataset of 12 million GUI interaction trajectories from unlabeled internet videos to overcome data scarcity and significantly enhance the generalization performance of multimodal GUI agents.

Original authors: Weimin Xiong, Shuhao Gu, Bowen Ye, Zihao Yue, Lei Li, Feifan Song, Sujian Li, Hao Tian

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Weimin Xiong, Shuhao Gu, Bowen Ye, Zihao Yue, Lei Li, Feifan Song, Sujian Li, Hao Tian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to use a computer, a phone, or a website. To do this, you need to show it millions of examples of people clicking buttons, typing text, and scrolling through pages.

The problem is that finding these examples is incredibly hard. Usually, researchers have to hire people to manually record and label every single click, which is slow, expensive, and limits them to just a few types of apps. It's like trying to learn a language by only reading a few pages of a dictionary.

Video2GUI is a new system that solves this by turning the entire internet into a giant, free classroom. Here is how it works, using some simple metaphors:

1. The Great Filter (Finding the Right Videos)

The internet is full of videos, but most are useless for teaching a robot (like cooking shows or cat videos). The researchers started with 500 million YouTube videos.

The Rough Sift: First, they used a smart AI to scan the titles and descriptions. This is like a librarian quickly checking the spines of books to see if they are about "computers" before even opening them. This cut the pile down to 20 million.
The Fine Sift: Next, they used a more advanced AI to actually "watch" the first minute of those videos. It checked: Is the screen clear? Is the voice explaining the steps well? Is it actually showing how to use software? This is like a strict teacher grading the videos to ensure they are high-quality tutorials.
The Result: They ended up with 4.2 million high-quality tutorial videos.

2. The Translator (Turning Video into Instructions)

Now they had the videos, but a robot can't just "watch" a video and understand it like a human does. The video needs to be translated into a structured list of instructions.

The Process: They used a powerful AI (like a super-smart translator) to watch the videos and break them down. It identified the goal (e.g., "Buy shoes"), the steps taken, and the exact moment each click happened.
The Magic: The AI didn't just guess; it looked at the video frames to find the exact coordinates of the buttons being clicked. It turned a 10-minute video of someone shopping online into a precise, step-by-step map: "At 0:05, click the search bar. At 0:10, type 'sneakers'."

3. The Library (WildGUI)

All these translated instructions were collected into a massive new library called WildGUI.

The Scale: This library contains 12 million interaction paths (trajectories) covering over 1,500 different apps and websites.
The Variety: It includes everything from Windows desktops to Android phones and Mac computers. It's like having a library that contains every possible way a human has ever used a computer, all organized and ready for a robot to study.

4. The Training (Teaching the Robot)

The researchers took two existing AI models (Qwen2.5-VL and Mimo-VL) and "fed" them this new library.

The Result: After studying this massive dataset, the robots got significantly better. They improved by 5% to 20% on various tests.
The Proof: They could now find buttons, click the right things, and navigate complex websites much better than before, matching or beating the best robots currently in existence.

The Bottom Line

The paper claims that by automating the process of turning internet videos into training data, they created a massive, diverse dataset that makes AI agents much smarter at using computers. They didn't invent a new type of robot; they just gave the existing ones a much better, larger, and more diverse textbook to study from.

They have released this dataset and the tools used to build it so that other researchers can use them to build even better AI agents in the future.

Technical Summary: Video2GUI

Problem Statement

The development of generalized Graphical User Interface (GUI) agents is currently constrained by a scarcity of large-scale, diverse training data. Existing approaches rely heavily on manually annotated interaction trajectories or simulated environments, which are costly to produce and suffer from limited scalability. Consequently, agents trained on these narrow datasets struggle to generalize to unseen interfaces, applications, and real-world tasks. While the internet contains a vast repository of unlabeled GUI tutorial videos, effectively leveraging this data is challenging due to the difficulty of filtering high-quality instructional content and the complexity of extracting structured, spatially grounded action trajectories from raw video without explicit annotations.

Methodology

To address these challenges, the authors propose Video2GUI, a fully automated framework designed to synthesize large-scale interaction trajectories from unlabeled internet videos. The pipeline operates through three progressive stages:

Coarse-to-Fine Video Filtering:
- Coarse Filtering (Metadata): Starting with 500 million video metadata entries (titles, descriptions, keywords), the system employs a lightweight classifier (fine-tuned Qwen2.5-7B) to filter out videos unrelated to GUI operations. This reduces the candidate pool to approximately 20 million videos, avoiding the computational cost of processing irrelevant content.
- Fine Filtering (Visual Scoring): A visual scoring model (fine-tuned Qwen2.5-Omni) evaluates the content of the first minute of each candidate video across three dimensions: Topic Relevance (focus on teaching GUI operations), Instruction Clarity (coherence of narration), and Recording Quality (visual fidelity). This stage retains only 4.16 million high-quality videos (~300,000 hours).
Trajectory Extraction:
- Using a sliding-window strategy with historical context memory, the framework employs a powerful vision-language model (Gemini-3-Pro) to convert video segments into structured instruction-trajectory pairs.
- The model extracts task instructions, action timestamps, action details, and the reasoning behind each action. It generates a sequence of steps where each action is associated with a precise timestamp and a low-level textual description.
Action Spatial Grounding:
- To overcome the resolution limitations of compressed video frames, the system retrieves high-resolution screenshots (a triplet of frames at $t-0.5s$ , $t$ , and $t+0.5s$ ) for each extracted action timestamp.
- Gemini-3-Pro is used to map the low-level instructions to precise screen coordinates (bounding boxes or points), effectively grounding the actions in the UI. This multi-frame approach mitigates issues caused by rapid visual changes in GUI interactions.

Training Strategy:
The authors utilize the resulting dataset, WildGUI, for a two-stage agent training process:

Stage 1 (Continual Pre-training): Models (Qwen2.5-VL and Mimo-VL) are pre-trained on WildGUI using a mixed objective of GUI grounding, action prediction, and trajectory modeling.
Stage 2 (Post-training): The pre-trained models are fine-tuned on curated, high-quality open-source datasets to refine performance on specific downstream tasks.

Key Contributions

Video2GUI Framework: A scalable, fully automated pipeline that extracts grounded GUI interaction trajectories from billions of unlabeled internet videos, eliminating the reliance on manual annotation.
WildGUI Dataset: The construction of a large-scale, diverse pre-training dataset containing 12.7 million interaction trajectories and 124.5 million screenshots. It spans over 1,500 applications and websites across desktop, mobile, and web platforms.
Performance Gains: Extensive experiments demonstrate that pre-training on WildGUI yields consistent improvements (5–20%) across multiple GUI grounding and agent benchmarks, enabling models to match or surpass state-of-the-art performance despite having smaller parameter counts.
Open Release: The authors commit to releasing both the WildGUI dataset and the Video2GUI pipeline to facilitate future research in scalable GUI agent training.

Experimental Results

The authors evaluated the approach using Qwen2.5-VL and Mimo-VL backbones across several benchmarks:

GUI Grounding: On ScreenSpot-Pro and OSWorld-G, WildGUI-pretrained models showed significant gains. For instance, Mimo-VL-7B achieved a state-of-the-art average score of 67.6 on OSWorld-G, surpassing larger models like Qwen3-VL-32B (60.6) and Seed1.5-VL (62.9).
Offline Agent Evaluation: On AndroidControl and CAGUI, the models demonstrated improved planning and execution capabilities. Mimo-VL-7B achieved a step success rate of 71.4 on AndroidControl-High, a substantial improvement over its base model.
Online Agent Evaluation: In dynamic environments (OSWorld and AndroidWorld), the full training pipeline (Stage 1 + Stage 2) nearly doubled the success rate on AndroidWorld (31.9% vs. 16.4% for the base model), demonstrating strong generalization to open-ended, real-world tasks.
Scaling Analysis: A scaling study revealed a strong positive correlation between the volume of pre-training tokens (up to 200 billion) and downstream task accuracy, with performance continuing to improve without evident saturation.

Significance and Claims

The paper claims that the primary bottleneck for generalized GUI agents is data scarcity and diversity, not just model architecture. By demonstrating that unlabeled internet videos can be effectively transformed into high-quality, structured training data, Video2GUI offers a scalable pathway to overcome this limitation.

The authors assert that:

Data Quality is Paramount: The coarse-to-fine filtering and VLM-driven extraction strategies are critical for ensuring the synthesized data is accurate and diverse enough to train robust agents.
Offline Data Generalizes: Pre-training on offline video data provides a critical foundation that enables agents to generalize effectively to online, dynamic environments, even though the training data itself is static.
Scalability: The Video2GUI pipeline is capable of processing internet-scale data, suggesting that the field can move beyond the constraints of small, manually curated datasets toward truly large-scale, diverse agent training.

The work concludes that scaling training with diverse, offline video data is a promising and necessary direction for developing capable, general-purpose autonomous GUI agents.