Technical Summary: Grounding Computer Use Agents on Human Demonstrations (GROUNDCUA & GROUNDNEXT)

Problem Statement

The development of reliable Computer-Use Agents (CUAs) capable of operating software on behalf of users hinges on grounding: the ability to accurately map natural language instructions to specific on-screen UI elements. While large datasets exist for web and mobile interactions, high-quality resources for desktop environments remain limited. Desktop applications present unique challenges, including high-resolution displays, dense layouts, visually similar elements, and user-specific artifacts that introduce variability unseen during training. Existing automated datasets often suffer from incomplete accessibility signals, underrepresentation of small icon-only controls, and a lack of fine-grained annotations necessary for precise localization. Consequently, agents often fail to execute plans correctly when they cannot identify the exact button, menu item, or control required.

Methodology

1. The GROUNDCUA Dataset

To address the scarcity of high-quality desktop data, the authors introduce GROUNDCUA, a large-scale, human-annotated dataset derived from expert demonstrations.

• Data Collection: The dataset comprises 56,000 screenshots from 87 open-source applications across 12 categories (e.g., Office, Development, CAD, Scientific). Unlike prior works relying on automated traversal or random state generation, data was collected by trained annotators performing real-world tasks (e.g., editing spreadsheets, running simulations).
• Annotation Process: Keyframes capturing the state immediately before user actions were extracted. Annotators provided 3.56 million human-verified annotations, labeling every visible element with bounding boxes, textual names, and categories. Approximately 50% of elements were assigned to high-level categories (e.g., Menu, Button, Slider).
• Instruction Generation: Leveraging dense annotations (bounding boxes, OCR, context), the authors constructed a 700K instruction-tuning dataset. Instructions were generated in three types to mimic real-world diversity:
  • Direct: Describing attributes or text (e.g., "Click the 'Save' button").
  • Functional: Focusing on the action (e.g., "Open a new tab").
  • Spatial: Using relative positioning (e.g., "Click the icon left of 'Files'").
• Scale & Diversity: The dataset features high-resolution images (0.39M to 7.0M pixels) and an average of 64 annotations per screenshot, significantly denser than existing benchmarks. It includes fine-grained elements like small icons and toolbars often missed by automated pipelines.

2. The GROUNDNEXT Model Family

The authors developed GROUNDNEXT, a series of vision-language models (3B and 7B parameters) based on Qwen2.5-VL-Instruct, trained to map instructions to target UI coordinates.

• Stage 1: Supervised Fine-Tuning (SFT): Models were trained on the curated 700K instruction-image pairs from GROUNDCUA. Both the vision encoder and language model were fine-tuned.
• Stage 2: Reinforcement Learning (RL): A post-training stage using Relative Leave-One-Out (RLOO) optimization.
  • Reward Function: A customized discrete reward based on the normalized distance between the predicted coordinate and the ground-truth bounding box. Predictions inside the box receive positive rewards, while those outside receive penalties scaled by their distance.
  • Data: 10K new samples (distinct from the SFT set) were used for RL to ensure generalization.

Key Contributions

1. GROUNDCUA Dataset: The largest expert-annotated desktop grounding dataset to date, featuring 56K screenshots, 3.56M+ elements, and high-resolution diversity across 87 applications. It addresses the gap in dense, fine-grained desktop supervision.
2. GROUNDNEXT Models: A family of 3B and 7B models trained with a two-stage SFT+RL pipeline. They achieve state-of-the-art (SOTA) performance on desktop benchmarks using significantly fewer training samples (700K SFT + 10K RL) compared to prior models trained on millions of samples.
3. Efficiency and Generalization: The work demonstrates that high-quality, expert-driven data can outperform larger, less precise datasets. The models show strong cross-platform generalization to mobile and web environments despite being trained exclusively on desktop data.

Results

Benchmarks

The models were evaluated on five benchmarks: ScreenSpot-Pro, OSWorld-G, UI-Vision, MMBench-GUI, and ScreenSpot-v2.

• SFT Performance: GROUNDNEXT-3B (SFT) achieved an average score of 66.4 (including UI-Vision) and 68.4 (excluding UI-Vision), outperforming the next best 3B SFT baseline (GUI-Actor-3B) by a significant margin (+12.1 points with UI-Vision). GROUNDNEXT-7B (SFT) similarly led the 7B SFT group.
• RL Performance: Adding RL further improved results. GROUNDNEXT-3B (RL) reached 68.4 (Avg) and GROUNDNEXT-7B (RL) reached 70.5.
• Comparison: Despite using less than 1/10th of the SFT data used by models like JEDI (9M samples), GROUNDNEXT outperformed existing SFT baselines and remained competitive with RL-tuned systems.

Agentic Performance (OSWorld-Verified)

Evaluated in a multi-step agentic setting using OpenAI o3 as a planner:

• GROUNDNEXT-3B (RL) achieved an overall score of 50.6, significantly outperforming other 3B open-source models (e.g., OpenCUA-A3B at 17.7) and surpassing larger models like OpenCUA-72B (46.1) and proprietary APIs like Qwen3-VL-Flash (41.6).
• It achieved performance comparable to JEDI-7B (51.0), despite being less than half the size, demonstrating high practical utility for resource-constrained systems.

Analysis

• Data Quality vs. Quantity: Training the same base model on 100K samples from various datasets (Aguvis, UGround, JEDI, etc.) showed that GROUNDCUA yielded the highest performance, confirming that dense, expert supervision is more effective than raw scale.
• RL Gains: RL provided modest but consistent improvements. Models trained on GROUNDCUA during SFT showed smaller RL gains than those trained on other datasets, suggesting the SFT data was already highly informative.
• Cross-Domain: The models generalized well to mobile and web interfaces (MMBench-GUI, ScreenSpot-v2), though performance on web tasks was slightly lower than on desktop, indicating a need for future multi-domain data integration.

Significance and Claims

The paper claims that high-quality, expert-driven datasets are critical for advancing general-purpose computer-use agents. The primary takeaway is that data quality and density can substitute for raw data volume. By focusing on human-verified, fine-grained annotations of real-world desktop interactions, the authors demonstrate that smaller models can achieve superior grounding performance with efficient training.

The work positions GROUNDCUA and GROUNDNEXT as foundational resources for the open research community, aiming to unlock grounding as a core capability for reliable, adaptable agents. The authors emphasize that while their current RL reward design is simple, the dense annotations in GROUNDCUA enable future development of more precise reward signals. They also highlight the importance of open-source applications in the dataset for ensuring broad, license-compliant coverage of real-world software functionality.