Technical Summary: Covering Human Action Space for Computer Use: Data Synthesis and Benchmark

1. Problem Statement

Computer-Using Agents (CUAs) aim to automate on-screen work, yet their reliability on complex, low-frequency interactions remains poor, limiting user trust. Analysis of failure cases in advanced models (e.g., GPT-5.4) reveals a "long-tail" pattern in GUI operations: while simple clicks are handled well, a disproportionate share of task failures stems from complex, diverse interactions such as editing tables, manipulating text, drawing on canvases, and annotating images.

The paper identifies two primary bottlenecks hindering the deployment of GUI-based CUAs:

1. Lack of Benchmarks: Existing benchmarks (e.g., ScreenSpot-Pro, UI-Vision) are predominantly "click-centric" and "widget-centric," focusing on standard GUI elements. They fail to evaluate complex interactions like dragging, freehand drawing, or operations on non-widget modalities (tables, natural images).
2. Data Scarcity: There is a significant shortage of large-scale training data for these complex interactions. Current data pipelines largely inherit web-crawl and accessibility-tree paradigms, which produce click labels for widgets but lack the diversity required for generalizable grounding capabilities.

2. Methodology

A. The CUActSpot Benchmark

To address the evaluation gap, the authors introduce CUActSpot, a benchmark designed to reflect real-world computer-use scenarios beyond simple clicking.

• Modalities: It covers five distinct modalities: GUI (standard widgets), Text (selection, insertion), Table (spreadsheet operations, cell resizing), Canvas (graphic object manipulation), and Natural Image (object cutout, boundary drawing).
• Action Types: It includes click (1 point), drag (2 points), and draw (N points), covering both ordered (e.g., dragging along an arrow) and unordered actions.
• Evaluation Metrics: The benchmark employs a rigorous rule-based evaluation system involving:
  • Correct Regions: Predicted coordinates must fall within these areas.
  • Banned Regions: Predictions falling here are immediately marked incorrect to prevent metric gaming.
  • Rank Attributes: Used for order-sensitive actions (e.g., dragging from start to end).
• Construction: The benchmark consists of 206 manually curated, diverse samples across 33 detailed task types, designed to minimize reliance on specialized software knowledge and focus on grounding capability.

B. Renderer-Based Data Synthesis Pipeline

To solve the data scarcity issue, the authors propose a fully synthetic data generation pipeline that generates 50 million samples.

• Rendering: For each modality, code-based tools render screenshots and extract precise coordinate metadata (bounding boxes, control points, vertices).
  • GUI/Tables: Rendered via web tools (Playwright, HTML/CSS generation) with randomized styles and structures.
  • Text/Canvas: Rendered using Python graphics libraries (PyQt5, Matplotlib) with randomized fonts, shapes, and layouts.
  • Natural Images: Derived from the SAM dataset, with regions annotated using contour extraction algorithms.
• Instruction Synthesis: An advanced LLM (OpenAI o3) is prompted to select salient elements from the rendered metadata, perform intermediate coordinate calculations, and generate complex operation instructions and corresponding action traces (PyAutoGUI code).
• Diversity: The pipeline ensures high diversity in visual styles, task structures, and interaction types, avoiding the bias of single-modality scaling.

C. Model Training

The authors train Phi-Ground-Any-4B, a 4-billion parameter Vision-Language Model (VLM) based on Phi-3.5-VL.

• Training Strategy: The model is pre-trained/mid-trained on the 50M synthetic samples combined with OpenCUA data.
• Ablation Studies: Experiments investigate the impact of data composition, specifically comparing scaling a single modality versus increasing modality and task diversity.

3. Key Contributions

1. CUActSpot Benchmark: A new evaluation standard covering five modalities and complex action types (drag, draw) that better reflects real-world CUA requirements than prior click-centric benchmarks.
2. Data Synthesis Pipeline: A scalable, renderer-based framework that generates 50M high-quality training samples for complex GUI interactions without relying on manual annotation.
3. Phi-Ground-Any-4B: An open-source model that achieves state-of-the-art performance among models with fewer than 32B parameters on complex interaction benchmarks.
4. Empirical Insights: The discovery of "Variety Scaling," demonstrating that increasing the diversity of task types and modalities is more effective for general interactive capability than simply scaling the volume of data within a single modality.

4. Experimental Results

• Benchmark Performance: On CUActSpot, Phi-Ground-Any-4B outperforms all open-source models smaller than 32B parameters. It achieves an overall score of 44.4%, significantly surpassing models like OpenCUA-7B (39.8%) and UI-TARS-1.5-7B (28.5%).
• Comparison with Commercial Models: While GPT-5.4 achieves a higher score (63.6%) on CUActSpot, Phi-Ground-Any-4B remains competitive among open-source solutions.
• Generalization: The model shows strong cross-task generalization, successfully completing detailed tasks not explicitly present in the training set (e.g., editing text within visual content).
• Ablation Findings:
  • Variety Scaling: Adding diverse modalities (Text, Table, Canvas, Image) to the training set consistently improves performance across all benchmarks, whereas scaling a single modality yields diminishing returns.
  • Benchmark Sensitivity: Models fine-tuned on specific software data (e.g., Phi-Ground's application data) show gains on traditional benchmarks (ScreenSpot-Pro, UI-Vision) but often suffer performance drops on CUActSpot, highlighting the distribution shift and the limitations of current benchmarks in measuring general grounding ability.
• OSWorld Evaluation: When used as a grounder in an end-to-end agent setup (with GPT-5.4 as the planner), Phi-Ground-Any-4B demonstrates robust drag-and-drop capabilities in complex spreadsheet tasks, outperforming larger models in specific grounding scenarios.

5. Significance and Claims

The paper claims that the current "widget-and-click" prior in GUI grounding research is insufficient for the next generation of Computer-Using Agents. By introducing CUActSpot and a scalable synthesis pipeline, the authors provide the community with the necessary tools to evaluate and train models on the "long-tail" of human computer interactions.

The authors emphasize that task diversity is a critical driver for learning generalizable grounding capabilities, potentially more so than raw data scale. They position their work as a diagnostic step toward more robust CUAs, acknowledging that while synthetic data enables broad coverage, aligning with real-world distributions and handling long-horizon, stateful scenarios remain open challenges for future work. The release of the benchmark, data, code, and models aims to facilitate further research in this direction.