Technical Summary: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision

Problem Statement

Modern Computer-Use Agents (CUAs) require a structured perception of the screen to reliably ground instructions and execute actions. This involves identifying visible UI elements, their locations, and their textual content. However, current state-of-the-art models rely heavily on "sparse" grounding datasets (e.g., Mind2Web, ScreenSpot) that annotate only the single element relevant to a specific task step. This sparse supervision leaves the majority of on-screen elements unlabeled, forcing models to learn shortcuts that may fail to form a complete screen state. Consequently, these models struggle with robustness, generalization to new layouts, and out-of-distribution scenarios. Furthermore, practical deployment often demands low-latency, on-device inference, which is difficult to achieve with massive foundation Vision-Language Models (VLMs).

Methodology

1. Dataset: ScreenParse

The authors introduce ScreenParse, a large-scale dataset designed for complete screen parsing. Unlike prior datasets that offer sparse annotations, ScreenParse provides dense annotations for all visible UI elements across 771,000 web screenshots, totaling 21 million elements.

• Annotation Scope: Each element is annotated with a bounding box, a semantic type from a 55-class taxonomy (covering links, buttons, text, containers, etc.), and associated text.
• Hierarchy: The dataset preserves the Document Object Model (DOM) hierarchy, annotating both atomic elements and semantically meaningful container elements (e.g., navigation bars, cards).
• Generation Pipeline (Webshot): To overcome the cost of human annotation, the authors developed Webshot, a fully automated, scalable pipeline:
  1. Crawling & Rendering: 1 million unique URLs are crawled and rendered using Playwright in a standardized viewport (1440x900).
  2. DOM Extraction: The pipeline extracts the DOM tree, filtering for visible elements and removing degenerated or off-screen artifacts.
  3. VLM Refinement: A VLM (Qwen3-VL-8B) refines the coarse DOM-derived class labels against a 55-class taxonomy.
  4. Quality Filtering: A "VLM-as-a-judge" module scores annotation quality (coverage, false positives, duplication, localization) to filter out low-quality pages.

2. Model: ScreenVLM

Using ScreenParse, the authors train ScreenVLM, a compact, 316M-parameter Vision-Language Model.

• Architecture: It adapts the Granite Docling architecture (based on Idefics3), coupling a SigLIP-2 vision encoder with a lightweight Granite 165M autoregressive decoder.
• Representation (ScreenTag): The model outputs a structured, XML-like sequence called ScreenTag. This format serializes elements as typed tags followed by discretized location tokens (0–500 grid) and optional text, enabling autoregressive decoding of the full screen state.
• Training Objective: To address the issue where standard cross-entropy loss is dominated by text tokens, the authors introduce a structure-aware weighted loss. This loss upweights tokens critical to structure (tags and location coordinates) relative to text tokens, ensuring the model prioritizes accurate layout and typing over mere transcription.

Key Contributions

1. ScreenParse Dataset: A web-scale dataset with dense, complete annotations for 21M UI elements across 55 classes, generated via an automated pipeline.
2. Webshot Pipeline: A scalable, fully automated framework for collecting and refining dense screen parsing annotations without human intervention.
3. ScreenVLM: A compact VLM architecture that achieves high-fidelity dense parsing, outperforming much larger foundation models.
4. Structure-Aware Loss: A novel training objective that improves the fidelity of predicted layouts by prioritizing structural tokens.
5. Transferability Evidence: Demonstration that dense screen supervision provides transferable structural priors that benefit not only ScreenVLM but also existing foundation VLMs and detector-based parsers when fine-tuned on ScreenParse.

Experimental Results

In-Domain Performance (ScreenParse)

ScreenVLM substantially outperforms larger foundation VLMs on dense parsing metrics:

• PageIoU: ScreenVLM achieves 0.606, significantly surpassing Qwen3-VL-8B (0.294) and InternVL3-2B (0.111).
• Label PageIoU: ScreenVLM reaches 0.197, compared to 0.051 for Qwen3-VL-2B.
• Efficiency: ScreenVLM is ~6x smaller (316M params) and ~4x faster in throughput than 2B-scale VLMs, making it suitable for low-latency deployment.

Out-of-Distribution Transfer

• GroundCUA: ScreenVLM maintains strong performance (PageIoU 0.251) on this diverse, multi-platform benchmark, outperforming prompted foundation models (e.g., Qwen3-VL-8B at 0.060).
• ScreenSpot: On sparse grounding benchmarks (Web/PC/Mobile), ScreenVLM shows strong pixel coverage (PixCov), though element-level recall (Recall@50) drops on PC/Mobile splits, indicating a domain gap between web and native UIs.

Impact on Other Models

Fine-tuning existing foundation VLMs (Qwen3-VL-2B, InternVL3-2B) and detector-based parsers (OmniParser, YOLO, RT-DETR) on ScreenParse yields consistent and substantial improvements across all benchmarks. For instance, fine-tuning InternVL3-2B on ScreenParse improves its PageIoU on ScreenParse from 0.111 to 0.509. This suggests that dense screen supervision provides generalizable structural priors for UI understanding.

Ablation Study

The structure-aware weighted loss significantly improves performance. Compared to standard cross-entropy, the structure-aware variant improves PageIoU on ScreenParse by 2.4% and GroundCUA by 11.1%, with the largest gains observed in out-of-distribution scenarios (e.g., ScreenSpot PC Recall improved by 72.1%).

Significance and Claims

The paper argues that treating complete screen parsing as a core training objective is a natural remedy for the limitations of sparse grounding. The authors claim that:

1. Dense supervision is superior: It enables models to form a holistic screen state, which is a prerequisite for robust downstream planning and action execution.
2. Scalability is achievable: The Webshot pipeline demonstrates that high-quality, dense annotations can be generated at scale without human labeling.
3. Transferability is key: The structural priors learned from dense screen parsing are transferable, improving both specialized parsers and general foundation VLMs.
4. Efficiency is critical: Compact models trained on dense data can outperform larger foundation models while being viable for on-device, low-latency applications.

The authors acknowledge limitations, noting that ScreenParse is predominantly web-centric, leaving a domain gap for native desktop/mobile applications, and that residual noise from DOM extraction remains a challenge. Future work is proposed to expand dense parsing to native UIs and leverage these models as visual backbones for vision-language-action agents.