Technical Summary: Learning GUI Grounding with Spatial Reasoning from Visual Feedback

Problem Statement

Graphical User Interface (GUI) grounding is the task of mapping a natural language instruction to a specific pixel coordinate on a screen to execute an action (e.g., a click). While recent Vision-Language Models (VLMs) have advanced, they frequently fail to predict accurate numerical coordinates on high-resolution GUIs with complex layouts. This failure stems from a lack of spatial semantic alignment: the model struggles to implicitly map its understanding of a visual element to discrete coordinate tokens.

Existing approaches typically frame GUI grounding as a one-step prediction problem, where the model generates coordinates directly from a screenshot. A critical limitation of this paradigm is the absence of a visual feedback loop. The model is supervised only on the final numerical output and receives no information regarding where its prediction actually lands on the screen. Consequently, the model fails to develop a robust alignment between its numerical predictions and their corresponding on-screen positions, leading to poor spatial reasoning capabilities.

Methodology: GUI-CURSOR

To address these limitations, the authors propose GUI-CURSOR, which reframes GUI grounding from a static, one-step task into a dynamic, interactive search task.

1. Interactive Search Environment

Instead of predicting a final coordinate immediately, the model controls a virtual cursor on the GUI image. The process operates as a sequence of steps:

• Observation: At each step $t$, the model observes the GUI screenshot with the cursor at its current position $(x_t, y_t)$.
• Reasoning & Action: The model generates a response containing a thinking process (analyzing the target, current cursor location, and spatial relationships) followed by an action. The action is either a new coordinate $(x_{t+1}, y_{t+1})$ to move the cursor or a STOP command if the cursor is correctly positioned.
• Visual Feedback: The environment renders the cursor at the new position and provides an updated screenshot for the next step. This explicit visual feedback allows the model to verify its predictions and incrementally refine its spatial understanding.

2. Training with Reinforcement Learning

The model is trained using Group Relative Policy Optimization (GRPO) within this interactive environment. The training objective relies on a custom trajectory-based reward function ($R_T$) designed to guide both the final accuracy and the search behavior.

The reward function consists of:

• Position Reward ($r_p$): A dense reward based on the distance between the final cursor position and the target bounding box, considering both proximity to the edge and centrality within the box.
• Trajectory Penalties: To prevent degenerate strategies (e.g., stopping immediately without moving, moving back and forth, or moving away from the target), the authors introduce four specific penalties:
  • False Stop Penalty ($r_{FS}$): Penalizes stopping when the cursor is outside the target.
  • False Move Penalty ($r_{FM}$): Penalizes continuing to move after the cursor has already entered the target box.
  • False Direction Penalty ($r_{FD}$): Penalizes movements that increase the distance to the target compared to the initial prediction.
  • Repeated Position Penalty ($r_{RP}$): Penalizes predicting the same coordinate more than once.

3. Cursor-Centric Focusing (CCF)

To handle high-resolution images efficiently without incurring prohibitive computational costs, the authors introduce a two-stage strategy:

• Training: Images are downsampled to a manageable resolution (e.g., 1920×1080).
• Inference: For images larger than the training resolution, the model first performs a coarse prediction on the full image. It then crops a region centered on this prediction (maintaining the training resolution) and performs fine-grained movement within this focused view. This balances training efficiency with inference accuracy.

Key Contributions

1. Reformulation of GUI Grounding: The paper shifts the paradigm from static coordinate prediction to a dynamic, interactive process where the model learns coordinate-spatial alignment through visual feedback.
2. GUI-CURSOR Model: The introduction of a GUI grounding model trained via reinforcement learning in an interactive environment, utilizing a dense, trajectory-aware reward function that enforces rational search behaviors.
3. Enhanced Spatial Reasoning: Comprehensive evaluation demonstrates that learning GUI grounding by moving a cursor improves not only grounding accuracy but also general spatial reasoning capabilities, which generalize to out-of-distribution domains.

Experimental Results

The authors evaluated GUI-CURSOR on multiple benchmarks, including ScreenSpot-Pro, OSWorld-G, UI-Vision, and ScreenSpot-v2, using base models Qwen2.5-VL-7B and UI-TARS-1.5-7B.

• Grounding Performance: GUI-CURSOR consistently outperformed strong baselines (including supervised fine-tuning methods like SeeClick, UGround, and RL-based methods like GTA1). Notably, on the challenging ScreenSpot-Pro benchmark, GUI-CURSOR (based on UI-TARS-1.5-7B) surpassed the leading model GTA1 by 8.1%, despite using significantly fewer training samples (8K vs. 64K).
• Agentic Tasks: In online agentic evaluations on OSWorld, GUI-CURSOR achieved higher success rates with fewer action steps compared to GTA1, demonstrating effectiveness in realistic, diverse scenarios.
• Spatial Reasoning:
  • Cursor-in-Box Test: A custom test revealed that standard VLMs (even fine-tuned ones) struggle with simple spatial classification (determining if a cursor is inside a box) and exhibit strong center biases. GUI-CURSOR, without explicit training on this classification task, achieved significantly higher F1 scores, suggesting its grounding ability is founded on more robust spatial understanding.
  • Generalization: On benchmarks like SPHERE and SpatialMQA, GUI-CURSOR showed improvements in reasoning and combining skills, indicating that the interactive training paradigm enhances spatial reasoning capabilities beyond the GUI domain.
• Efficiency: The model learns to adaptively conduct more steps only on difficult examples (e.g., small targets), solving 95% of instances within two steps.

Significance and Claims

The paper claims that the primary significance of this work lies in demonstrating that visual feedback loops are essential for aligning numerical predictions with spatial reality in VLMs. By reframing grounding as a search task, the model learns to "think" about spatial relationships and correct its trajectory, rather than relying on a single-shot guess.

The authors assert that this approach:

1. Solves the spatial semantic alignment problem inherent in one-step prediction.
2. Improves data efficiency, achieving state-of-the-art results with fewer training samples.
3. Cultivates a more robust spatial reasoning capability that generalizes to out-of-distribution tasks, suggesting that the ability to interactively search for targets is a fundamental component of spatial understanding in multimodal agents.

The work concludes that learning GUI grounding by moving a cursor is a superior strategy for developing agents that can effectively navigate and interact with complex digital interfaces.