Technical Summary: VISTA (View-Consistent Self-Verified Training) for GUI Grounding

1. Problem Statement

GUI grounding involves mapping a natural language instruction and a screenshot to a precise click coordinate. While recent work has applied Group Relative Policy Optimization (GRPO) to this task using rule-based "point-in-box" rewards, the authors identify a critical failure mode: reward degeneracy.

In standard GRPO, a group of rollouts is sampled from a single fixed screenshot view. This leads to two problematic scenarios:

1. All-Fail Groups: On difficult instances, all sampled rollouts miss the target, resulting in a group of zero rewards.
2. All-Success Groups: On easy instances, all rollouts hit the target, resulting in a group of full rewards.

In both cases, the group mean ($\mu$) and standard deviation ($\sigma$) collapse, yielding a relative advantage ($A$) of zero for all samples. This eliminates the learning signal. The authors note that in fixed-view GRPO, fewer than 5% of training groups are "informative" (containing a mix of successes and failures). Furthermore, because GUI coordinates are tightly coupled to screenshot geometry, a prediction correct in one view may be incorrect after a crop, even if the target remains visible. Standard GRPO fails to expose these geometric sensitivities because it does not vary the input view within a comparison group.

2. Methodology: VISTA

The paper proposes VISTA (View-Consistent Self-Verified Training), a GRPO-based framework designed to construct informative comparison groups and stabilize coordinate generation. It consists of two primary components:

A. View-Consistent Group Rollout

Instead of sampling multiple completions from a single fixed screenshot, VISTA constructs each GRPO group from multiple target-preserving views of the same GUI instance.

• Target-Preserving Crops: For a given task $(I, q, B)$, the system generates $K$ cropped views. Each crop is guaranteed to fully contain the target bounding box $B$.
• Exact Remapping: The target box coordinates are geometrically remapped to the new cropped coordinate frame.
• Group Construction: The model generates $n$ completions per view (where $K \times n = G$, the total group size).
• Benefit: This ensures that the group contains semantically equivalent inputs (same instruction and target) but geometrically different screenshots. This increases the likelihood of observing mixed outcomes (some hits, some misses) within a single group, thereby restoring non-zero relative advantages.

B. Self-Verified Cross-View Anchor

While multi-view rollouts improve exploration, they can introduce instability in coordinate generation, particularly on ambiguous views. To address this without reverting to unconditional supervised fine-tuning (SFT), VISTA introduces a self-verified anchor.

• Conditional Activation: An oracle answer (the center of the remapped target box) is only introduced as a training signal if the current policy has already produced at least one maximum-reward rollout within the group.
• Baseline Exclusion: The oracle answer is excluded from the calculation of the group mean ($\mu_m$) and standard deviation ($\sigma_m$). It is treated as a separate supervised signal.
• Mechanism:
  • If the group is "self-verified" (contains a successful model rollout), the anchor is activated with a positive advantage derived from the model-only baseline.
  • If the group is "all-fail" (no model rollout succeeds), the anchor is not activated. This prevents the model from learning from an oracle when it has not yet demonstrated the ability to solve the instance, avoiding the collapse of the learning signal into unconditional imitation.

C. Training Objective

The final loss function combines the standard clipped GRPO objective for model-generated rollouts with an advantage-weighted supervised loss for the anchor (when active):
$$\mathcal{L}_{VISTA} = -\frac{1}{G+1} \left[ \sum \ell_{clip} + \lambda_a \ell_{anchor} \right] + \beta \mathcal{L}_{KL}$$
The anchor loss is only computed if the group is self-verified, ensuring the supervised signal stabilizes training only when the policy has already shown competence.

3. Key Contributions

1. View-Consistent Group Rollout: A novel group construction strategy that samples from multiple target-preserving views of the same instance rather than a single fixed view. This significantly increases the fraction of informative groups (from <5% to ~20%) and restores reward variance.
2. Self-Verified Cross-View Anchor: A conditional stabilization mechanism that injects oracle guidance only when the model has already succeeded in at least one view within the group. This prevents the anchor from shifting the GRPO baseline or supervising uninformative "all-fail" groups.
3. Comprehensive Validation: Extensive experiments across five benchmarks (ScreenSpot-Pro, ScreenSpot-V2, MMBench-L2, OSWorld-G-R, OSWorld-G) and multiple model scales (4B to 30B+).

4. Experimental Results

The paper reports consistent improvements across Qwen3-VL backbones (4B, 8B, 30B-A3B) and Qwen3.5-initialized models.

• ScreenSpot-Pro Performance: VISTA significantly improves accuracy on this challenging benchmark, which features small targets and dense interfaces.
  • Qwen3-VL 4B: Improved from 55.5 to 63.4.
  • Qwen3-VL 8B: Improved from 52.7 to 65.8.
  • Qwen3-VL 30B-A3B: Improved from 53.7 to 67.0.
  • Qwen3.5 35B-A3B: Improved from 71.7 to 72.9 (compared to standard GRPO).
• Robustness: VISTA demonstrates higher worst-view accuracy and lower prediction flip rates compared to standard GRPO, indicating better stability under geometric perturbations.
• Ablation Studies:
  • Removing the view-consistent crop (using standard GRPO) drops performance.
  • Removing the self-verified gate (using unconditional oracle injection) severely degrades performance (e.g., dropping ScreenSpot-Pro from 64.3 to 57.8 in one ablation), confirming the necessity of the conditional activation.
  • The combination of both components yields the best results.

5. Significance and Claims

The paper claims that VISTA addresses a fundamental bottleneck in applying GRPO to GUI grounding: the lack of informative relative advantages due to view-invariant reward degeneracy.

• Shift in Design Focus: The authors argue that the key to improving GUI grounding with RL is not just how to reward a click, but how to construct the comparison group. By varying the view geometry while preserving semantics, VISTA forces the policy to learn robust coordinate generation rather than memorizing specific pixel locations.
• Stability without Imitation: The self-verified anchor allows the model to benefit from ground-truth guidance without turning the RL process into unconditional imitation learning, which can suppress exploration.
• Generalizability: The method is shown to be effective across different model scales and backbones, suggesting that view-consistent group construction is a generalizable training signal for GUI grounding tasks.

The authors conclude that VISTA improves both accuracy and robustness, particularly on difficult instances where standard GRPO fails to provide a learning signal. They acknowledge limitations regarding mixed actionable/refusal datasets and the sensitivity of optimization to aggressive cropping, suggesting future work in adaptive scheduling.