Technical Summary: Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding

1. Problem Statement

Graphical User Interface (GUI) grounding requires Vision-Language Models (VLMs) to identify specific interface elements in high-resolution screenshots and predict precise screen coordinates. This task is particularly challenging due to the small size, visual similarity, and dense arrangement of target elements.

Existing post-training methods face significant limitations in this coordinate-sensitive domain:

• Supervised Fine-Tuning (SFT): Treats annotated coordinates as hard targets, providing little information beyond the final answer and failing to exploit the teacher's uncertainty or "dark knowledge" regarding plausible coordinate tokens.
• Reinforcement Learning (e.g., GRPO): Optimizes task outcomes but relies on sparse rewards and multiple rollouts, offering weak guidance for fine-grained localization.
• Naive On-Policy Self-Distillation (OPSD): While promising for providing dense token-level teacher signals, naive OPSD is ill-suited for GUI grounding. In OPSD, the teacher is queried on prefixes generated by the student. If the student generates an incorrect prefix (a wrong spatial hypothesis), the teacher's subsequent logits may become a plausible continuation of that wrong coordinate rather than a signal toward the true target. This leads to unreliable teacher signals that degrade performance.

2. Methodology: Quality-Aware Self-Distillation

The authors propose Quality-Aware Self-Distillation, a method that calibrates coordinate-token supervision by leveraging the spatial verifiability of GUI grounding. The approach combines two complementary mechanisms: Soft Correctness-Aware Gating and Teacher-Probability Scaling.

2.1 Privileged Information Construction

Following prior work (GUI-SD), the method constructs a privileged teacher input $(x^+, I^+)$ where the target region is visually highlighted and task-irrelevant regions are suppressed via layout-preserving regional masking. The student is conditioned on the original input $(x, I)$, while the teacher is conditioned on the privileged input $(x^+, I^+)$ to generate signals.

2.2 Soft Correctness-Aware Gating

This mechanism addresses the issue of the teacher providing plausible but incorrect continuations based on a flawed student prefix.

• Verification: For a coordinate-token position $t$, the method checks if the teacher's top-predicted coordinate token ($d^*_t$) can still be completed into a valid coordinate within the ground-truth bounding box interval ($B_{a(t)}$) given the current student-generated prefix.
• Binary Indicator: A binary indicator $h_t$ is set to 1 if completion is possible (compatible) and 0 otherwise (incompatible).
• Soft Gating: Instead of discarding incompatible signals entirely (hard gating), the method applies a soft gate $g_t = \alpha + (1-\alpha)h_t$.
  • If compatible ($h_t=1$), $g_t = 1$.
  • If incompatible ($h_t=0$), $g_t = \alpha$ (set to 0.5 in experiments).
  • This down-weights unreliable signals rather than eliminating them, preserving potentially useful corrective information while reducing negative influence.

2.3 Teacher-Probability Scaling

Correctness gating alone does not account for the teacher's confidence. Two signals may both pass the gate, but one may be highly confident while the other is near a decision boundary.

• Scaling Factor: The method uses the teacher's probability of the top coordinate-token prediction, $p_t = P^t_T(d^*_t)$, as a scaling factor.
• Mechanism: Higher $p_t$ (higher confidence) results in a stronger distillation weight, while lower $p_t$ (higher uncertainty) softens the signal. This ensures that even spatially compatible signals are weighted according to their reliability.

2.4 Weighted Reverse-KL Objective

The final loss function combines these factors into a token-level weight $w_t$:
$$w_t = (1 - r_t) + r_t \cdot g_t \cdot \lambda \cdot p_t$$
Where $r_t$ is an indicator for coordinate tokens, $\lambda$ is a scaling coefficient (set to 3), and $g_t$ is the soft gate.

• Non-coordinate tokens are distilled with weight 1.
• Compatible coordinate tokens are weighted by $\lambda p_t$.
• Incompatible coordinate tokens are weighted by $\alpha \lambda p_t$.

3. Key Contributions

1. Quality-Aware Self-Distillation: A novel method for GUI grounding that calibrates teacher signals based on their reliability (spatial compatibility) and certainty (probability), rather than treating all teacher signals as equally trustworthy.
2. Empirical Study of Signal Reliability: The paper utilizes the spatially verifiable nature of GUI grounding to empirically study how unreliable teacher signals should be treated in on-policy self-distillation, demonstrating that a soft down-weighting strategy is superior to hard discarding or blind imitation.
3. Complementary Mechanisms: The authors demonstrate that neither soft gating nor probability scaling alone improves performance; their combination is necessary. Gating suppresses unreliable signals, while scaling calibrates the strength of the remaining reliable signals.

4. Experimental Results

Experiments were conducted on six GUI grounding benchmarks (ScreenSpot-Pro, ScreenSpot-v2, UIEG, OSWorld-G, OSWorld-G-R, and MMG) using Qwen3.5-9B as the backbone.

• Performance: The proposed method achieved a macro-average accuracy of 72.23, outperforming the strongest baseline (GUI-SD) by 2.16 points.
• Comparisons: It significantly outperformed SFT (+4.14 points) and GRPO (+6.37 points), as well as Naive-OPSD and the Vision-PV-Only baseline.
• Ablation Studies:
  • Component Synergy: Using only gating or only scaling resulted in lower performance than the baseline, confirming their complementary roles.
  • Gating Strength: Soft gating ($\alpha=0.5$) outperformed hard gating ($\alpha=0$), suggesting that completely discarding failed-gate signals removes valuable corrective guidance.
  • Scaling Coefficient: A scaling coefficient $\lambda=3$ yielded optimal results, balancing the suppression of unreliable signals with the need for sufficient supervision on reliable tokens.

5. Significance and Claims

The paper claims that spatial verifiability is a critical, underutilized signal for improving teacher-signal reliability in self-distillation. Unlike general token prediction tasks where reliability is inferred through indirect proxies (entropy, perplexity), GUI grounding allows for direct verification of whether a prediction remains feasible given the ground-truth constraints.

The authors argue that their method represents a shift from "blind imitation" to "trustworthy imitation." By softly down-weighting and scaling teacher signals based on their compatibility with the target region and the teacher's confidence, the method mitigates the risk of reinforcing incorrect spatial hypotheses while preserving the dense, fine-grained learning signals that make self-distillation effective. The work suggests that for tasks with verifiable constraints, reliability-aware distillation strategies can significantly enhance model performance without requiring complex reinforcement learning setups.