Technical Summary: HyperClick – Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning

Problem Statement

Autonomous Graphical User Interface (GUI) agents rely on GUI grounding—the ability to map natural language instructions to precise pixel coordinates on screen elements—to execute user commands. While recent advances in Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) have improved grounding accuracy, a critical reliability gap remains: current models often exhibit overconfidence. Their confidence signals (whether probabilistic token likelihoods or verbalized self-assessments) are poorly aligned with actual grounding correctness. This misalignment leads to agents making high-confidence errors, which is particularly dangerous in real-world automation where a single intermediate error can cause total task failure. Existing methods primarily optimize for localization accuracy but largely overlook whether the model's confidence accurately reflects the quality of its predictions.

Methodology: HyperClick Framework

To address the overconfidence problem, the authors propose HyperClick, a framework that enhances trustworthy GUI grounding through Self-Critiqued Reinforcement Learning (SCRL). Unlike prior approaches that treat grounding as a binary classification task, HyperClick explicitly integrates confidence estimation into the decision-making process.

1. Problem Formulation

The task is formulated as a click formulation, where the policy model predicts a single point $\hat{p} = (\hat{x}, \hat{y})$ representing the center of the target UI element. The model is trained to output both the coordinate and a verbalized confidence score.

2. Confidence Modeling (Spatial Confidence Target)

Instead of relying on token-level probabilities (which can be confounded by tokenization), HyperClick constructs a bounded spatial confidence target based on Gaussian error analysis.

• Target Construction: For a ground-truth bounding box $b$ with center $\mu$, a Gaussian kernel generates a confidence value $C(\hat{p})$ for any predicted point $\hat{p}$.
• Constraints: The confidence is set to 0 if the point falls outside the bounding box. Inside the box, the value decays based on the distance from the center.
• Adaptive Variance: To handle varying element sizes, the standard deviation ($\sigma$) of the Gaussian is scaled by the box dimensions ($\sigma = \alpha \cdot \text{size}$). This ensures smaller elements require higher precision (sharper peaks), while larger elements allow for broader high-confidence regions.

3. Training Objective and Rewards

The policy is optimized using Group Relative Policy Optimization (GRPO) with a composite reward function $R = R_{format} + R_{correctness} + R_{confidence}$:

• Format Reward ($R_{format}$): Enforces a strict output template <point>[x,y]</point><confidence>conf</confidence>, ensuring parseability and preventing overly long completions.
• Correctness Reward ($R_{correctness}$): A binary reward (1 or 0) indicating whether the predicted point falls within the ground-truth bounding box.
• Confidence Alignment Reward ($R_{confidence}$): An $L_2$ constraint that aligns the model's generated verbalized confidence $\hat{c}$ with the spatial confidence target $C(\hat{x}, \hat{y})$. This rewards the model for assigning high confidence to correct, precise clicks and low confidence to incorrect or imprecise ones.

Key Contributions

1. Systematic Analysis of Overconfidence: The authors systematically evaluate existing general-purpose and GUI-specific models on the ScreenSpot-Pro benchmark, revealing that current agents are significantly overconfident in their predictions, with confidence scores far exceeding actual accuracy.
2. HyperClick Framework: They introduce a novel SCRL framework that jointly optimizes grounding accuracy and confidence alignment. By training the model to expose uncertainty alongside its click prediction, HyperClick enables confidence-based abstention, allowing agents to refrain from acting when uncertain.
3. Dual Reward Mechanism: The paper proposes a specific dual-reward mechanism combining binary correctness with a bounded spatial confidence target, demonstrating that this approach effectively reduces high-confidence errors without sacrificing grounding performance.

Experimental Results

The authors evaluated HyperClick on four challenging benchmarks: ScreenSpot, ScreenSpot-V2, ScreenSpot-Pro, and MMBench-GUI.

• Performance: HyperClick (based on Qwen2.5-VL-3B and 7B) maintains competitive or superior grounding accuracy compared to state-of-the-art SFT and RL baselines across mobile, desktop, and web platforms. Notably, it shows significant gains on high-resolution, complex benchmarks like ScreenSpot-Pro.
• Confidence Alignment: Reliability diagrams on ScreenSpot-Pro show a dramatic reduction in Expected Calibration Error (ECE). While baseline models (e.g., UI-TARS, Qwen2.5-VL) exhibit ECEs around 0.48–0.63 (indicating severe overconfidence), HyperClick-7B reduces ECE to 0.249.
• Ablation Studies:
  • The combination of all three reward components (format, correctness, confidence) yields the best results (93.7% accuracy on ScreenSpot-V2), whereas using only confidence alignment leads to reward hacking.
  • The adaptive variance parameter $\alpha$ is critical; $\alpha = 1/4$ provides the optimal trade-off between concentration and spread.
  • The framework generalizes well, improving performance when applied to other baselines like MiMo-VL.

Significance and Claims

The paper claims that trustworthy GUI grounding is a prerequisite for reliable autonomous agents. By moving beyond simple accuracy metrics to include confidence-quality alignment, HyperClick addresses a fundamental limitation in current GUI agents: the inability to distinguish between reliable and unreliable actions.

The authors position this work as a step toward safer GUI automation. By enabling agents to "know what they don't know" through calibrated verbalized confidence, the framework supports selective abstention, preventing error propagation in multi-step tasks. The paper concludes that while the current work focuses on grounding, the principles of confidence alignment are essential for future extensions into GUI planning, where unreliable decision-making can have cascading negative effects. The authors emphasize that their approach is model-agnostic regarding the confidence signal and scalable across different VLM architectures.