Technical Summary: Uncertainty Quantification for Computer-Use Agents

Problem Statement

Computer-use agents transform visual-language model (VLM) predictions into executable GUI actions, specifically single-step click coordinates. Unlike passive classification, an incorrect prediction in this context triggers an unintended action on the host system. Consequently, reliable uncertainty quantification (UQ) is essential for critical deployment mechanisms such as error rejection, calibration, miss-severity ranking, and defining spatial safety regions.

However, existing evidence on post-hoc UQ for these agents is fragmented. Prior studies typically evaluate isolated model-dataset pairs or specific UQ families, leaving it unclear whether UQ method rankings remain stable when the agent architecture, benchmark geometry, or observable interface (e.g., open-weight internals vs. closed-source APIs) changes. The central question addressed is: When does UQ generalize in computer-use agents?

Methodology: The ARGUS Benchmark

The authors introduce ARGUS (Assessing Regime-wise Generalization of Uncertainty Scoring), a unified cross-regime benchmark designed to evaluate the transferability of post-hoc UQ methods.

Experimental Setup

• Regimes: A regime is defined by the combination of an agent, a dataset, and an observable model interface.
• Models:
  • Open-weight Panel: 4 VLM agents (Qwen2.5-VL-7B, Qwen2.5-VL-72B-AWQ, UI-TARS-1.5-7B, POINTS-GUI-G-8B).
  • Closed-source Panel: 3 frontier vendors (GPT-5.4, Claude Sonnet 4.6, Gemini 3.1 Pro) via API only.
• Datasets: 4 single-step GUI-grounding benchmarks (SCREENSPOT-V2, SCREENSPOT-PRO, OSWORLD-G, UI-VISION-EG).
• UQ Methods:
  • Open-weight: 27 methods across 7 families (Logit, Sampling, Hybrid, Density/Probe, Attention, Verbalised, VLM-native). These utilize internal signals like logits, hidden states, and attention maps.
  • Closed-source: A harmonized 8-method subset compatible with API-only interfaces (dropping methods requiring internal states).
• Evaluation Metrics:
  • Error Detection: AUROC (incorrect clicks as positives).
  • Selective Execution: PRR (oracle-normalized rejection quality) and AURC.
  • Calibration: ECE and Brier score (after isotonic regression).
  • Graded Severity: AUSE (miss-only sparsification error on normalized distance).
  • Transfer: Spearman rank correlation ($\rho$) between UQ method rankings across different regimes.
  • Spatial Coverage: Conformal click-disk radius and coverage gap.

Protocol

The benchmark uses a strict 80/20 calibration/test split repeated over 50 seeds. Learned probes and density estimators are trained only on the calibration split. All headline metrics are computed on held-out test records. For closed-source models, methods requiring unavailable internal signals are dropped rather than proxied to ensure fair comparison.

Key Results

1. Selective Transfer of UQ Rankings

The primary finding is that UQ rankings exhibit selective generalization:

• Stable within Fixed Models: Rankings are highly stable across datasets for a fixed model. For example, in the POINTS-GUI-G model, cross-dataset transfer reached a Spearman $\rho = 0.969$. The average transfer across all 120 open-weight pairs was $\rho = 0.705$.
• Unstable across Model Classes: Rankings degrade significantly when moving between model classes (e.g., vanilla VLMs to specialist GUI agents).
• Unstable across Interfaces: Transfer from open-weight models to closed-source APIs is statistically indistinguishable from zero. On the shared 8-method intersection, the average cross-tier transfer was $\rho = +0.08$ (95% CI includes zero). This implies that UQ recommendations for open-weight models cannot be extrapolated to closed-source vendors.

2. Regime-Dependent Reliability

No single UQ family dominates all regimes. The "best" family depends on the specific configuration:

• Open-weight: Density/Probe methods (e.g., SAPLMA, SEP, Mahal-RMD) are the most stable family across models and datasets.
• Closed-source: Verbalised self-assessment (e.g., Verbalised-1S/2S) and Hybrid methods (e.g., CCP) perform best. Notably, Verbalised-1S achieved an AUROC of 0.966 on Gemini for SCREENSPOT-V2, driven by a sharply bimodal score distribution.
• Model Transitions: Moving from vanilla to specialist agents causes Attention, Verbalised, and VLM-native families to lose AUROC across all datasets. In contrast, Density/Probe methods remain relatively stable.

3. Divergence of Objectives

Binary error detection (AUROC) and graded miss-severity ranking (AUSE) often select different top methods. In the open-weight panel, the top method for AUROC and AUSE agreed on only 2 of 16 cells. This indicates that a score good at identifying if an error occurred is not necessarily good at ranking how severe the error is.

4. Spatial Coverage Limitations

Score-level discrimination is insufficient for deployment. While conformal click disks (using plug-in UQ scores) can shrink radii by 40–60% compared to fixed baselines, coverage can degrade under calibration-test or interface mismatches. For instance, closed-source coverage showed systematic under-coverage on certain vendors, suggesting that marginally valid conformal prediction requires careful validation of exchangeability assumptions.

Contributions

1. Cross-Regime Benchmark: ARGUS provides the first unified evaluation of post-hoc UQ for executable GUI grounding across open-weight and API-only interfaces, covering 27 methods and 4 datasets.
2. Ranking-Transfer Analysis: The paper quantifies where UQ method rankings generalize (fixed model, varying dataset) and where they break (changing model class or interface).
3. Harmonized Protocol: A strict protocol for API-only comparison that avoids proxy substitutions for unavailable internal signals, ensuring fair cross-tier evaluation.
4. Regime-Dependent Reliability: Evidence that model class, API observability, and spatial failure geometry fundamentally alter which UQ families are reliable.
5. Deployment Evaluation: Comprehensive assessment of UQ for rejection, calibration, severity ranking, and conformal spatial coverage, accompanied by a released Python package (argus-uq) with per-item records and analysis scripts.

Significance and Claims

The paper claims that uncertainty quality is not an intrinsic property of a UQ method but depends jointly on the method, the model, the dataset geometry, the observable interface, and the deployment objective.

The authors emphasize that "selective transfer" is the governing principle:

• Do not extrapolate: UQ recommendations for open-weight models should not be extrapolated to closed-source vendors; they must be re-ranked on the target calibration split.
• Do not assume universality: A method that ranks errors well for one GUI agent may fail on another.
• Do not rely solely on scores: High AUROC does not guarantee spatial coverage; conformal regions must be validated for coverage gaps.

The work concludes that deployment requires a regime-aware UQ panel (a small set of candidate methods) rather than a single universal score, with final selection validated on the specific target calibration split. The authors release argus-uq to facilitate this regime-aware selection process.