Technical Summary: How Benchmarks Mis-Score Computer-Use Agents

1. Problem Statement

Computer-use agents (CUAs) are increasingly deployed to navigate web interfaces and operate desktop software. However, their evaluation relies heavily on brittle, scripted oracles that inspect final states, URLs, or DOM fields. This creates a "deployment–measurement gap," particularly as tasks evolve from short episodes to long-horizon, cross-application workflows.

The core problem identified is that a benchmark score is not a direct observation of capability but the output of a complex pipeline. This pipeline is susceptible to four distinct failure modes that distort evaluation:

1. Task Construction: Stale tasks, leaked instructions, or broken environments (e.g., expired credentials, drift in live sites).
2. Trajectory Observation: Incomplete logs that omit decisive visual evidence (screenshots, tool responses).
3. Scoring: Evaluators that reject valid alternative solutions or reward superficial correctness achieved via invalid shortcuts.
4. Reporting: Aggregate success rates that obscure the specific causes of failure, preventing targeted engineering improvements.

Collapsing these distinct issues into a single "agent failure" category leads to mis-ranked systems and misdirected engineering efforts.

2. Methodology

The authors propose a four-stage reliability framework for CUA evaluation: Task Construction, Execution Environment, Scoring, and Reporting. To validate this framework, they conducted an empirical audit of 150 public failure-scored trajectories from five benchmarks spanning web navigation, enterprise workflows, and desktop control (OSWorld, WebArena, VisualWebArena, WorkArena, and AssistantBench).

Audit Procedure

• Sampling: Trajectories were selected where the recorded cumulative reward was zero (FAIL).
• Labeling: Two vision-enabled LLM judges (GPT-5.5 and Claude Sonnet) independently reviewed the full trajectories, including reasoning, actions, and screenshots. Their judgments were anchored by two human groups who reviewed a subset of the data blindly.
• Decision Logic:
  • Stage 1 (Verdict Validity): Determined if a "FAIL" was a genuine agent failure, an evaluator false negative (the agent succeeded but was rejected), a broken task (environment/specification issues), or unclear.
  • Stage 2 (Diagnosis): For genuine failures, a three-tier diagnostic taxonomy (adapted from MAST) was applied to localize the earliest decisive failure:
    • Tier 1 (Planning/Specification): Violations, loops, or hallucinated features.
    • Tier 2 (Execution/Grounding): Incorrect coordinates, state loss, or tool argument errors.
    • Tier 3 (Verification/Feedback): Premature termination, missed verification, or feedback-blind repetition (no-ops).

3. Key Results

Verdict Reliability

The audit revealed that 15.3% of recorded FAIL verdicts were incorrect.

• 10.7% were evaluator false negatives: The agent completed the task, but the script or checker rejected valid alternatives (e.g., WebArena's string matching rejecting correct semantic answers).
• 4.7% were broken tasks: The task was unsatisfiable due to environment issues (e.g., dead search engines, CAPTCHA walls, or empty system files in the VM).
• 3.3% remained unclear due to insufficient released evidence.

Failure Diagnostics

Among the 122 genuine agent failures, the scalar success rate concealed a skewed distribution of error types:

• Tier 3 (Verification/Feedback): 39.3% of failures. The dominant subcategory was feedback-blind repetition (29.5%), where agents repeated actions despite the screen state remaining unchanged.
• Tier 1 (Planning/Specification): 35.2% of failures, driven by specification violations and planning loops.
• Tier 2 (Execution/Grounding): Only 13.9% of failures.
• Observability Impact: Removing screenshots from the audit data significantly reduced the ability to detect false negatives and lowered agreement between judges, proving that screenshot-inclusive trajectories are necessary for valid auditing.

4. Design Guidelines and Contributions

Based on these findings, the paper derives stage-specific design rules for reliable CUA benchmarks:

• Task Construction: Move toward "living task pools" with procedural refresh to prevent saturation. Tasks must be authored by domain experts to ensure professional validity, and tasks/environments/oracles must be versioned together.
• Environment: The execution stack (OS, app versions, scaffolds) must be pinned and disclosed to ensure reproducibility. Retrieval boundaries must be explicitly declared to prevent contamination.
• Scoring: Checkers must test exactly what the task statement declares, accepting all valid solutions within the defined space. Scoring should shift from outcome-only checks to calibrated, rubric-based judges that evaluate the process.
• Reporting: Benchmarks must release evidence-complete trajectories (screenshots, tool I/O, timestamps) alongside scores. Reports should include per-failure locus and process-stage diagnoses, not just aggregate success rates. Additionally, latency (deadline-miss rates) and cost (dollars per verified success) should be treated as first-class metrics.

5. Significance and Scope

The paper argues that a CUA failure verdict warrants distrust when replay evidence contradicts the checker or reveals invalid task conditions. It posits that the current reliance on scalar success rates obscures the true nature of agent limitations, particularly in verification and planning.

Scope and Limitations:

• The empirical claims are restricted to GUI computer-use evaluation across web, desktop, and enterprise workflows.
• The audit covers 150 trajectories; while the sample size is modest, the wide confidence intervals still indicate that evaluator errors occur at a material rate.
• The study focuses on false negatives (missed successes) and does not estimate false positives (incorrectly passing agents), as it sampled only failure-scored trajectories.
• The paper does not propose new agent architectures but rather calls for a reform in how evaluation infrastructure is built and interpreted to support evidence-based governance.