Technical Summary: Scaling Agents for Computer Use via Behavior Judge

Problem Statement

Computer-use agents (CUAs) hold significant promise for automating everyday digital tasks across operating systems. However, their performance on long-horizon, complex problems remains unreliable. The core challenge lies not merely in solving individual steps but in sustaining correctness across dozens or hundreds of interactions. Small errors compound over time, feedback is often delayed, and environmental noise (such as UI changes or latency) leads to high variance in outcomes.

While prior work has attempted to scale performance within a single rollout (e.g., step-wise Best-of-N), these approaches have yielded limited gains on long-horizon tasks. An alternative is "wide scaling"—generating multiple rollouts in parallel and selecting the best one. However, this approach faces a fundamental bottleneck: evaluation. Agent trajectories are information-dense, multimodal, and often admit multiple valid solutions. Reliably determining which of several long-horizon trajectories is correct is difficult because most details are irrelevant to task success, and scripted automatic evaluation struggles with the nuance of computer-use tasks.

Methodology: Behavior Judge (BJudge)

To address the evaluation bottleneck and enable effective wide scaling, the authors introduce Behavior Judge (BJudge). This framework transforms dense, raw agent trajectories into compact, interpretable representations to facilitate principled selection among candidates.

1. The Wide Scaling Paradigm

Unlike single-rollout scaling, BJudge generates multiple candidate trajectories ($\tau$) using a set of base policies $\{\pi_m\}$. These policies can include different models or stochastic decoding from the same model to capture diversity. The objective is to select the trajectory $\hat{\tau}$ that maximizes the task reward $R(\tau, I)$.

2. Behavior Narrative Generation

The first key component is the Behavior Narrative Generator. Instead of feeding raw screenshots and actions directly to a judge, the system converts each transition $(s_i, a_i, s_{i+1})$ into a "behavior narrative" ($\phi_i$).

• Fact Extraction: A Vision-Language Model (VLM) analyzes the state before and after an action to derive facts describing what the agent actually did and how it affected the environment.
• Visual Augmentation: To ensure accuracy, particularly for pointer interactions (clicks, drags), the generator overlays markers on the pre-action screenshot and extracts zoomed crops from the post-action screenshot. This helps verify pixel-level changes and reduces hallucinations regarding whether an action succeeded.
• Narrative Construction: The final narrative $\tilde{\tau}$ consists of the initial state, the sequence of derived facts, and the final state. This filters out irrelevant details while preserving task-relevant action-effect summaries.

3. Comparative Behavior Evaluator

The second component is the Comparative Behavior Evaluator. Rather than ranking trajectories independently, this module performs a direct comparison of the behavior narratives.

• Mechanism: A VLM evaluator is prompted to compare all candidate narratives simultaneously using a single-round multiple-choice question (MCQ) format.
• Advantage: This approach allows the judge to cite and contrast specific facts between candidates, enabling a more informed selection than independent ranking. It is designed to be token-efficient and faster than multi-round tournament comparisons.

4. Improved Agentic Framework (Agent S3)

To support BJudge, the authors introduce Agent S3, an improved baseline framework built upon Agent S2. Key architectural changes include:

• Flat Policy: Removal of hierarchical planning in favor of a flat policy that can replan at any time based on the current instruction and history, leveraging the strong context-maintenance capabilities of modern foundation models.
• Integrated Coding Agent: A specialized coding agent is natively integrated into the GUI agent's action space. It can be invoked for programmatic edits (e.g., bulk file operations, structured parsing) when more efficient than direct GUI manipulation. It operates in a bounded inner loop with a sandboxed VM, providing execution feedback to the GUI policy.

Key Contributions

1. Wide Scaling Paradigm: The paper establishes that carefully selecting from multiple parallel trajectories substantially improves robustness and coverage for CUAs, provided an effective selection mechanism exists.
2. Behavior Judge Framework: A novel system that converts dense trajectories into compact behavior narratives and uses comparative evaluation to select the optimal solution, addressing the evaluation bottleneck of wide scaling.
3. New State-of-the-Art (SoTA): By combining BJudge with the Agent S3 baseline, the authors achieve a new SoTA on the OSWorld benchmark.
4. Generalization: The method demonstrates strong zero-shot generalization to different operating systems (Windows and Android).

Experimental Results

OSWorld Performance

On the OSWorld benchmark (361 tasks, 100-step limit), the proposed method achieves a 72.6% success rate.

• This represents a 9.2% absolute improvement over the previous SoTA (63.4%).
• It surpasses the reported human-level performance of 72.36%.
• Using an ensemble of GPT-5 and Opus 4.5, the system shows that effective scaling can push agent capabilities beyond human benchmarks on these tasks.

Efficiency and Ablation

• Agent S3 Efficiency: Compared to the hierarchical Agent S2, Agent S3 reduces LLM calls per task by 52.3% and task completion time by 62.4%, while improving the success rate by 13.8%.
• Representation Ablation: Behavior narratives significantly outperform other representations. Compared to a "Screenshot Only" baseline (56.0%) and a "Trajectory Summary" baseline (55.0%), Behavior Narratives achieved a 60.2% success rate (with GPT-5 Mini), a 3.4% absolute gain.
• Selection Strategy: BJudge outperforms adapted independent ranking methods (like WebJudge) across varying rollout budgets, demonstrating that comparative evaluation is necessary for scalable selection.

Generalization

• WindowsAgentArena: BJudge (N=3) achieved a 6.4% improvement over the Agent S3 baseline (56.6% vs 50.2% at 100 steps).
• AndroidWorld: BJudge (N=3) achieved a 3.5% improvement over the Agent S3 baseline (71.6% vs 68.1%).

Model Ensembling

Experiments with mixture-of-models ensembles revealed that diversity is key to task coverage. The combination of GPT-5 and Claude Opus 4.5 yielded the highest success rate (71.6%), while the most diverse mixture (including all tested models) achieved the highest task coverage (Pass@N) at 80.5%.

Significance and Claims

The paper claims that the primary barrier to scaling computer-use agents is not the generation of solutions, but the evaluation and selection of those solutions. By introducing Behavior Judge, the authors provide a practical framework that enables wide scaling to be effective.

The authors emphasize that effective scaling requires structured trajectory understanding and selection. Simply generating more rollouts without a robust mechanism to compare them yields diminishing returns. BJudge solves this by abstracting raw, noisy trajectories into high-level behavior narratives, allowing the system to identify the correct path even in complex, long-horizon environments.

The results highlight that with the right framework, CUAs can not only match but surpass human performance on complex desktop automation tasks, suggesting a viable path toward reliable, automated digital workflows. The paper notes that these results rely on controlled environments (virtual machines) where independent rollouts are possible, acknowledging that direct deployment on live user desktops presents additional challenges regarding state isolation.