🤖 AI

Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation

This paper proposes a reinforcement learning framework for computer-use agents that leverages autonomous vision-language models to generate noisy reward signals, which are then corrected via a noise-aware estimator to significantly improve task success rates across diverse desktop environments.

Original authors: Marta Sumyk, Oleksandr Kosovan

Published 2026-06-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Marta Sumyk, Oleksandr Kosovan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to use a computer. You want the robot to open a spreadsheet, find a specific number, and email it to a boss. This is what researchers call a Computer-Use Agent (CUA).

The big problem is: How do you tell the robot it did a good job?

In video games, the computer knows instantly if you won or lost. But on a real desktop (like Windows or macOS), there is no "Game Over" screen. The robot might click the right button, but the window might not pop up, or it might type the wrong email address. Usually, a human has to look at the screen and say, "Yes, that worked," or "No, try again." But you can't hire a human to watch every single attempt a robot makes; it's too slow and expensive.

The Solution: The "Robot Teacher"

The authors of this paper proposed a clever workaround. Instead of a human, they used a super-smart AI (a Vision-Language Model) to act as a Robot Teacher.

Here is how the system works:

The Student: The computer-use agent tries to do the task.
The Teacher: Once the task is done, the Robot Teacher looks at the final screenshot of the screen and the original instruction. It then gives a simple grade: "Pass" (1) or "Fail" (0).
The Loop: The student uses this grade to learn and try again.

The Catch: The Teacher Makes Mistakes

The problem is that the Robot Teacher isn't perfect. Sometimes it thinks a failed task was a success (a False Positive), and sometimes it thinks a successful task failed (a False Negative).

If you just blindly trust the teacher, the robot learns the wrong lessons.

Analogy: Imagine a coach who accidentally claps and cheers when the player kicks the ball into the wrong goal. The player thinks, "Great job!" and keeps doing it. Eventually, the player gets really good at scoring on the wrong goal.

The Magic Fix: The "Noise-Corrected" Score

The paper's main contribution is a mathematical "correction filter."

The researchers realized they could measure how often the teacher makes mistakes. They ran a test set where they knew the real answers and compared them to what the teacher said. They calculated:

How often does the teacher say "Pass" when it should be "Fail"?
How often does the teacher say "Fail" when it should be "Pass"?

Using these numbers, they created a formula to clean up the teacher's grade before giving it to the robot.

Analogy: If the teacher is known to be 20% too generous, the system automatically subtracts a little bit of "generosity" from every "Pass" grade before the robot sees it. This ensures the robot learns from the truth, not the teacher's bias.

What Happened?

They tested this on three different computer systems: macOS, Windows, and Linux.

Without the fix: The robot learned a little bit, but it was shaky and sometimes got worse at general tasks because it was confused by the teacher's mistakes.
With the fix: The robot improved significantly. On average, its success rate jumped by 12.6% compared to just starting from scratch, and 5.1% better than using the uncorrected teacher.

The Bottom Line

The paper proves that you can teach a robot to use a computer without a human watching every move. You just need a second AI to grade the work, as long as you mathematically correct for that second AI's mistakes.

This makes it possible to train robots on massive amounts of data, making them much better at handling real-world computer tasks on different operating systems.

Technical Summary: Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation

Problem Statement

Computer-Use Agents (CUAs) are designed to autonomously operate graphical user interfaces (GUIs) based on natural language instructions. While recent architectures have shown promise in generalizing across applications and operating systems, they remain unreliable in unconstrained, real-world settings. State-of-the-art agents achieve success rates of only ~60% on the OSWorld benchmark and ~40% on macOSWorld.

A primary bottleneck preventing effective Reinforcement Learning (RL) fine-tuning for CUAs is the absence of scalable, machine-readable reward signals. Unlike robotics simulators or game environments where success criteria are explicit, desktop task objectives are often implicit, visually grounded, and unstructured. Existing solutions rely on brittle, task-specific heuristics (e.g., DOM checks) or costly manual annotations, neither of which scales across diverse applications and operating systems. Furthermore, while Vision-Language Models (VLMs) offer a potential for autonomous evaluation, their outputs are imperfect, containing false positives and false negatives. Naively using these noisy outputs as reward signals introduces bias and destabilizes policy optimization.

Methodology

The authors propose a principled framework for integrating autonomous VLM-based evaluators into the RL loop for CUAs. The core innovation is treating the evaluator not as a source of ground truth, but as a noisy binary reward channel, and deriving a statistically grounded, noise-corrected reward estimator.

1. Autonomous Evaluation Setup

Evaluator: A Vision-Language Model (Qwen2-VL-7B) is used in a zero-shot setting. It receives the final GUI screenshot and the original natural language instruction, outputting a binary signal $\tilde{r}_i \in \{0, 1\}$ indicating task completion.
Noise Characterization: The evaluator's imperfections are modeled via conditional error rates:
- $e_+$ : False Negative rate ( $P(\tilde{r}=0 | r^*=1)$ ).
- $e_-$ : False Positive rate ( $P(\tilde{r}=1 | r^*=0)$ ).
Separability: The framework assumes a separability constant $D = 1 - e_+ - e_- > 0$ , ensuring the evaluator provides informative feedback beyond random guessing.

2. Noise-Corrected Reward Estimator

Since the true reward $r^*$ is unobserved, the authors derive an estimator $\hat{r}$ that is asymptotically unbiased.

Theoretical Derivation: Based on the relationship $E[\tilde{r} | r^*] = e_- + D \cdot r^*$ , the true reward is estimated as:
$\hat{r} = \frac{\tilde{r} - e_-}{D}$
Practical Implementation: In practice, error rates are unknown. The authors estimate $\hat{e}_+$ and $\hat{e}_-$ from a held-out calibration dataset (disjoint from the training set). The plug-in estimator is:
$\hat{r}^\sharp = \frac{\tilde{r} - \hat{e}_-}{\hat{D}}$
where $\hat{D} = 1 - \hat{e}_+ - \hat{e}_-$ . This estimator corrects for the bias introduced by the evaluator's false positives and negatives.

3. Policy Optimization

The framework utilizes Proximal Policy Optimization (PPO).

Reward Structure: Rewards are terminal-only, defined at the end of an episode ( $T$ ).
Objective: The agent maximizes the expected return $J(\theta) = E[\gamma^T \hat{r}_i]$ , where $\hat{r}_i$ is the noise-corrected reward.
Advantage Estimation: The terminal corrected reward is propagated backward through the trajectory to compute advantage estimates, enabling stable policy updates despite the noise in the feedback signal.

4. Experimental Setup

Datasets: A synthetic dataset of 7,560 tasks across 42 applications on macOS, Windows, and Linux was created. Additionally, task descriptions from OmniAct, GUI-World, and GUIDE were incorporated.
Benchmarks: Evaluation was conducted on macOSWorld, Windows Agent Arena, and OSWorld.
Baselines: The study compares zero-shot baselines against PPO fine-tuning using:
1. Raw evaluator rewards ( $\tilde{r}$ ).
2. Corrected reward estimator ( $\hat{r}$ ).
3. Unified (cross-OS) vs. Per-OS model parameterization.

Key Results

Experiments demonstrate that explicitly modeling and correcting evaluator noise significantly improves agent performance compared to using raw evaluator signals.

Performance Gains: The noise-corrected reward estimator combined with per-OS fine-tuning achieved the highest success rates across all benchmarks:
- macOSWorld: Improved from 0.084 (zero-shot) to 0.203.
- Windows Agent Arena: Improved from 0.331 to 0.442.
- OSWorld: Improved from 0.283 to 0.432.
Comparison to Raw Rewards: Using raw evaluator rewards yielded moderate improvements but was consistently outperformed by the corrected estimator. In some cases (e.g., OSWorld with unified models), raw rewards led to performance degradation compared to the zero-shot baseline, highlighting the bias introduced by uncorrected noise.
Generalization: Per-OS models fine-tuned with corrected rewards demonstrated superior generalization compared to unified cross-OS models, suggesting that OS-specific interface conventions are critical for robust learning.
Magnitude of Improvement: The corrected approach improved success rates by an average of 12.6 percentage points over zero-shot performance and 5.1 percentage points over raw evaluator fine-tuning.

Significance and Claims

The paper claims that autonomous evaluation can serve as a practical, scalable reward signal for RL in GUI environments, provided that the inherent noise of the evaluator is explicitly modeled and corrected.

Scalability: The framework eliminates the need for manual success labels or task-specific heuristics, enabling RL fine-tuning on large-scale interaction data across diverse operating systems.
Robustness: By deriving an asymptotically unbiased reward estimator, the method stabilizes policy optimization in the presence of imperfect feedback, addressing a central bottleneck in CUA development.
Practicality: The correction mechanism is a "drop-in replacement" for raw rewards in standard policy-gradient methods like PPO, requiring no fundamental changes to the RL algorithm itself.

The authors conclude that while challenges remain regarding non-stationary error rates and sparse rewards, this approach represents a significant step toward practical RL pipelines for general desktop agents that learn from interaction without human supervision.