💻 computer science

Agentic Reward Modeling: Verifying GUI Agent via Progressive Trajectory-Grounded Interaction

This paper introduces VAGEN, a framework that employs a tool-augmented verifier agent with a progressive, surface-to-latent verification mechanism to overcome the limitations of existing passive and over-probing reward modeling methods, thereby significantly improving the accuracy and efficiency of GUI agent evaluation on benchmarks like OSWorld-Verified and AndroidWorld.

Original authors: Chaoqun Cui, Jing Huang, Shijing Wang, Liming Zheng, Qingchao Kong, Zhixiong Zeng

Published 2026-07-28

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Chaoqun Cui, Jing Huang, Shijing Wang, Liming Zheng, Qingchao Kong, Zhixiong Zeng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to use a computer. You want it to open apps, click buttons, and finish tasks like buying a book or organizing files. To teach the robot, you need a way to tell it when it did a good job and when it messed up. This is the world of Reinforcement Learning, where an AI learns by trying things and getting "rewards" for success. But here's the tricky part: how do you know if the robot actually finished the job? Did it really buy the book, or did it just click a button that looked like a buy button? This is the problem of Reward Modeling. If the robot thinks it succeeded when it didn't, it will keep making the same mistake. If it thinks it failed when it actually succeeded, it will get confused and stop trying. Getting this "score" right is the difference between a clumsy robot and a helpful assistant.

For a long time, scientists tried two main ways to grade the robot. The first was like a strict teacher with a checklist: "Did the file appear? Yes. Did the window close? Yes." This works for simple tasks but breaks down when the job is creative or open-ended. The second way was to ask a super-smart AI (a Large Language Model) to look at a video of the robot's work and guess if it succeeded. This is scalable, but the AI is passive; it can only look at what it sees on the screen. It can't peek inside the computer's "brain" to see if a file was actually saved in the background. It's like a teacher grading a math test by only looking at the student's final answer sheet, without checking if the student actually did the work or if the answer was a lucky guess.

This paper introduces a new, smarter way to grade these computer-using robots. The authors, Chaoqun Cui and colleagues, propose a system called VAGEN. Instead of just watching a video or checking a checklist, VAGEN uses a "Verifier Agent"—a second AI detective that can actively investigate. Think of it like a detective who doesn't just read the suspect's alibi (the robot's video) but also has the power to go into the crime scene, check the hidden files, and run tests to see if the story adds up. The authors found that this active, investigative approach is much better at spotting the truth, especially when the robot's actions leave clues that aren't visible on the screen. They showed that this method is not only more accurate but also efficient, proving that sometimes you need to get your hands dirty to know if a job is truly done.

The Detective in the Digital World

So, how does this new detective, VAGEN, actually work? The authors realized that checking if a robot finished a task is often easier than actually doing the task itself. It's like the difference between baking a cake and checking if the cake is done. The baker (the "Actor" robot) has to mix ingredients, watch the oven, and decorate the cake. The judge (the "Verifier") just needs to poke the cake with a toothpick or smell the kitchen to know if it's ready. The authors call this the "easy to verify, hard to solve" property.

To make this happen, VAGEN uses a Progressive Verification Mechanism. Imagine the verifier is a detective solving a mystery, and they have a specific strategy to avoid wasting time. They don't jump straight to breaking down doors; they start with the easiest clues and only get more aggressive if necessary.

Stage 1: The Quick Glance (Static Assessment)
First, the verifier looks at the final picture of the computer screen and a summary of what the robot did. It asks, "Does this look like a win?" If the screen clearly shows a "Order Confirmed" message, the detective says, "Case closed!" and moves on. This is fast and cheap.

Stage 2: Rewinding the Tape (Visual Retrospection)
If the final picture is confusing—maybe the screen is blank, or the message is hidden—the detective doesn't give up. Instead, they rewind the video. They look at screenshots from earlier steps to find clues. Maybe the robot clicked the right button five minutes ago, even if the final screen is messy. This is like checking the security camera footage to see if the suspect actually entered the building.

Stage 3: Breaking and Entering (Proactive Probing)
Sometimes, the clues aren't on the screen at all. Maybe the robot was supposed to save a file to a hard drive, but the screen just shows a generic "Done" message. The screen is lying, or at least, it's hiding the truth. This is where VAGEN gets really cool. The verifier agent has special tools to actively interact with the computer. It can run code, check the file system, or even click buttons itself to test if the file is really there. It's like the detective finally getting a warrant to search the suspect's house. They don't just trust the story; they go and check the basement.

The paper shows that this "surface-to-latent" approach (going from the surface of the screen to the hidden depths of the system) is a game-changer. The authors tested VAGEN on two big sets of tasks: OSWorld-Verified (desktop computers) and AndroidWorld (mobile phones).

The Results: Smarter and Faster

When the authors ran the numbers, VAGEN didn't just do okay; it crushed the competition. On the desktop benchmark, while other methods struggled to get above 80% accuracy, VAGEN hit 92.9% accuracy when judged by human experts. Even more impressively, it did this without needing to check every single screenshot or run endless tests. It was smart enough to stop early when it found the answer.

The paper also highlights a crucial finding: Active interaction is necessary, but not always the first step. Previous methods that relied heavily on "probing" (checking the system) were often slow and inefficient because they didn't look at the video evidence first. They were like detectives who immediately break down the door without even looking at the front door first. VAGEN, however, uses the video evidence to guide its investigation. It only breaks into the system when the video isn't enough. This balance made it much more efficient, using fewer computer resources (like "steps" or "tokens") to get a better answer.

The authors also tested if they could make the verifier even better by asking it to check the same task multiple times (a strategy called "scaling"). They found that even when they restricted the verifier to only "read" data and not change anything, it got even smarter. This suggests that the verifier's main job is to investigate, not to mess with the computer.

Why This Matters

The big takeaway here is that we don't have to choose between a lazy observer and a busybody detective. We can have both. By combining the passive observation of the robot's video with the active power to check the computer's hidden states, VAGEN creates a reward system that is both reliable and efficient.

The authors argue that this is the future of training AI agents. If we want robots that can truly help us with our computers and phones, we need a way to grade them that doesn't get fooled by a pretty picture. VAGEN suggests that by giving our AI graders the tools to dig deeper, we can teach our robots to be much more competent. The paper doesn't claim to have solved every problem in the world, but it does show a clear path forward: stop just watching the show, and start checking the backstage.

Technical Summary: Agentic Reward Modeling via VAGEN

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) is a critical framework for advancing Graphical User Interface (GUI) agents. However, existing reward modeling paradigms face complementary limitations that hinder scalable and accurate training:

Rule-Based Methods: While accurate for specific tasks, they rely on manually engineered scripts (e.g., unit tests) that lack scalability. They fail to handle open-ended tasks or tasks with multiple valid solutions and require prohibitive human effort.
LLM-as-a-Judge (Passive Verification): These methods leverage Large Language Models (LLMs) to analyze task trajectories (screenshots and reasoning text). While scalable, they are inherently passive, unable to revisit specific intermediate states autonomously. More critically, they suffer from partial state observability; they rely on visual evidence which may miss latent system states (e.g., file system changes, background processes) crucial for determining task success.
Active Environment Interaction: Recent approaches introduce a verifier agent to probe the environment, mitigating observability issues. However, these methods often over-rely on probing, ignoring direct evidence already present in the trajectory. This leads to inefficiency, as the verifier may perform redundant interactions even when the trajectory contains sufficient clues.

The core challenge is to unify the efficiency of trajectory analysis with the rigor of active environment interaction without incurring excessive computational costs.

2. Methodology: The VAGEN Framework

The authors propose VAGEN (Verification via Agentic Trajectory-Grounded Environment Interaction), a framework that employs a tool-augmented verifier agent governed by a Progressive Verification Mechanism.

2.1 Core Philosophy

VAGEN operates on a "surface-to-latent" and "cheap-to-expensive" design philosophy. It prioritizes extracting direct evidence from the recorded trajectory before resorting to costly environment interactions. This is underpinned by the observation that GUI tasks often follow an "easy to verify, hard to solve" property: a verifier agent can often confirm task completion with fewer steps and higher success rates than the actor agent required to solve the task.

2.2 Key Components

Tool-Augmented Verifier Agent ( $\pi_e$ ):
The verifier is equipped with four essential tools:
- Check Screenshot: Retrieves specific screenshots from the actor's trajectory for visual inspection.
- Execute Shell: Runs shell commands to verify latent system states (e.g., file attributes).
- Execute Python: Executes Python code for complex logical checks.
- Computer Use: Directly interacts with the GUI environment (sharing the actor's action space) to probe for signals.
- Note: In mobile environments (AndroidWorld), shell and Python tools are excluded due to interface limitations.
Trajectory Memory Consolidation:
To reduce noise, an LLM summarizes the actor's trajectory, filtering out subjective reasoning (which may contain hallucinations) and retaining only factual state observations and atomic actions. This creates a concise operation history $H$ for the verifier.
Progressive Verification Mechanism:
The verification process unfolds in three sequential stages, allowing for early stopping if sufficient evidence is found:
- Stage 1: Static Assessment: The agent evaluates the terminal screenshot and operation summary without tools. If clear evidence exists, it outputs a reward and confidence immediately.
- Stage 2: Visual Retrospection: If the terminal state is ambiguous, the agent invokes the Check Screenshot tool to inspect historical intermediate screenshots, constructing a set of indirect visual evidence.
- Stage 3: Proactive Probing: If visual evidence is insufficient (e.g., for non-visual system changes), the agent transitions to active interaction using Execute Shell, Execute Python, or Computer Use to query latent states.

2.3 Test-Time Scaling Strategies

Read-Only Scaling (Verifier): To scale verification without the high cost of environment state restoration, the verifier is restricted to a read-only action space. This allows multiple verification samples to be run serially on a single environment instance without resetting.
Reward-Guided Scaling (Actor): VAGEN guides the actor agent via Best-of-N rejection sampling. The verifier evaluates $N$ sampled trajectories, and the one with the highest confidence among successful predictions is selected.

3. Key Contributions

Paradigm Shift: The paper identifies and advocates for a trajectory-grounded interactive verification paradigm, unifying passive trajectory analysis with active environment interaction to address the limitations of existing methods.
VAGEN Framework: The introduction of a tool-augmented verifier agent utilizing a Progressive Verification Mechanism that balances efficiency and rigor by prioritizing trajectory evidence before environmental probing.
Empirical Validation: Extensive experiments demonstrating that VAGEN outperforms both passive (LLM-as-a-Judge) and active (ProRe) baselines in accuracy and efficiency, while effectively guiding RL training and test-time scaling.

4. Experimental Results

The framework was evaluated on OSWorld-Verified (desktop) and AndroidWorld (mobile) benchmarks.

Verification Accuracy: VAGEN achieved state-of-the-art performance, surpassing baselines like ProRe and ZeroGUI. On OSWorld-Verified with human evaluation ground truth, VAGEN achieved 92.9% accuracy (vs. 87.3% for ProRe) with a class-balanced setup, and 93.4% with a class-imbalanced setup.
Efficiency: VAGEN demonstrated a favorable performance-efficiency trade-off. It required fewer input images and output tokens compared to ProRe and achieved verification with fewer average steps (17.4 vs. 25.2 for ProRe on OSWorld).
Reinforcement Learning: When used as a reward signal for online RL training (GRPO) with UI-TARS-1.5-7B, VAGEN improved the actor's success rate to 34.3%, outperforming the strongest baseline (ProRe) by 3.8 points.
Test-Time Scaling: VAGEN showed superior sample efficiency in Best-of-N scaling, matching the performance of other methods at $N=8$ with only $N=5$ samples. Read-only scaling further improved verifier accuracy without environment resets.

5. Significance and Claims

The paper claims that VAGEN provides a robust solution for the "partial state observability" problem in GUI agent reward modeling. By grounding verification in the trajectory first and escalating to environment interaction only when necessary, VAGEN avoids the inefficiency of pure probing methods while overcoming the blindness of passive LLM judges.

The authors emphasize that this approach enables reliable binary reward signals essential for RLVR, which are currently difficult to obtain at scale. The framework is presented as a plug-in verifier capable of improving both the training of GUI agents and their inference-time performance through better reward guidance and rejection sampling. The results suggest that the "easy to verify, hard to solve" property is a viable foundation for scalable agent verification in complex digital environments.