💬 NLP

Building Autonomous GUI Navigation via Agentic-Q Estimation and Step-Wise Policy Optimization

This paper introduces a novel MLLM-centered framework for autonomous GUI agents that combines agentic-Q estimation and step-wise policy optimization to enable efficient, self-generated data collection and stable reinforcement learning, allowing the Ovis2.5-9B model to outperform larger-scale competitors on navigation and grounding benchmarks.

Original authors: Yibo Wang, Guangda Huzhang, Yuwei Hu, Yu Xia, Shiyin Lu, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Lijun Zhang

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yibo Wang, Guangda Huzhang, Yuwei Hu, Yu Xia, Shiyin Lu, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Lijun Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart but inexperienced robot to navigate the internet. Your goal is for this robot to click buttons, type text, and scroll through websites to complete tasks like "Find the price of an iPhone 14" or "Book a flight to Paris."

This paper introduces a new, highly efficient way to train this robot, solving three major headaches that usually plague AI developers: expensive data, unclear feedback, and unstable training.

Here is the breakdown of their solution using simple analogies:

1. The Problem: The "Black Box" of Feedback

Usually, training an AI to navigate a website is like playing a video game where you only get a "Game Over" or "You Win" message at the very end of a 50-level marathon.

The Issue: If the robot fails at step 3, you don't know if it was because of step 3, or because it made a bad choice in step 45.
The Cost: To fix this, humans usually have to watch hours of video and manually label every single click as "good" or "bad." This is incredibly expensive and slow.
The Instability: Because the internet changes constantly (websites update, pop-ups appear), the "game" the robot is playing is never the same twice, making it hard to learn consistently.

2. The Solution: The "Agentic-Q" Coach

The authors propose a two-step training method that acts like a smart coach who watches the robot play and gives immediate feedback.

Step A: The "Agentic-Q" Estimator (The Instant Coach)

Instead of waiting until the end of the task to say "Good job" or "Bad job," they train a special AI model (the Agentic-Q) to act as a coach.

How it works: Every time the robot makes a move (like clicking a button), the Coach instantly evaluates: "If the robot does this, what are the chances it will eventually succeed?"
The Magic Trick: The Coach learns this by looking at the robot's own past attempts. If the robot eventually succeeded, the Coach learns that the steps leading up to it were "good." If it failed, those steps were "bad."
Why it's great: The Coach doesn't need a human to watch. It generates its own training data by watching the robot play, then uses that data to grade the robot's future moves instantly.

Step B: Step-Wise Policy Optimization (The Drill Sergeant)

Once the Coach is ready, the robot starts training again.

The Old Way: The robot plays a whole game, gets a score at the end, and tries to guess what it did wrong.
The New Way: The robot plays one step. The Coach immediately says, "That was a 90% chance of success!" or "That was a 10% chance!" The robot uses this instant feedback to adjust its brain immediately.
The Benefit: This is like a video game where you get a score after every single jump, rather than waiting until the end of the level. This makes learning much faster and more stable.

3. The Secret Sauce: "Sliding Window" and "Focus"

The authors noticed that if the robot remembers everything it did since the beginning of the day, it gets confused (like trying to remember every word of a 10-hour conversation).

The Fix: They use a Sliding Window. The Coach only looks at the last few steps the robot took. It's like a coach saying, "Forget what you did 20 minutes ago; focus on what you just did." This keeps the robot's brain clear and prevents it from getting stuck in bad habits.

4. The Results: Beating the Giants

They tested this new method (called Ovis2.5) on real websites like Amazon, Google Maps, and Apple.

The Surprise: Their model, which is relatively small (9 billion parameters), performed better than massive, expensive models from big tech companies (like GPT-4o and Claude 3.7/4).
The "Boundary Traversal" Superpower: In one cool example, the robot was asked to find an iPhone 14 on the Apple website. The robot realized the product was gone. Instead of getting stuck or giving up, it decided to switch strategies. It opened a new tab, went to Google, searched there, and found the price. It showed it could think, "This path is blocked, let's try a different road."

Summary Analogy

Imagine you are teaching a child to ride a bike.

Old Method: You let them ride for 10 miles. At the end, you say, "You fell down three times." The child has no idea which pedal stroke caused the fall.
This Paper's Method: You ride alongside them with a special sensor (the Agentic-Q). The moment they wobble, the sensor says, "Lean left!" The child corrects immediately. You don't need a human to watch every second; the sensor learns from the child's own riding history to give perfect, instant advice.

The Bottom Line: This paper shows that by giving AI an "instant feedback loop" that it can generate itself, we can build smarter, faster, and more adaptable internet navigators without needing armies of human teachers.

1. Problem Statement

The paper addresses the challenges in developing autonomous agents for Graphical User Interfaces (GUIs) using Multimodal Large Language Models (MLLMs). While MLLMs have advanced, real-world GUI agents face three critical bottlenecks:

High Data Curation Costs: Creating high-quality interaction trajectories (state-action pairs) typically requires expensive expert annotation. Unlike standard image-text data, GUI trajectories require detailed reasoning and action sequences.
Sparse and Delayed Supervision: GUI tasks often involve long-horizon interactions where feedback (success/failure) is only available at the very end of a trajectory. This makes it difficult to optimize intermediate steps, as the reward signal is delayed.
Unstable Optimization: Existing Reinforcement Learning with Verifiable Rewards (RLVR) frameworks tightly couple policy optimization with the execution environment. Since GUI environments (e.g., live websites) are non-stationary (changing layouts, load issues), this coupling leads to inefficient and unstable training.

2. Methodology

The authors propose a novel framework centered on two core components: Agentic-Q Estimation and Step-Wise Policy Optimization (SWPO). The framework operates in three stages:

A. Data Curation & Cold Start

Detection-Rerunning: To handle environmental noise (e.g., captchas, crashes), the system uses expert models to detect failed trajectories caused by external issues and reruns them.
Data Stratification: Tasks are ranked by difficulty based on the success rate of multiple rollouts. Extremely easy (always success) or hard (always failure) tasks are excluded to focus training on informative samples.
Cold Start (SFT): The base model (Ovis2.5-9B) is first fine-tuned on expert web navigation trajectories to acquire basic GUI grounding and navigation capabilities.

B. Agentic-Q Estimation

Instead of waiting for final rewards, the framework trains a Q-model to evaluate the contribution of each individual step to the final task completion.

Backward Propagation: Final task outcomes (Success/Failure) are propagated backward to intermediate steps. A step is labeled with the return ( $G_i$ ) of the trajectory it belongs to.
Training Objective: The Q-model is trained as a binary classifier to predict the probability of task success given a state, thought, and action.
Stabilization Techniques:
- Sliding Window: Limits the historical context in the state input to prevent the model from losing focus on the current step, mitigating entropy collapse.
- Action-Focus: Masks the "thought" (reasoning) part during training, forcing the Q-model to focus on the action itself.

C. Step-Wise Policy Optimization (SWPO)

The policy is optimized using the Q-model's predictions as rewards, decoupling the training from the live environment.

Critic-Free Optimization: The framework adapts critic-free algorithms (GRPO, RLOO, REINFORCE++) to work with step-wise data.
Key Modifications:
- Step-wise Rewards: Uses Q-model predictions ( $g_i$ ) instead of final binary rewards.
- Action-Level Return Filtering: Discards samples where the agent generates identical actions (low diversity) or where the Q-model's predicted returns have very low variance (low signal).
- Adaptive Group Weighting: Weights groups of samples inversely proportional to their average return. This ensures the model pays more attention to "harder" tasks (lower average return) which provide more learning signal.

3. Key Contributions

Agentic-Q Estimation: A novel approach to generate dense, step-wise supervision signals from sparse terminal rewards, enabling fine-grained policy optimization without human annotation.
Decoupled Optimization: By using a learned Q-model for intermediate rewards, the policy update is decoupled from the non-stationary GUI environment, significantly improving training stability and efficiency.
Self-Generated Data: The entire training pipeline (data collection and optimization) relies on trajectories generated by the policy itself, eliminating the need for costly expert annotations.
Stabilization Strategies: Introduction of sliding-window and action-focus techniques to prevent entropy collapse and ensure robust Q-model generalization.

4. Experimental Results

The framework was evaluated using Ovis2.5-9B as the base model against state-of-the-art models (including GPT-4o, Claude 3.7/4 Sonnet, Qwen3-VL, and UI-TARS).

GUI Grounding (ScreenSpot v1/v2):
- The Ovis2.5-9B models achieved state-of-the-art performance among 9B-scale models.
- They surpassed larger proprietary models (e.g., GPT-4o, Claude 4 Sonnet) and significantly outperformed other open-source models like UI-TARS and Qwen3-VL.
- Notably, the grounding capability did not degrade after RL training; it slightly improved.
GUI Navigation (WebVoyager & Online-Mind2Web):
- WebVoyager: The models achieved success rates of ~78-79%, improving over the SFT baseline by ~10 points and outperforming all baselines, including GPT-4o (57.6%) and Claude 4 (82.1% - note: Ovis2.5-RLOO was 77.8%, slightly lower than Claude 4 but competitive with larger models).
- Online-Mind2Web (Out-of-Distribution): Despite training only on WebVoyager data, the models achieved 64.0% average success on Online-Mind2Web, surpassing GPT-4o (37.0%) and Claude 3.7 (61.0%), demonstrating exceptional generalization.
- Efficiency: The average number of steps to complete tasks was significantly reduced (e.g., from 22.15 to 13.57), indicating more efficient navigation.
Emergent Behavior: The agents demonstrated "Boundary Traversal," where they could recognize when a task was impossible within the current website context and autonomously switch strategies (e.g., switching from a specific product page to a Google search) to complete the task.

5. Significance

This work represents a paradigm shift in training autonomous GUI agents. By moving away from environment-coupled RLVR and expensive expert data curation, the proposed framework offers a scalable, cost-effective, and stable path to high-performance agents.

Scalability: The self-play nature of the framework allows for infinite data generation without human intervention.
Generalization: The ability to outperform larger models on out-of-distribution benchmarks suggests that step-wise optimization leads to more robust reasoning capabilities.
Practicality: The decoupling of training from the execution environment makes the approach viable for real-world deployment where environments are dynamic and unpredictable.

In summary, the paper demonstrates that with the right intermediate supervision mechanism (Agentic-Q) and optimization strategy (SWPO), a 9B parameter model can rival or exceed the performance of much larger, proprietary frontier models in complex, real-world GUI navigation tasks.