🤖 AI

Faithful Mobile GUI Agents with Guided Advantage Estimator

This paper introduces Faithful-Agent, a two-stage framework that enhances the reliability of vision-language GUI agents by combining evidence-grounded supervised fine-tuning with a guided advantage estimator (GuAE) to significantly reduce hallucinations and improve task success rates while maintaining general instruction-following capabilities.

Original authors: Haowen Hu, Pengzhou Cheng, Zheng Wu, Lingzhong Dong, Gongshen Liu, Zhuosheng Zhang

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haowen Hu, Pengzhou Cheng, Zheng Wu, Lingzhong Dong, Gongshen Liu, Zhuosheng Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read robot assistant that can look at your phone screen and follow your instructions, like "Buy me an iPhone 17" or "Reset my password." This robot uses a powerful brain (a Vision-Language Model) to understand what it sees and what you say.

However, the paper points out a major flaw: this robot is often a "confident liar."

The Problem: The Robot That Guesses Instead of Looking

The authors call this behavior "unfaithful." Here is what that looks like in real life:

The "Memory" Trap: Imagine you ask the robot to "Reset your password." The robot sees a screen that is actually blank or covered by a pop-up ad. Instead of saying, "I can't see the button," it remembers that usually the reset button is in the top right corner. It blindly clicks there, even though the button isn't there. It's relying on a memorized shortcut rather than looking at the actual evidence.
The "Daydream" Trap: Imagine you ask the robot to "Buy an iPhone on Amazon," but the screen is currently showing Apple Music. The robot ignores the fact that it's in the wrong app and keeps trying to buy the phone inside the music app. It drifts away from your actual instruction because it's not paying attention to the context.

The robot is essentially "hallucinating" its way through tasks, guessing what should be there instead of checking what is there.

The Solution: "Faithful-Agent"

The researchers created a new training method called Faithful-Agent. Think of this as a strict coach teaching the robot a new rule: "If you can't see it, don't touch it."

The training happens in two stages, like learning to drive:

Stage 1: The "Stop and Think" Lesson (SFT)

First, they teach the robot to abstain.

The Analogy: Imagine a student taking a test. If they don't know the answer, a normal student might guess wildly. A "faithful" student is taught to raise their hand and say, "I don't have enough information to answer this."
How it works: The robot is trained to recognize when the screen is blocked, confusing, or doesn't match the instruction. Instead of guessing, it learns to press "Back," go "Home," or stop the task entirely. This stops it from making wild, ungrounded guesses.

Stage 2: The "Smart Coach" (RFT with GuAE)

This is the most technical part, but here is the simple version.
The robot learns by trying many different actions and seeing which one gets a "reward" (a good score). But in the real world, rewards are often sparse (you only get a reward if you finish the whole task) and binary (you either got it right or wrong; there's no "almost right").

The Problem: When the robot tries 8 different guesses and they all look "wrong" (or all look "right"), the training system gets confused. It can't tell which guess was slightly better because they all look the same. The authors call this "Advantage Collapse." It's like a teacher trying to grade a class where every student got a 50/50; the teacher can't figure out who needs more help.
The Fix (GuAE): The researchers invented a new "Guided Advantage Estimator" (GuAE).
- The Analogy: Imagine a coach who refuses to let the team's performance score drop to zero just because everyone played similarly. The coach adds a "safety anchor" to the scoring system. Even if the team is stuck in a rut, the coach ensures there is still a tiny, clear signal telling the players, "Keep trying, but be more careful."
- This prevents the robot from getting stuck in a loop where it stops learning because the feedback looks too flat.

The Results

The paper tested this new robot against others on a "Trap" dataset (scenarios designed to trick the robot, like hidden buttons or wrong apps).

Before: The old robots failed these traps about 86% of the time (Success Rate was only ~14%). They kept guessing and failing.
After: The Faithful-Agent succeeded in these traps 80% of the time.

Most importantly, the robot didn't lose its ability to do normal tasks. It became more reliable without becoming "dumb" or overly cautious. It learned to say, "I can't do this right now," instead of crashing the system with a wrong guess.

Summary

The paper introduces a way to train AI agents to stop guessing when the evidence is missing. By teaching them to stop and back out when things look wrong, and by using a smarter scoring system to keep them learning even when the feedback is vague, the robot becomes much more trustworthy and less likely to make up its own reality.

Technical Summary: Faithful Mobile GUI Agents with Guided Advantage Estimator

1. Problem Statement

Vision-language model (VLM) based graphical user interface (GUI) agents have demonstrated strong capabilities in mobile, web, and desktop automation. However, recent research indicates these agents often behave unfaithfully. This unfaithfulness manifests in two primary failure modes:

Ungrounded Interaction: Agents rely on memorized shortcuts or superficial patterns rather than conditioning their decisions on the actual visible interface state. This leads to actions taken without verifying the current screen (e.g., clicking a button that is occluded or non-existent).
Instruction Deviation: Agents drift from explicit user instructions, particularly when the instruction conflicts with the current task step or interface state, often proceeding into unrelated interfaces.
Internal Inconsistency: There is a mismatch between the agent's reasoning trace (thought) and its executed action, where the reasoning implies one intent but the action follows another.

These failures are exacerbated in sparse reward environments typical of GUI tasks, particularly when dealing with discrete, enumerated actions (e.g., "System Button," "Terminate"). Standard Reinforcement Fine-Tuning (RFT) methods, such as Group Relative Policy Optimization (GRPO), suffer from advantage collapse in these scenarios. When rollout groups under the same input yield near-identical rewards (common in low-variance, near-binary outcomes), within-group normalization drives advantages toward zero, eliminating the learning signal needed to refine faithful behaviors like abstention (e.g., pressing "Back" or "Home" when evidence is missing).

2. Methodology: Faithful-Agent

The authors propose Faithful-Agent, a framework that reformulates GUI interaction as a "faithfulness-first" problem. The core requirement is that agents must prioritize evidence groundedness (decisions supported by the interface and instruction) and internal consistency (actions matching reasoning). When evidence is missing or conflicting, a faithful agent should abstain from speculative execution.

The framework employs a two-stage training pipeline:

Stage I: Faithfulness-Oriented Supervised Fine-Tuning (SFT)

Goal: To instill abstainment behaviors under evidence perturbations (e.g., occlusion, content tampering, goal shifts).
Mechanism: The model is trained on a dataset containing both general trajectories and "Trap" perturbations. In Trap scenarios, the agent is supervised to recognize missing or conflicting evidence and take safe, abstaining actions (e.g., SYSTEM_BUTTON(BACK), SYSTEM_BUTTON(HOME), or TERMINATE) rather than guessing.
Outcome: This provides a "faithful cold start," stabilizing subsequent RFT by reducing unfaithful exploration.

Stage II: GRPO with Guided Advantage Estimator (GuAE)

Goal: To further amplify faithfulness and mitigate advantage collapse under sparse, near-binary rewards.
Reward Design: The reward function combines:
- Action-Match Reward ( $R_{AM}$ ): Evaluates if the predicted action matches the ground truth, allowing partial credit for continuous arguments (e.g., coordinates) but near-binary scoring for discrete actions.
- Thought-Action Consistency Reward ( $R_{Cons}$ ): Penalizes mismatches between the intermediate reasoning trace and the executed action, discouraging speculative execution.
Guided Advantage Estimator (GuAE): To address the bottleneck where standard GRPO advantages vanish in low-variance groups, the authors introduce GuAE, which consists of two components:
1. Anchor Regularization: The empirical reward set is augmented with static anchor points $\{0, 1\}$ . This ensures that even if all rollout rewards are identical (e.g., all 0 or all 1), the group statistics (mean and variance) remain non-zero, preventing the advantage from collapsing to zero.
2. Variance-Adaptive Tempering (VAT): A mechanism that scales the advantage based on the group's dispersion relative to a reference volatility (derived from a maximum-entropy uniform distribution). It sharpens signals in low-variance groups (where subtle differences matter) and dampens noisy updates in high-variance groups.

3. Key Contributions

Definition of Step-wise Faithfulness: The paper formally defines faithfulness as a twofold requirement: evidence groundedness and internal consistency, explicitly specifying abstention when key evidence is missing or conflicting.
Two-Stage Training Recipe: A novel pipeline combining SFT to instill abstention priors and GRPO-based RFT to reinforce step-wise faithfulness.
Guided Advantage Estimator (GuAE): An anchor-based, variance-adaptive tempering mechanism built upon GRPO. It prevents advantage collapse in low-variance rollout groups under sparse GUI rewards, enabling the policy to refine faithful actions where standard optimizers fail.
Thought-Action Consistency Reward: An auxiliary reward signal that aligns stated intent with executed actions, reducing speculative behaviors.

4. Experimental Results

The authors evaluate Faithful-Agent on a cross-scenario dataset comprising general trajectories and controlled "Trap" perturbations (occlusion, content tampering, goal shifts).

Performance on Trap Scenarios: Faithful-Agent (Stage II) significantly outperforms strong baselines (including Qwen3-VL-8B, UI-TARS, and OS-Atlas). Specifically, it elevates the Trap Success Rate (SR) from 13.88% (baseline) to 80.21%.
General Performance: The framework preserves robust general instruction-following capabilities, achieving an overall success rate of 71.67% and maintaining high performance on general (non-trap) tasks.
Ablation Studies:
- Removing the Trap data during training leads to a sharp drop in Trap performance, confirming the necessity of faithfulness-oriented supervision.
- Replacing GuAE with standard GRPO results in a lower Trap SR (78.58% vs. 80.21%), demonstrating that GuAE effectively preserves learning signals in low-variance groups.
- Adding the Thought-Action Consistency reward ( $R_{Cons}$ ) further improves performance, particularly in reducing contradictory thought-action pairs.
Stability: Diagnostics show that GRPO+GuAE maintains larger advantage magnitudes and smoother gradient norms compared to vanilla GRPO, which suffers from increasing near-zero advantage ratios as training progresses.

5. Significance and Claims

The paper claims that Faithful-Agent addresses a critical gap in GUI automation: the tendency of agents to hallucinate or rely on shortcuts when interface evidence is ambiguous or conflicting. By prioritizing faithfulness through a dedicated training pipeline and a novel advantage estimator, the method enables agents to:

Detect and recover from missing or conflicting evidence (e.g., by pressing "Back" or "Home").
Avoid speculative execution that leads to errors in complex, multi-step tasks.
Achieve high reliability in "Trap" scenarios without sacrificing general utility.

The authors emphasize that their approach bridges the gap between assessing faithfulness and actively training agents to be faithful to underlying evidence. They note that while more capable agents can be misused, the framework includes safety mechanisms (abstention) and recommends running agents in sandboxed environments with least-privilege permissions to mitigate privacy and security risks.

The work suggests that optimizing for step-wise faithfulness yields consistently better overall results than optimizing solely for task completion, particularly in dynamic and perturbed GUI environments.