💻 computer science

HATS: Hardness-Aware Trajectory Synthesis for GUI Agents

This paper introduces HATS, a Hardness-Aware Trajectory Synthesis framework that improves GUI agent generalization by identifying and prioritizing semantically ambiguous actions through a closed-loop process of hardness-driven exploration and alignment-guided refinement.

Original authors: Rui Shao, Ruize Gao, Bin Xie, Yixing Li, Kaiwen Zhou, Shuai Wang, Weili Guan, Gongwei Chen

Published 2026-03-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rui Shao, Ruize Gao, Bin Xie, Yixing Li, Kaiwen Zhou, Shuai Wang, Weili Guan, Gongwei Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to use your smartphone or a website. You want the robot to be able to book a flight, send a text, or buy groceries just by listening to your voice commands.

To teach this robot, you need to show it thousands of examples of "doing things." This is called training data. In the past, researchers tried to generate these examples automatically, but they hit a major wall: the robot became very good at simple tasks (like opening a menu) but terrible at anything tricky.

The paper "HATS" (Hardness-Aware Trajectory Synthesis) introduces a new way to teach these robots. Here is the simple breakdown using some everyday analogies.

The Problem: The "Easy Mode" Trap

Imagine you are teaching a child to cook.

Old Method: You tell the child to "make dinner." They start by doing the easiest things: opening the fridge, turning on the stove, and stirring an empty pot. They do this 1,000 times.
The Result: The child thinks they are a master chef because they know how to open the fridge. But if you ask them to actually chop an onion or season a steak (the tricky parts), they fail because they never practiced those hard steps.

In the world of AI, existing methods were like this. They generated millions of "easy" instructions (e.g., "tap the back button") but ignored the hard, confusing parts of using an app. These hard parts are things like:

Two buttons that look exactly the same but do different things depending on what page you are on.
A task that only works if you do Step A, then Step B, then Step C in the exact right order.

The Solution: HATS (The "Smart Coach")

The authors created HATS, which stands for Hardness-Aware Trajectory Synthesis. Think of HATS as a Smart Coach who doesn't just let the student practice whatever they want. Instead, the Coach actively looks for the things the student is bad at and forces them to practice those specific skills.

HATS works in a loop with two main parts:

1. The "Hardness Detector" (Exploration)

Instead of wandering around randomly, HATS asks: "Where is the robot getting confused?"

The Metaphor: Imagine a video game player who keeps dying at the same boss level. A normal player might just keep playing the easy levels to get high scores. HATS is like a coach who says, "Stop! You keep failing at this specific boss. Let's go back and practice that specific fight until you get it right."
How it works: The system looks for "Semantic Ambiguity." This is a fancy way of saying "confusing situations." If the robot sees a button and doesn't know what it does, HATS marks that spot as "Hard" and makes sure the robot visits it again and again to learn the nuance.

2. The "Reality Check" (Refinement)

Once the robot tries to do a hard task, HATS doesn't just accept the attempt. It plays a game of "Spot the Difference."

The Metaphor: Imagine you give a recipe to a chef. The chef tries to cook it, but they burn the toast because you didn't say "watch the heat."
- Old Method: The system would just write down, "Chef made toast," and move on.
- HATS Method: The system watches the chef. It sees the toast is burnt. It says, "Wait, the recipe was vague. Let's rewrite the recipe to say 'Watch the heat carefully.' Now, try again."
How it works: HATS generates an instruction, lets the robot try it, and then checks: "Did the robot actually do what the instruction said?" If the robot failed because the instruction was unclear, HATS fixes the instruction and tries again. It keeps looping until the instruction is perfect and the robot succeeds.

The Magic Loop

The genius of HATS is that these two parts talk to each other in a circle:

The Coach finds a hard spot (Exploration).
The Reality Check tries to solve it and fixes the instructions (Refinement).
If the Reality Check fails, it tells the Coach: "That was really hard! Let's mark that as a high-priority training spot for next time."
The Coach then goes back to find even more tricky spots based on that feedback.

Why Does This Matter?

The results are impressive. When they tested this new method:

On Android apps, the new AI was 100% better at solving tasks than the previous best method.
On Websites, it was 215% better.

Summary

Think of HATS as a teacher who refuses to let a student skip the hard math problems. Instead of letting the student solve easy addition problems 1,000 times to feel smart, the teacher identifies the confusing algebra problems, rewrites the explanations until they make sense, and forces the student to master them.

By focusing on the hard, confusing, and ambiguous parts of using technology, HATS creates a robot that doesn't just know how to tap a screen—it actually understands how to get things done in the real world.

1. Problem Statement

Graphical User Interface (GUI) agents powered by Large Vision-Language Models (VLMs) require high-quality trajectory data (instruction-action-state pairs) for effective training. However, existing automated trajectory synthesis pipelines, such as OS-GENESIS, suffer from two critical limitations:

Semantic Ambiguity Gap: Current methods rely on random exploration or shallow search, which overwhelmingly generates "semantic-intuitive" actions (e.g., simple taps, scrolling) that are easy to learn but offer little generalization value. They fail to capture semantic-ambiguous actions, where the meaning of an interaction depends on subtle visual cues, context, or sequential prerequisites (e.g., a "+" button that adds a contact in one context but a folder in another).
Instruction-Execution Misalignment: Existing "one-shot" reverse synthesis methods generate instructions directly from observed traces without validation. This often leads to instructions that are vague, lack necessary context, or cannot be faithfully replayed, resulting in noisy supervision and poor agent performance.

2. Methodology: HATS Framework

The authors propose HATS (Hardness-Aware Trajectory Synthesis), a closed-loop framework designed to explicitly target and resolve semantic ambiguity. The framework integrates two complementary modules within a unified Hardness-Driven Monte Carlo Tree Search (HD-MCTS) procedure:

A. Hardness-Driven Exploration Module

Instead of random walks, this module steers the agent's exploration toward "hard" states—those associated with high semantic ambiguity.

Hardness Definition: Hardness is defined as the degree of semantic ambiguity associated with an action.
Selection Policy: The agent uses a modified Upper Confidence Bound (UCB) policy to traverse an action tree. Unlike standard MCTS, the value estimate $Q(v, a)$ is updated based on a hardness reward derived from alignment feedback, prioritizing under-represented and semantically complex interactions.
Expansion & Rollout: The agent expands the tree by executing unvisited actions and performs bounded-depth rollouts to generate candidate trajectories.

B. Alignment-Guided Refinement Module

This module acts as a quality control gate, transforming noisy exploration traces into reliable training data.

Multi-Round Replay: Instead of accepting a one-shot instruction, the system iteratively synthesizes a natural language instruction for a trajectory segment, executes it in the environment, and compares the resulting execution sequence against the original reference trajectory.
Similarity Verification: It calculates an Action-Level Reconstruction Recall ( $R$ ) metric. If the execution fails to match the reference (low recall), the instruction is deemed misaligned.
Iterative Repair: The system refines the instruction by injecting missing context (e.g., specific UI element IDs, navigation steps) and re-executes until the alignment score meets a threshold ( $R \geq R_{min}$ ) or a refinement budget is exhausted.

C. Closed-Loop Integration (HD-MCTS)

The two modules operate in a feedback loop:

Exploration supplies challenging trajectories to the refinement module.
Refinement computes the alignment score. Low alignment (high difficulty) generates a high hardness reward.
Backpropagation: This reward is backpropagated up the MCTS tree, biasing future exploration toward regions of the state space that are semantically ambiguous and difficult to align. This turns "noise" (misalignment) into a useful learning signal.

3. Key Contributions

HATS Framework: A unified, closed-loop trajectory synthesis framework that explicitly targets semantic-ambiguous actions, addressing the data quality bottleneck in GUI agent training.
Hardness-Driven Exploration: A novel exploration policy that prioritizes rare, high-value, semantically complex actions over trivial ones, significantly improving data diversity.
Alignment-Guided Refinement: A multi-round validation mechanism that replaces unreliable one-shot synthesis with iterative replay and repair, ensuring strict instruction-execution alignment.
Hardness Reward Mechanism: A method to convert semantic misalignment into a scalar reward signal, enabling the agent to autonomously discover and learn from difficult interactions.

4. Experimental Results

The authors evaluated HATS on two major benchmarks: AndroidWorld (mobile) and WebArena (web), using various VLM backbones (InternVL2, Qwen2-VL).

Performance Gains: HATS consistently outperformed state-of-the-art baselines (including OS-GENESIS, Task-Driven, and Zero-Shot methods).
- AndroidWorld: Achieved a 22.60% overall success rate with InternVL2-4B, a 100% improvement over OS-GENESIS (11.30%).
- WebArena: Achieved 20.60% overall success rate, a 215% improvement over OS-GENESIS (6.53%).
Category-Specific Improvements: The most significant gains were observed in categories known for semantic complexity, such as Productivity & Work (P&W) and Social & Communication (S&C), where HATS doubled or tripled the success rates of baselines.
Ablation Studies:
- Refinement: Multi-round refinement increased action-level recall from 0.26 to 0.40 and boosted success rates by ~7%.
- Hardness Metric: The recall-based hardness metric was shown to correlate better with downstream performance than precision-based metrics.
- Data Distribution: HATS shifted the data distribution to include more P&W tasks (46.6% vs. 31.2% in baselines), directly correlating with improved generalization on complex workflows.

5. Significance

Bridging the Generalization Gap: By focusing on semantic ambiguity, HATS enables agents to handle real-world workflows that require context-aware reasoning, moving beyond simple, scripted interactions.
Data Efficiency: The framework demonstrates that high-quality, semantically rich data can be synthesized automatically without human annotation, provided that "hardness" is used as a guiding signal.
Robustness: The iterative alignment process ensures that the training data is free from the supervision noise that plagues current synthetic datasets, leading to more reliable and robust GUI agents.
Scalability: The closed-loop design offers a scalable path to generating diverse, high-fidelity datasets for the next generation of autonomous digital assistants.