🤖 AI

AutoSurfer -- Teaching Web Agents through Comprehensive Surfing, Learning, and Modeling

AutoSurfer is a novel web trajectory generator that employs systematic breadth-first exploration, exploration-guided task synthesis, and trajectory-based refinement to overcome data scarcity and hallucination issues, thereby significantly improving web agent performance and task diversity on the WebArena benchmark.

Original authors: Fazle Elahi Faisal, Qianhui Wu, Baolin Peng, Jianfeng Gao

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fazle Elahi Faisal, Qianhui Wu, Baolin Peng, Jianfeng Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to navigate a massive, complex library (a website) to find specific books, check them out, or write reviews. The problem is, the robot doesn't know the library's layout, and nobody has written a manual for it.

AutoSurfer is a new system designed to teach these "web robots" (called web agents) by acting like a very thorough, organized human librarian who explores the library first, writes a perfect guide, and then teaches the robot using that guide.

Here is how AutoSurfer works, broken down into simple steps:

1. The Problem: Robots Getting Lost

Current robots trying to navigate websites often fail because they haven't been trained on enough good examples.

The Old Way: Previous methods tried to teach robots by either:
- Guessing from the front door: They looked at the homepage and guessed what tasks might be possible, missing the hidden rooms deep inside the library.
- Wandering aimlessly: They walked around randomly, like a tourist bumping into walls. This often led to the robot making up tasks that didn't actually exist (hallucinations) or missing important features like dropdown menus that are hidden until you click them.

2. The Solution: AutoSurfer's Three-Step Strategy

AutoSurfer fixes this by using a "Learn as You Go" approach that mimics how a human would learn a new website.

Step A: The Systematic Explorer (The Breadth-First Search)

Instead of wandering randomly, AutoSurfer acts like a detective with a checklist.

The Queue: It keeps a list of every page it has found but hasn't fully explored yet. It starts at the homepage, then checks every link on that page, then every link on those pages, and so on.
The "Unfolding" Trick: Imagine a menu that hides options until you click "More." Old methods often missed these. AutoSurfer is smart enough to click the "More" button, see the new options, and immediately add those new options to its checklist. It does this recursively, peeling back layers of menus just like a human would.
No Redundancy: If it finds a "Search" box on the homepage, it doesn't waste time looking for a "Search" box on every single other page. It remembers what it already knows and skips the duplicates.

Step B: Writing the Story (Task Synthesis)

Once AutoSurfer has explored the website, it doesn't just say, "I clicked here." It writes a story.

Connecting the Dots: It looks at the path it took (the exploration trajectory) and creates a task based on that real path.
The Metaphor: Imagine you are teaching someone how to bake a cake.
- Old methods might say, "Buy flour, then buy eggs, then bake." (Isolated steps).
- AutoSurfer says, "First, you walked to the aisle, picked up the flour, walked to the checkout, paid, then walked to the oven..." It grounds the task in the actual journey, making the instructions much harder to misunderstand.

Step C: The Polish (Refinement)

The first draft of the instructions might be messy. Maybe the robot took a detour or clicked a button twice by mistake.

The Editor: AutoSurfer takes its own "rough draft" of the journey and gives it to a web agent (a smart robot) with a hint: "Here is the path we took; now, please do this task perfectly."
The Result: The agent cleans up the path, removes unnecessary steps, and ensures the instructions are 100% accurate. This creates a high-quality "textbook" for training.

3. The Result: A Super-Student

AutoSurfer turns these clean, verified paths into a dataset to train a specific "Website-Specific LLM" (a robot brain dedicated to one website).

The Test: The researchers tested this on a benchmark called WebArena (a simulated internet with sites like Reddit, a shopping store, and a code repository).
The Score: The robot trained by AutoSurfer got 24.23% of tasks right. The best previous method only got 19.59% right.
The Diversity: Not only was it more accurate, but it also learned a wider variety of tasks. It didn't just learn the easy stuff; it found the deep, complex features of the websites that other methods missed.

Summary Analogy

Think of training a web agent like training a new employee for a giant, confusing office building.

Old methods gave the employee a map with only the lobby marked and told them to "go find the breakroom." They got lost.
AutoSurfer sends a senior employee to walk every hallway, open every door, check every closet, and write a step-by-step guide based on the actual walk. Then, it sends that guide to the new employee. The new employee arrives knowing exactly where the breakroom is, how to get there, and what to do when they get there.

The paper claims that by being this thorough and systematic, AutoSurfer creates better training data, leading to web agents that are more accurate, less likely to make things up, and capable of handling complex, multi-step tasks on the web.

1. Problem Statement

Web agents powered by Multimodal Large Language Models (MLLMs) aim to automate complex tasks on websites. However, their accuracy is severely limited by the scarcity of high-quality web trajectory training data. Existing automatic data generation methods suffer from three primary flaws:

Incomplete Coverage: Methods relying on homepage-based task proposals (e.g., Explorer) or random-walk exploration (e.g., OS-Genesis, SynthAgent) fail to discover deep, hidden functionalities like multi-level menus or dynamic content.
Hallucination and Ambiguity: Task synthesis often relies on isolated page content or random actions, leading to hallucinated tasks that do not align with actual navigation paths or user intent.
Weak Trajectory Grounding: Generated trajectories often lack coherence because they are not grounded in a systematic exploration of the website's actual state space, leading to unreliable training data for Website-Specific LLMs (wLLMs).

2. Methodology: AutoSurfer

AutoSurfer is a comprehensive web trajectory generator designed to address these limitations through a three-stage pipeline: Systematic Exploration, Trajectory-Grounded Task Synthesis, and Trajectory-Guided Refinement.

A. Systematic Breadth-First Exploration

Unlike random walks, AutoSurfer employs a breadth-first traversal (BFS) strategy to comprehensively map a website's action space:

Queue-Based Navigation: It maintains a queue of discovered pages and their corresponding action traces. It propagates knowledge from earlier pages to later ones to avoid redundant exploration.
Recursive GUI Expansion: It treats static and dynamic elements differently:
- Static Elements: Systematically traverses fixed functionality (e.g., navigation bars).
- Dynamic Elements: Samples a representative subset of dynamic content (e.g., list items) to avoid redundancy while capturing functionality.
- Menu/Dropdown Handling: It recursively expands menus and dropdowns. If an action (e.g., clicking a profile name) reveals new elements, AutoSurfer identifies these new elements as a hierarchy, reconstructing leaf tasks with the full action prefix (e.g., "Click Profile" $\rightarrow$ "Click Settings").
Task Filtering: It excludes tasks involving authorization (login/signup), payments, ads, and pop-ups, focusing on actionable, state-changing interactions.

B. Task Synthesis with Exploration Trajectories

AutoSurfer synthesizes complex tasks by combining logically connected simple tasks discovered during exploration:

Action-Oriented Tasks: It identifies sequences of simple tasks spanning at least three GUI actions. It uses LLM reasoning to evaluate completeness, retaining only high-quality tuples.
Information-Seeking Tasks: It generates questions users might ask about the page content, grounded in the specific navigation path taken to reach that state.
Key Innovation: Unlike prior methods that synthesize tasks from isolated page content, AutoSurfer grounds tasks in the full exploration trajectory. This ensures the task description reflects the actual navigation context, reducing hallucinations.

C. Trajectory-Guided Refinement

The initial exploration data often contains redundant steps or sub-optimal paths (e.g., missing scroll actions). AutoSurfer refines these using a web agent (SynthAgent) augmented with the exploration trajectory as a hint:

The agent uses the exploration path as a "ground truth" guide to correct the task description and generate a clean, efficient trajectory.
This step filters out failed attempts and optimizes the action sequence, ensuring the final data is suitable for Supervised Fine-Tuning (SFT).

D. Model Training (wLLM)

The refined task-trajectory pairs are converted into an SFT dataset in ShareGPT format. The dataset includes the task description, page screenshot, accessibility tree, and action history. This data is used to fine-tune multimodal LLMs (specifically Qwen2.5-VL and Qwen3-VL) to create Website-Specific LLMs (wLLMs).

3. Key Contributions

Systematic Exploration Strategy: A novel BFS approach that recursively expands menus, distinguishes between fixed and dynamic elements, and propagates knowledge across pages to achieve near-complete website coverage.
Trajectory-Grounded Synthesis: A method that uses the full exploration path to synthesize tasks, significantly reducing hallucinations compared to methods relying on isolated page content or random walks.
Refinement via Exploration Hints: A refinement mechanism that uses the exploration trajectory as a constraint to steer the web agent toward correct execution paths, improving data quality.
First wLLM Training Framework: AutoSurfer is the first work explicitly designed to generate data for training wLLMs, demonstrating that comprehensive website coverage is necessary for high-performance, site-specific agents.

4. Experimental Results

The authors evaluated AutoSurfer on the WebArena benchmark (using CMS, Gitlab, Reddit, and Shopping environments) by fine-tuning Qwen2.5-VL-7B-Instruct.

Performance: AutoSurfer significantly outperformed state-of-the-art methods (Explorer, OS-Genesis, SynthAgent).
- Overall Accuracy: AutoSurfer achieved 24.23% overall task completion accuracy, compared to 19.59% for the best prior method (SynthAgent).
- Per-Website Gains: It showed consistent improvements across all tested websites, with notable gains in Reddit (34.62% vs. 19.64% for SynthAgent).
Task Diversity: AutoSurfer generated a more diverse set of tasks (Diversity Score: 0.7774) compared to OS-Genesis and SynthAgent (0.7459), indicating broader coverage of the website's action space.
Refinement Impact: Ablation studies showed that models trained on only exploration data achieved 11.54% accuracy, whereas models trained on refined data achieved 34.62%. This highlights the critical importance of the refinement stage in removing noise and optimizing trajectories.
Generalization: The approach was validated on a different model (Qwen3), showing consistent performance improvements, suggesting the data quality is model-agnostic.

5. Significance

Data Quality over Model Size: The paper demonstrates that high-quality, comprehensive training data can significantly boost the performance of smaller, efficient models (7B-8B parameters), potentially reducing the need for massive, expensive models for web automation.
Privacy and Efficiency: By enabling the training of high-quality wLLMs, AutoSurfer supports the deployment of on-device agents, enhancing user privacy and reducing latency.
Paradigm Shift: It moves the field from "random exploration" or "tutorial-based synthesis" to "systematic, human-like exploration," providing a robust framework for building reliable, general-purpose web agents.

In conclusion, AutoSurfer establishes a new standard for web trajectory generation by proving that comprehensive, systematic exploration combined with trajectory-guided refinement is essential for training accurate and reliable web agents.