🤖 AI

GTA: Generating Long-Horizon Tasks for Web Agents at Scale

This paper introduces GTA, a scalable framework that automatically generates realistic, multi-hop web agent tasks with executable trajectories by integrating crawling, retrieval-based seeding, and automated validation to overcome the limitations of existing benchmarks and enable robust training and evaluation.

Original authors: Tenghao Huang, Kung-Hsiang Huang, Prafulla Kumar Choubey, Yilun Zhou, Muhao Chen, Jonathan May, Chien-Sheng Wu

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tenghao Huang, Kung-Hsiang Huang, Prafulla Kumar Choubey, Yilun Zhou, Muhao Chen, Jonathan May, Chien-Sheng Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to use the internet. The robot has a brain (a language model) and hands (tools to click and type), but it's currently very clumsy. It can find simple answers, like "What is the capital of France?" but it struggles with complex missions, like "Find the cheapest flight to Tokyo, check if the hotel has a pool, and see if the flight time works with my doctor's appointment."

The problem is that we haven't been giving the robot enough good practice.

The Problem: Training with a Broken Map

Currently, most tests for these web robots are like giving them a map with only a "Start" and a "Finish" point, but no instructions on how to get there.

Old Benchmarks: These are like hand-written puzzles created by humans. They are few in number, and they often only test simple, one-step tasks.
Automatic Attempts: Some researchers tried to let robots explore websites on their own to create new puzzles. But this was like letting a toddler wander a mall to find toys: they only found the shiny, obvious things (like the toy store) and ignored the rest (like the pharmacy or the food court). It was also incredibly expensive and slow.

Because of this, robots are "overfitting." They memorized the specific puzzles they were given but can't handle real-world, messy situations where they have to jump between different pages and websites to solve a problem.

The Solution: GTA (The "Architect" Approach)

The authors introduce GTA (Generating Long-Horizon Tasks for Web Agents at Scale). Think of GTA not as a teacher handing out worksheets, but as a master architect building a training gym.

Here is how they built it, using simple steps:

The Crawl (Mapping the City): Instead of letting the robot wander blindly, GTA first sends out a fleet of digital "scouts" to map out entire websites (like e-commerce stores, government sites, and news portals). They build a complete "city map" of every page and how they connect.
The Seed (Picking the Challenge): Instead of guessing what a hard task looks like, GTA picks two random locations on the map (e.g., a page about a specific shoe and a page about a specific discount).
The Generation (Writing the Mission): An AI is asked to write a mission that requires visiting both locations to solve. For example: "Find the price of the shoe on Page A, then check if Page B has a coupon that makes it cheaper."
The Quality Control (The Referee): Before the mission is released, a strict referee checks it.
- Is it solvable? (Can you actually get the answer?)
- Is it clear? (Is there only one right answer?)
- Is it multi-hop? (Does it force the robot to visit more than one page?)
The Gold Path (The Answer Key): Crucially, GTA records the exact path the robot should take. This allows researchers to replay the mission perfectly and see exactly where the robot went wrong.

Why This Matters: The "Human vs. Robot" Gap

The authors tested this new gym with current robots and found a massive gap:

The Robots: When faced with these new, multi-step tasks, the robots failed most of the time (often scoring below 20%). They got lost, gave up too early, or tried to use the search bar instead of navigating the site properly.
The Humans: When humans tried the same tasks, they solved them easily (85% success rate).
The Search Engine: Even a simple Google search couldn't solve these tasks because the answer wasn't in one place; it was hidden across different pages.

Key Features of GTA

It's a "Living" Gym: Unlike old tests that are static (frozen in time), GTA can keep generating new tasks from live websites. This prevents robots from just memorizing the answers.
It's Global: It works in many languages (English, Italian, Japanese, German, Chinese), not just English. The robots struggled even more with non-English sites.
It's Cross-Website: Some tasks require the robot to jump from a health site to a finance site to solve a problem, mimicking how real people use the web.

The Bottom Line

GTA is a new, massive, and automated system for creating realistic, difficult training missions for web robots. It proves that current robots are still quite weak at navigating the complex, multi-step nature of the real internet. By providing a way to generate endless, high-quality, and verifiable challenges, GTA helps researchers figure out exactly where robots are failing so they can build better ones.

What it is NOT:

It is not a tool for doctors or businesses to use right now.
It does not claim to have solved the problem of web agents; it just provided a better way to test them and showed how much work is left to do.
It does not cover tasks that require logging into private accounts or making real purchases (due to safety rules).

Technical Summary: GTA – Generating Long-Horizon Tasks for Web Agents at Scale

Problem Statement

Web agents, which integrate language models with browsing and tool-use capabilities, face a critical bottleneck in scalability due to the lack of process-level supervision. Existing benchmarks are predominantly manually constructed, offering only coarse start–goal annotations without intermediate trajectories. This limits the ability to train and evaluate agents on the latent decision processes required for realistic, multi-hop, cross-page navigation.

Recent attempts at automatic task generation suffer from three primary limitations:

High Cost and Latency: Methods relying on repeated agent–environment rollouts (e.g., AgentTrek, NNetNav) are prohibitively expensive and slow.
Exploration Bias: LLM-driven policies tend to overfit to "obvious" paths (e.g., product detail pages), leaving significant portions of the site graph unexplored.
Lack of Compositionality: Many generated tasks collapse into single-hop retrieval or simple form-filling, failing to capture the multi-hop dependencies and open-ended objectives characteristic of real-world user workflows.

Methodology: The GTA Framework

The authors introduce GTA (Generating Long-Horizon Tasks for Web Agents at Scale), a scalable framework designed to produce realistic, graded-difficulty tasks paired with executable ground-truth trajectories. The pipeline decouples crawling from generation to improve efficiency and ensure coverage.

1. Benchmark Construction Pipeline

The framework operates in four distinct stages:

Crawling and Graph Construction: The system performs breadth-first crawling on publicly accessible domains to construct a directed web graph $G = (V, E)$ , where nodes represent pages and edges represent hyperlinks. A canonical shortest-path policy is enforced to prevent cycles and ensure unique parent assignments.
Retrieval-Based Seeding: Instead of policy-driven exploration, the system embeds nodes using an LLM-based encoder. For a seed node, it retrieves a candidate set of semantically related pages.
In-Context Generation: A generator LLM is prompted with content from a seed node and a retrieved secondary node to synthesize a query. This strategy is designed to enforce multi-hop dependencies by requiring reasoning across two distinct pages.
Automated Quality Control: A multi-stage verification protocol filters generated tasks:
- Multi-hop Validation: Rejects tasks that are merely conjunctions of independent single-hop queries.
- Answer Correctness: A verifier LLM checks if the answer can be derived solely from the supporting nodes.
- Ambiguity Detection: Filters queries lacking unique answers (e.g., missing temporal constraints).
- Solvability Check: Ensures a minimal executable path exists within the crawled graph to derive the answer.

2. Task Definition

A task is defined as multi-hop of order $n$ if solving it requires aggregating evidence from at least $n$ distinct webpages ( $n > 1$ ). The benchmark supports both intra-website tasks (reasoning within a single domain) and inter-website tasks (coordinating across multiple domains).

3. Scope and Scale

The pipeline was instantiated on over 50 public websites covering e-commerce, government, forums, news, healthcare, and finance. The resulting dataset includes:

5,000 intra-website tasks.
600 cross-website inter-website tasks.
Multilingual coverage (English, Italian, Japanese, German, Chinese).
A self-evolving ecosystem where users can generate up-to-date tasks from live web content.

Key Contributions

The paper outlines three primary contributions:

Problem Formulation: Formalizing the task of multi-hop web-agent generation, distinguishing it from trivial information lookup by requiring reasoning across multiple pages or domains.
Efficient Automatic Data Generation: Proposing a pipeline that significantly reduces per-task time and cost by separating crawling from generation. Each task includes a minimal, programmatically executable "gold path" for deterministic replay and step-level diagnostics.
Evolving Benchmark Ecosystem: Framing GTA not as a static dataset but as a self-evolving system. This allows for continuous refresh of evaluation data, reducing overfitting to static benchmarks and exposing agents to changing, realistic task distributions.

Results and Analysis

1. Benchmark Quality and Comparison

Query Structure: Unlike prior datasets (e.g., WebDS) that often consist of factual lookups, GTA tasks require integrating heterogeneous evidence (e.g., combining numerical stats with safety constraints).
Search Baseline Failure: When evaluated against a simple Google Search API baseline, GTA tasks yielded a correctness rate of only 14%, compared to 95–100% for AgentTrek and NNetNav. This confirms GTA tasks cannot be solved by single-hop retrieval.
Page Coverage: GTA achieves a page coverage rate of 77.4%–80.7% on test sites, significantly outperforming AgentTrek (20.2%), NNetNav (24.3%), and WebDS (11.0%), indicating broader exploration of site functionalities.
Human Evaluation: Human annotators rated GTA tasks as substantially more complex (2.35/3) than WebDS (1.90) and AgentTrek (1.23). GTA also achieved the highest overall quality score (1.82), balancing realism, reasoning depth, and utility.

2. Agent Performance

Experiments with Browser Use and AgentOccam agents revealed significant performance gaps:

Intra-Website Tasks: Agents achieved success rates often below 20–30% on GTA, a sharp decline compared to their performance on WebVoyager (up to 82%) and Mind2Web.
Inter-Website Tasks: Performance dropped further for cross-domain tasks (e.g., 4.2% success rate for Drugs.com & CVS Health integration), highlighting difficulties in managing distributional shifts and non-overlapping knowledge.
Multilingual Performance: Agents exhibited severe limitations in non-English environments, with Browser Use achieving 0% success on German and Italian sites, and only 2.9% on Japanese sites.

3. Error Analysis

Analysis of 100 failed tasks identified three dominant failure modes:

Failure to Reach Required Pages (90%): Agents failed to navigate to all necessary nodes, often due to brittle link-following or navigation loops.
Early Stopping (40%): Agents terminated tasks prematurely after retrieving partial evidence without verifying consistency across all sources.
Over-reliance on Search Boxes (40%): Agents frequently bypassed explicit web graph structures in favor of on-site search, leading to irrelevant results and broken reasoning chains.

Significance and Claims

The paper posits that GTA addresses the critical need for scalable, process-level supervision in web agent research. By decoupling crawling from generation and grounding tasks in site graphs, GTA produces compositional, verifiable tasks that expose the limitations of current agents in multi-hop reasoning and cross-domain integration.

The authors claim that GTA serves as a foundation for advancing robust, generalizable web agents by:

Revealing a significant human–agent performance gap in realistic, multi-hop scenarios.
Enabling detailed diagnostics of agent failures (e.g., navigation vs. reasoning).
Providing a mechanism to reduce overfitting through dynamic, self-evolving task generation.

The work concludes that while current agents struggle with the complexity of real-world web navigation, the GTA framework offers a rigorous, automated, and scalable path forward for evaluating and improving these systems.