Technical Summary: WebRetriever

Problem Statement

The rapid advancement of Large Language Models (LLMs) has driven the emergence of multimodal web agents capable of automating complex online tasks. However, the evaluation of these agents faces three critical limitations that hinder their reliable deployment:

1. Insufficient Scale and Diversity: Existing benchmarks suffer from limited website counts, narrow domain coverage, and a lack of diverse user intents, failing to capture the breadth required for real-world generalization.
2. Inadequate Evaluation Methodologies: Current automated evaluation methods, particularly those relying on "LLM-as-a-Judge," often fail to capture fine-grained interaction semantics. They frequently depend on visual screenshots alone, missing critical intermediate interaction data (e.g., network requests, query formulation) necessary for precise assessment.
3. Narrow Evaluation Protocols: Prevailing benchmarks predominantly measure navigation success (reaching a target page) while neglecting the ability to leverage external knowledge or perform end-to-end tasks involving information extraction. This creates a gap where high benchmark scores do not translate to practical application effectiveness.

Methodology

To address these limitations, the authors introduce WebRetriever, a large-scale benchmark, and NavEval, a novel automated evaluation framework.

1. WebRetriever Benchmark

WebRetriever is constructed to provide a comprehensive, realistic evaluation environment:

• Scale and Diversity: It encompasses 1,550 tasks across 800 real-world websites.
• Domain Coverage: The dataset spans eight core industry sectors (Technology, Finance, Education, Culture, Lifestyle, Healthcare, Industry, Public Services) and covers both General and Professional user intents.
• Task Design: Tasks are categorized by difficulty (Easy, Medium, Hard) based on the number of required action steps. They are designed to test not just navigation but also the understanding of underlying business logic.
• Maintenance: The dataset is designed for continuous maintenance, replacing obsolete tasks to ensure long-term validity.

2. NavEval: Automated Evaluation Framework

NavEval is an LLM-as-a-Judge framework designed to overcome the limitations of screenshot-only evaluation:

• Multi-Source Signal Integration: Unlike prior methods that rely on final screenshots or full trajectories, NavEval integrates task descriptions, action trajectories, filtered network requests, and the final screenshot.
• Rule-Based Filtering: A preprocessing step filters raw network requests to remove noise, normalize payloads, and retain only relevant subdomain-level interactions. This creates a compact, structured representation of the agent's query and filtering operations.
• Judgment Process: An LLM (specifically Claude-4.5-Sonnet in the experiments) analyzes these combined signals to determine binary task success (True/False), achieving high alignment with human judgment.

3. Three Evaluation Protocols

The benchmark employs three complementary protocols to holistically assess agent capabilities:

• Protocol I (Navigation Proficiency): Evaluates the agent's ability to reach a target page based solely on a user task description.
• Protocol II (Knowledge-Assisted Interaction): Assesses navigation performance when the agent is provided with structured operational documentation (simulating the use of external knowledge bases).
• Protocol III (End-to-End Task Completion): Measures the ability to navigate to a page and accurately extract specific information (text, documents, charts) to fulfill the query. This protocol avoids the pitfall of equating page arrival with task success.

Key Results

Extensive experiments were conducted on WebRetriever using state-of-the-art agents (e.g., SeeAct, Browser-Use, UI-TARS-1.5, Agent-E, Claude-4.5, Gemini-2.5-Pro).

• Performance Disparities: Human evaluation reveals that current web agents perform poorly on WebRetriever compared to existing benchmarks. The average human-assessed success rate drops significantly across protocols:
  • Protocol I: ~21.1%
  • Protocol II: ~29.2% (with operational documentation)
  • Protocol III: ~11.8% (end-to-end with extraction)
    This indicates that navigation success is an insufficient predictor of real-world effectiveness, particularly when information extraction is required.
• Impact of Operational Knowledge: Providing operational documentation (Protocol II) improves success rates by approximately 8-9%, demonstrating that agents struggle with unfamiliar websites without guidance.
• Evaluation Reliability: NavEval achieves a Human Agreement Rate (AR) of ~91.2%, significantly outperforming existing automated methods (which range from 55% to 81%). This high agreement holds across different agents and is validated on an external benchmark (Online-Mind2Web), where NavEval achieved a 97% AR.
• Ablation Insights: The inclusion of rule-based filtering in NavEval improves agreement by ~3.4 percentage points. Furthermore, the ablation study on Protocol III reveals that even when agents successfully navigate to target pages, their ability to extract correct information is unreliable, with success rates dropping by nearly half when extraction is required.

Significance and Contributions

The paper claims the following contributions to the field of web agent research:

1. A Large-Scale Benchmark: WebRetriever provides unprecedented scale, diversity, and coverage, enabling a more representative assessment of web agents in realistic online environments.
2. High-Precision Automated Evaluation: NavEval offers a scalable, reliable evaluation method that achieves near-human agreement, addressing the bottleneck of costly human annotation.
3. Comprehensive Evaluation Framework: The three proposed protocols disentangle navigation success from answer correctness and characterize behavioral reliability under knowledge injection. This framework provides diagnostic insights missing from prior benchmarks, revealing that current agents are far from reliably handling practical, end-to-end web tasks.

The authors conclude that WebRetriever establishes a rigorous foundation for advancing web agent development, highlighting the urgent need for benchmarks that reflect the complexity of real-world deployment scenarios.