AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web

1. Problem Statement

The paper addresses the emerging paradigm of the Agentic Web, where autonomous AI agents act on behalf of users to access information. Unlike the traditional web where users directly browse open pages, the Agentic Web is shifting toward a decentralized ecosystem. In this setting:

• Content Providers restrict direct access to their data, deploying their own Content Agents to manage and serve information via controlled interfaces (e.g., APIs, Model Context Protocol).
• User Agents must coordinate with these autonomous Content Agents to retrieve and synthesize information.
• The Challenge: This shift moves information access from centralized retrieval (where a single engine indexes the whole web) to decentralized coordination. It remains unclear how effective multi-agent coordination is compared to centralized retrieval, what the systemic impacts are (e.g., traffic concentration), and where the bottlenecks lie in this new architecture.

2. Methodology: AgentWebBench

The authors introduce AgentWebBench, a comprehensive benchmark designed to evaluate multi-agent coordination in this decentralized environment.

A. Architecture

The benchmark simulates a decentralized ecosystem with two distinct agent types:

1. User Agent: Acts on behalf of the user. It has no direct access to the document corpus. It must analyze user intent, select relevant websites, query Content Agents, and synthesize final answers.
2. Content Agents: Autonomous agents assigned to specific website domains (100 websites total). Each maintains its own private retrieval index and can only retrieve documents within its own domain.

B. Dataset & Environment

• Corpus: Built from the ClueWeb22-B dataset, selecting the 100 websites with the largest document counts.
• Scale: Approximately 18.4 million documents across diverse domains (e-commerce, news, academic, social media, etc.).
• Retrieval Infrastructure: Each Content Agent uses an independent dense retrieval index (based on MiniCPM-Embedding-Light) to simulate realistic, domain-specific access constraints.

C. Tasks

The benchmark evaluates four common web information needs:

1. Web Search: Ranked retrieval of documents for a query (354 samples).
2. Web Recommendation: Inferring user intent from browsing history to recommend the next document (281 samples).
3. Question Answering (QA): Synthesizing concise answers from multi-source evidence (53 samples).
4. Deep Research: Generating comprehensive reports via iterative planning and synthesis (331 samples).

D. Coordination Strategies

The study compares three strategies to isolate the impact of reasoning and communication:

1. ToolE (Tool-based Embedding): Selects websites via embedding similarity; retrieves documents via tools. No agent-to-agent communication.
2. ToolP (Tool-based Prompted): Uses LLM reasoning to select websites; retrieves via tools. No agent-to-agent communication.
3. Multi-Agent: Enables full agent-to-agent communication. The User Agent and Content Agents reason autonomously, allowing for iterative refinement and adaptive information gathering.

E. Models Evaluated

Seven advanced Large Language Models (LLMs) were tested, including various sizes of the Qwen3 series (4B to 80B), DeepSeek-V3.2, GPT-5, and Gemini-3.

3. Key Contributions

1. First Comprehensive Benchmark: AgentWebBench is the first benchmark specifically designed to evaluate agent performance in a decentralized, multi-agent web ecosystem across four distinct task types.
2. Decentralized vs. Centralized Analysis: It provides a rigorous comparison showing that while decentralized coordination generally lags behind centralized retrieval, the gap narrows significantly with model scale and can even reverse for specific tasks like QA.
3. Ecosystem Insights: Beyond task accuracy, the benchmark characterizes the emerging paradigm by analyzing:
   • Traffic Concentration: How agents filter sources.
   • Test-Time Scaling: The impact of "thinking" modes.
   • Failure Modes: Distinguishing between User Agent planning errors and Content Agent retrieval failures.

4. Key Results & Findings

A. Performance Gap and Model Scaling

• General Lag: Decentralized multi-agent coordination generally underperforms centralized retrieval (where the agent has direct corpus access) because the User Agent cannot directly access the full corpus.
• Scaling Law: The performance gap shrinks as model scale increases. Stronger models (e.g., Qwen3-80B, GPT-5, Gemini-3) significantly improve coordination capabilities.
• Task-Dependent Reversal: On Question Answering, Multi-Agent coordination can outperform centralized retrieval. This is attributed to the iterative nature of evidence gathering, which fits the agentic setting better than single-pass retrieval.

B. Coordination Strategy Analysis

• Website Selection: LLM reasoning (ToolP) is superior to embedding similarity (ToolE) for selecting relevant websites.
• Content Agent Stability: Content agents are less stable for retrieval-heavy tasks (like Web Search) compared to tool-based retrieval. However, in generative tasks (Deep Research), the performance gap between ToolP and Multi-Agent narrows.

C. Systemic Impact: Traffic Concentration

• Filter Effect: Decentralized access acts as a strong filter. Multi-agent coordination tends to concentrate traffic on a small subset of high-authority websites (e.g., Wikipedia, ScienceDirect).
• Diversity Loss: This reduces the discoverability of smaller or niche content providers compared to centralized baselines, potentially limiting information diversity.

D. Efficiency and Test-Time Scaling

• Thinking Mode: Enabling "thinking" (reasoning before acting) significantly improves both interaction reliability (fewer malformed requests) and task performance, particularly for complex tasks like Web Search and QA.
• Interaction Budget: Effective coordination requires sufficient interactions. Stronger models (Gemini-3) use more agents and requests to achieve better results, suggesting that efficiency is not just about minimizing turns but about strategic exploration.

E. Failure Analysis

• User Agent Bottlenecks: Failures in Question Answering are predominantly attributed to the User Agent (poor planning, incorrect synthesis, or misinterpreting evidence granularity).
• Content Agent Bottlenecks: Failures in Web Search are often due to Content Agents failing to retrieve the specific ground-truth document even when the correct website is selected (low recall).
• Synthesis Errors: Even with correct evidence, User Agents often fail to synthesize the correct answer (e.g., selecting "The Sun" instead of "Light" in a Plato analogy).

5. Significance and Future Directions

• Paradigm Shift: The paper highlights a fundamental shift in how the web is accessed, moving from open crawling to controlled, agent-mediated interactions.
• Research Roadmap: It identifies clear paths for improvement:
  • User Agents need better planning, intent inference, and answer synthesis capabilities.
  • Content Agents need more reliable retrieval mechanisms and higher-quality evidence verification.
  • Coordination Strategies must balance interaction costs with information quality, potentially using adaptive budgeting.
• Policy Implications: The findings warn that the Agentic Web could lead to traffic concentration, where a few dominant platforms capture most user attention, potentially stifling the diversity of the open web.

In conclusion, AgentWebBench establishes that while the Agentic Web faces effectiveness challenges due to access constraints, the paradigm is viable and improving with model advancements. It provides a critical foundation for developing robust, safe, and efficient multi-agent systems for the future internet.