WebAggregator: Enhancing Compositional Reasoning Capabilities of Deep Research Agent Foundation Models

1. Problem Statement

Current Deep Research agents suffer from a fundamental imbalance: they are retrieval-heavy but reasoning-light.

• The Gap: Existing systems excel at "entity-seeking" (finding a specific fact on a page) but fail at compositional reasoning—the ability to aggregate distributed, heterogeneous information from multiple sources into coherent logical insights.
• Limitations of Current Benchmarks: Existing datasets (e.g., WebWalkerQA, TaskCraft) often rely on simple multi-hop retrieval or static entity localization. They lack the environmental complexity (dynamic content, files, visual elements) and the depth of reasoning required for real-world scientific discovery.
• The Core Issue: Even when agents successfully retrieve all necessary reference documents, they often fail to synthesize the information correctly due to deficits in logical aggregation, error accumulation, and ineffective self-refinement.

2. Methodology: The WebAggregator Framework

The authors propose WebAggregator, an automated data synthesis pipeline designed to shift the agentic paradigm from retrieval-centric to compositional aggregation. The framework consists of two primary phases:

A. Proactive Explorer

• Function: An autonomous agent that navigates real web environments starting from "anchor URLs."
• Capabilities: It utilizes a comprehensive toolkit including search, static page parsing, dynamic interaction (clicks, scrolls, JS handling), file processing (PDFs, CSVs), and visual perception (screenshots).
• Goal: To gather interconnected, diverse knowledge resources (text, figures, files) rather than just single-page facts. The agent is forced to visit a minimum number of pages to ensure knowledge breadth.

B. Compositional Logic Proposer

• Function: Transforms the raw, unstructured information collected by the Explorer into complex, verifiable QA pairs.
• Taxonomy: The system employs a rigorous taxonomy of 12 granular logical operations derived from a deconstruction of Deep Research problems, categorized into four high-level types:
  1. Element Operations: Mathematical calculations, inverse questions, and specific entity retrieval.
  2. Set Operations: Filtering, intersection, union, sorting, and counting across datasets.
  3. Scientific Analysis: Computation-intensive tasks requiring statistical analysis (e.g., standard deviation, correlation coefficients), trend prediction, and batch data processing.
  4. Temporal Reasoning: Analyzing changes over time and calculating intervals.
• Synthesis: The Proposer instantiates these high-level guidelines into concrete, multi-hop questions that cannot be answered by direct extraction but require synthesizing data from multiple sources.

C. Quality Control & Data Curation

• Two-Stage Verification:
  1. Self-Refinement: The Proposer uses a checklist to revise questions for clarity and self-containment.
  2. Agent Verification: A separate agent attempts to solve the generated tasks to verify that the answer is derivable from the provided URLs and that the logic holds.
• Rejection Sampling: The pipeline generates a large volume of trajectories, which are then filtered for correctness, format, and exception handling. This results in a high-quality dataset of ~10,000 QA pairs (WebAggregatorQA) based on 50,000+ unique URLs across 12 domains.

3. Key Contributions

1. WebAggregatorQA Dataset: A new benchmark specifically designed to test Compositional Reasoning (CR) alongside Information Seeking (IS). Unlike prior benchmarks, it requires agents to perform complex operations (e.g., calculating the standard deviation of GDP across filtered countries) even when all reference data is provided.
2. WebAggregator Models: A family of fine-tuned foundation models (based on Qwen2.5 and Qwen3) trained on the WebAggregatorQA corpus.
3. New Benchmark (WebAggregatorQA Testbed): A manually annotated test set (159 samples) that reveals a critical finding: accurate retrieval does not guarantee task success. Even top-tier models like Claude-3.7-Sonnet score poorly (28.3%) on this test set, highlighting a severe bottleneck in reasoning rather than retrieval.
4. Taxonomy of Failure Modes: A detailed analysis categorizing agent failures into:
   • Information Seeking Failures (cannot find sources).
   • Foundation Capability Failures (tool misuse, context limits).
   • Compositional Reasoning Failures (faulty logic, error accumulation, ineffective self-refinement).

4. Experimental Results

The authors evaluated their models on GAIA-text, WebWalkerQA, XBench, and the new WebAggregatorQA.

• Performance Gains:
  • WebAggregator-32B surpasses GPT-4.1 and matches Claude-3.7-Sonnet on GAIA, WebWalkerQA, and XBench.
  • On the challenging WebAggregatorQA test set, the 32B model achieves a Pass@3 score of 28.9%, significantly outperforming zero-shot baselines (e.g., GPT-4.1 at 25.8% and Claude-3.7-Sonnet at 28.3% on the full test set, though the paper notes the difficulty gap).
  • Small Models: Even the 8B variant shows parity with GPT-4.1 on GAIA-text, demonstrating high data efficiency.
• Behavioral Shifts:
  • Reduced Tool Redundancy: Fine-tuned models exhibit a lower "Tool Call Density" (fewer tool calls per step) but higher "Reasoning Density."
  • Deeper Synthesis: The models shift from mechanical retrieval to internal logical synthesis. They spend more tokens on "Type B" reasoning (deduction, error correction) rather than "Type A" procedural planning.
  • Tool Diversity: Post-fine-tuning, agents utilize a more diverse and balanced set of tools per trajectory, moving away from simple search/visit loops.

5. Significance and Conclusion

• Paradigm Shift: The paper argues that the future of Deep Research agents lies not in better retrieval, but in compositional reasoning. Retrieval is a prerequisite, but the "performance ceiling" is determined by the ability to aggregate and reason over scattered evidence.
• Benchmarking: It introduces a necessary standard for evaluating the reasoning capabilities of agents, moving beyond simple fact-checking to complex, multi-step logical synthesis.
• Future Direction: The work suggests that scaling model size alone is insufficient; targeted fine-tuning on reasoning-intensive, verifiable data is crucial for achieving autonomous scientific discovery. The authors open-source the code, data, and models to facilitate further research in this domain.

In summary, WebAggregator demonstrates that by synthesizing data that forces agents to perform complex logical aggregation, we can fundamentally transform agent behavior, reducing reliance on brute-force tool usage and enabling deeper, more accurate reasoning in complex web environments.