A Benchmark for Deep Information Synthesis

1. Problem Statement

Current Large Language Model (LLM) agents are increasingly capable of tool use (web browsing, code execution) but are evaluated primarily on shallow fact retrieval or single-source information seeking. Existing benchmarks (e.g., GAIA, BrowseComp) often fail to assess an agent's ability to:

• Synthesize information from multiple, heterogeneous sources (structured tables, unstructured text, different domains).
• Perform deep reasoning to infer insights that are not explicitly stated in a single document.
• Operate effectively in real-world, time-consuming scenarios involving global data diversity and non-English contexts.
• Avoid memorization by ensuring answers cannot be retrieved via simple verbatim lookup.

The authors argue that there is a critical gap in evaluating agents on tasks that require planning, multi-step navigation, and the integration of disparate data points to form coherent, verifiable insights.

2. Methodology: The DEEPSYNTH Benchmark

A. Task Design and Criteria

DEEPSYNTH consists of 120 challenging tasks designed to mimic real-world analytical problems. The tasks adhere to five strict criteria:

1. Multi-source Synthesis: Agents must combine data from multiple sources (e.g., correlating economic data with health statistics).
2. Real-world Inspiration: Tasks are derived from scenarios faced by policymakers, travel agents, and researchers (e.g., "Which non-ASEAN countries recovered to 95% of 2019 visitor levels by 2023?").
3. Verifiable Answers: Answers are closed-form (JSON/Dictionary) to allow for automated, stable evaluation, avoiding subjective open-ended grading.
4. Diversity: Tasks span 67 countries and 7 domains (socio-economic, finance, environment, science, education, transportation, political).
5. Robustness: Tasks are constructed to be non-retrievable via simple search or pre-training memorization, forcing multi-step reasoning.

B. Data Collection Pipeline

The dataset was constructed using a rigorous four-stage expert-driven pipeline:

1. Data Source Identification: 16 experts curated 223 official, trustworthy data sources (e.g., government statistics, Eurostat).
2. Hypothesis Generation: Experts formulated plausible, insightful hypotheses based on these sources (e.g., "Is there a linear relationship between air quality and pneumonia deaths?").
3. Hypothesis Validation: Experts manually analyzed the data to verify hypotheses and derive tasks with verifiable answers. Sources failing verifiability were discarded.
4. Task Formulation: Experts created the final questions, intermediate reasoning steps, supporting evidence (URLs), and gold-standard answers.
5. Validation: A second independent annotation round ensured 100% agreement on answers, resulting in the final 120 tasks.

C. Task Characteristics

• Complexity: Average task requires navigating 4.2 web pages and reading 1–15 documents/tables.
• Reasoning Depth: Average of 7.54 intermediate reasoning steps per task.
• Operations: Tasks involve trend detection, correlation analysis, anomaly detection, ranking, and counting/comparing.
• Annotation Cost: Approximately 5.5 hours per task to create.

3. Evaluation Setup

Models Tested

The benchmark evaluated:

• State-of-the-Art LLMs: GPT-4.1, GPT-5 (various versions), Gemini-2.5-Pro, DeepSeek-R1, o4-mini.
• Agentic Frameworks:
  • o3-deep-research: OpenAI's specialized deep research agent.
  • smolagents: A minimalist framework using ReAct loops and code execution.
  • OWL: A multi-agent framework with planner/executor roles.

Metrics

• Exact Match (EM): Strict requirement for all keys and values to be correct.
• F1 Score / Precision / Recall: Based on correct key-value pairs in the JSON output.
• LLM-Judge Score: A "soft" metric using GPT-4.1 to evaluate semantic equivalence and allow for small numerical margins (1–5.5%), capturing partial correctness.

4. Key Results

A. Overall Performance (The "Difficulty" Gap)

The benchmark proved extremely difficult for current state-of-the-art systems:

• Best LLM: Gemini-Pro-2.5 achieved the highest F1 score of 6.25 (LLM-Judge: 5.0).
• Best Agent: o3-deep-research achieved the highest F1 score of 8.97 (LLM-Judge: 17.5).
• Exact Match: Almost zero tasks were solved perfectly (EM = 0.0) by most models, indicating a total failure to produce structurally and factually exact outputs.
• Conclusion: Even the most advanced reasoning models struggle significantly with deep information synthesis.

B. Analysis of Failure Modes

1. Navigation & Synthesis Errors: The primary failure modes were Navigation Errors (15/32 cases, e.g., wrong URL, failed access) and Synthesis Errors (16/32 cases, e.g., correct data retrieved but incorrect logical conclusion).
2. Planning Deficit: Providing agents with ground-truth intermediate steps (planning) significantly improved performance (e.g., GPT-4.1 F1 jumped from 3.46 to 9.36), suggesting current agents lack robust planning capabilities.
3. Geographical Bias: Performance dropped to 0.0 F1 on tasks related to Africa, highlighting severe data and model biases against under-represented regions.
4. Error Propagation: Early retrieval failures (Steps 1–2) led to near-total error propagation (91–100%) in subsequent reasoning steps. If the agent fails to find the right data initially, it cannot recover.
5. Tool Usage: Agents with tool augmentation (web search + code interpreter) outperformed base LLMs, but the gains were marginal (e.g., +5.68 F1 for o3-deep-research vs. base o3), indicating that tool use alone is insufficient without better reasoning integration.

5. Key Contributions

1. DEEPSYNTH Dataset: Release of a high-quality, 120-task benchmark covering 67 countries and 7 domains, specifically designed to test deep information synthesis rather than shallow retrieval.
2. Empirical Evidence of Limitations: Demonstration that current SOTA agents (including GPT-5 and DeepSeek-R1) fail to solve even a small fraction of these tasks, with the best agent achieving less than 9% F1.
3. Diagnostic Analysis: Identification of specific bottlenecks:
   • Inability to plan multi-step reasoning chains.
   • Fragility in navigating diverse, non-English, or region-specific data sources.
   • High variance in output (low consistency) and inability to recover from early retrieval errors.
4. Methodological Framework: A rigorous, expert-driven data collection pipeline that ensures tasks are verifiable, non-retrievable, and grounded in real-world utility.

6. Significance

DEEPSYNTH serves as a crucial stress test for the next generation of AI agents. It shifts the evaluation paradigm from "can the model find a fact?" to "can the model synthesize a complex, multi-source insight?" The results indicate that while LLMs have advanced in reasoning, they are not yet reliable for real-world analytical tasks requiring deep synthesis. The benchmark highlights the urgent need for research into:

• Robust navigation and error recovery in web browsing.
• Improved planning and decomposition of complex tasks.
• Mitigation of geographical and linguistic biases in training data and retrieval systems.
• Better integration of external tools with internal reasoning processes.

The paper concludes that DEEPSYNTH will guide future research toward building agents capable of genuine, goal-directed problem solving in complex, information-rich environments.