💬 NLP

Can Deep Research Agents Retrieve and Organize? Evaluating the Synthesis Gap with Expert Taxonomies

This paper introduces TaxoBench, a benchmark demonstrating that current Deep Research Agents significantly lag behind human experts in both retrieving essential papers and organizing them into coherent taxonomies, revealing distinct bottlenecks in retrieval accuracy and hierarchical structure synthesis.

Original authors: Ming Zhang, Jiabao Zhuang, Wenqing Jing, Kexin Tan, Ziyu Kong, Jingyi Deng, Yujiong Shen, Yuhui Wang, Zhenghao Xiang, Qiyuan Peng, Yuhang Zhao, Ning Luo, Renzhe Zheng, Jiahui Lin, Mingqi Wu, Long Ma

Published 2026-08-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ming Zhang, Jiabao Zhuang, Wenqing Jing, Kexin Tan, Ziyu Kong, Jingyi Deng, Yujiong Shen, Yuhui Wang, Zhenghao Xiang, Qiyuan Peng, Yuhang Zhao, Ning Luo, Renzhe Zheng, Jiahui Lin, Mingqi Wu, Long Ma, Shihan Dou, Maxm Pan, Tao Gui, Qi Zhang, Xuanjing Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to write the ultimate guidebook for a massive, chaotic library. You don't just need to find the books; you need to figure out how to organize them on the shelves so that a reader can actually understand the story of the field. This is the job of a "survey," a type of academic paper that summarizes everything we know about a specific topic. For years, scientists have hoped that Artificial Intelligence (AI) could do this heavy lifting automatically. They imagined "Deep Research Agents"—super-smart AI bots that could scour the internet, find every important paper, and then arrange them into a perfect, logical map of knowledge.

But here's the catch: finding a book is easy; organizing it is hard. To understand if these AI bots are truly ready for the job, we need to test two specific skills. First, Retrieval: Can the AI find the exact same books that a human expert would choose? If the AI misses the most important papers, the guidebook is useless. Second, Organization: Once the AI has the books, can it build a "taxonomy"? Think of a taxonomy as a family tree for ideas. It's not just a list; it's a hierarchy with a main topic at the top, branching out into big categories, then smaller sub-categories, all the way down to individual papers. A good taxonomy helps you see how ideas connect. If the AI just dumps papers into a messy pile or creates a confusing, flat list, it hasn't really done the job.

This paper, titled "Can Deep Research Agents Retrieve and Organize?", puts seven of the world's most advanced AI research agents to the test. The researchers built a special challenge called TaxoBench, which uses 72 real, high-quality surveys written by human experts as the "gold standard." They asked the AI agents to either find the papers from scratch or, in a second test, organize a pre-selected list of papers into a tree structure. The results were a bit of a reality check for the AI world.

The study found that while these AI agents are getting better at talking and writing, they are still struggling with the basics of research. When asked to find the essential papers that experts use, the very best AI agent only managed to find about 20.92% of them. That means it missed nearly 80% of the most important books in the library. The other agents did even worse, with some finding fewer than 5% of the key papers. It's like asking a tour guide to show you the top 10 landmarks in a city, and they only manage to point out two or three, while missing the rest.

The second part of the test was even more revealing. Even when the researchers gave the AI agents the perfect list of papers (so they didn't have to worry about finding them), the agents still failed to organize them like a human expert. The human experts built deep, multi-layered trees with an average depth of 4.86 levels (like a tree with a trunk, big branches, smaller branches, and twigs). The AI agents, however, kept building shallow, flat structures with an average depth of only about 3 levels. They were missing the middle layers of organization.

When the researchers tried to force the AI to build deeper trees by giving it specific instructions, the AI didn't get smarter; it just got messy. It started breaking the tree apart into tiny, single-paper branches, creating a "forest" of isolated twigs instead of a structured tree. This is called "over-segmentation." The AI was so afraid of grouping things together that it ended up putting almost every paper in its own tiny, lonely category.

The paper concludes that there is a massive "synthesis gap" between what AI can do today and what human experts can do. The AI is currently stuck at a "floor" where its organizational skills are barely better than random chance. Even though the AI models are getting more powerful, they haven't closed this gap yet. The researchers suggest that before we can trust AI to write our scientific guidebooks, we need to solve two separate problems: teaching them how to find the right evidence, and teaching them how to build a deep, logical structure to hold that evidence together. Until then, human experts are still the only ones who can truly map the territory of knowledge.

Technical Summary: TaxoBench – Evaluating the Synthesis Gap in Deep Research Agents

1. Problem Statement

Deep Research (DR) agents are increasingly automating the generation of academic surveys by searching the web, retrieving papers, and producing structured overviews. However, existing benchmarks primarily evaluate the quality of the generated text, factuality, or citation correctness, failing to jointly assess two foundational capabilities at the intersection of information acquisition and structural generation:

Retrieval: Can agents retrieve the specific set of papers that human experts consider essential for a survey?
Organization: Can agents organize these papers into a hierarchical taxonomy that mirrors expert-constructed structures?

Current evaluation methods often rely on flat clustering metrics (e.g., ARI, V-Measure) that ignore hierarchical structure or use LLM-judged outlines without grounding in paper-to-category assignments. This creates a "synthesis gap" where it remains unclear whether agents can replicate the expert process of identifying core literature and synthesizing it into a coherent, multi-level taxonomy.

2. Methodology: TaxoBench

The authors introduce TaxoBench, a benchmark designed to isolate and evaluate retrieval and organization capabilities separately.

Dataset Construction

TaxoBench is built from 72 highly cited LLM-related surveys spanning 8 AI/ML subdomains.

Ground Truth: Each survey includes an expert-authored taxonomy (hierarchy of categories) and a set of 3,815 cited papers manually mapped to expert categories by Ph.D.-level annotators.
Statistics: The dataset features an average of 354.5 citations per survey, a hierarchy depth of 4.9, and 14.0 paper categories per taxonomy.

Evaluation Modes

TaxoBench evaluates systems in two distinct modes to separate retrieval from organization:

Deep Research Mode (End-to-End): Agents receive only a survey topic. They must use web tools to retrieve papers and then organize them into a taxonomy. This tests both retrieval and organization simultaneously.
Bottom-Up Mode (Organization Isolation): Agents are provided with the exact set of expert-cited papers ( $P^*$ ). They must organize this fixed set into a taxonomy. This isolates the organization capability, removing retrieval as a variable.

Metrics

The paper proposes a suite of metrics to evaluate both leaf-level assignments and hierarchy-level structures:

Leaf-Level: Adjusted Rand Index (ARI) and V-Measure (Homogeneity/Completeness) to evaluate paper-to-category assignment.
Hierarchy-Level:
- US-TED / US-NTED (Unordered Semantic Tree Edit Distance): Measures global structural divergence between the model's taxonomy and the expert reference, accounting for semantic label similarity (via embeddings) and ignoring sibling order.
- Sem-Path (Semantic Path Similarity): A per-paper metric measuring the consistency of the root-to-leaf path. It aligns ancestor chains and penalizes mismatches in semantic labels and path length.
Calibration: The authors introduce "depth-matching" controls to account for the fact that models often produce shallower trees than experts, which artificially depresses Sem-Path scores. They also establish "floor" baselines (e.g., Flat taxonomies) to interpret raw scores.

3. Key Contributions

TaxoBench Dataset: A benchmark pairing 72 expert taxonomies with 3,815 mapped papers, enabling separate evaluation of retrieval and organization.
Hierarchy-Aware Metrics: Development of US-TED, US-NTED, and Sem-Path, which capture structural and semantic alignment beyond flat clustering. The authors also distinguish between alignment-based findings (agreement with a specific expert reference) and capability-based findings (structural statistics independent of specific labels).
Comprehensive Evaluation: Evaluation of 7 Deep Research Agents and 16 frontier LLM configurations (including o3, GPT-5, Claude, Gemini, etc.) across both modes.

4. Key Results

Retrieval Bottleneck (Deep Research Mode)

Severe Retrieval Deficit: Even the best agent (o3) retrieves only 20.92% of the expert-cited papers (Recall). Four of the seven agents fall below 10% F1.
Correlation with Organization: Retrieval quality strongly correlates with hierarchy-level structure (Spearman $\rho=0.89$ for F1 vs. Sem-Path). Agents that fail to retrieve core papers fail to build coherent hierarchies.
Self-Retrieval: While some agents retrieve the source survey itself, this does not inflate Recall scores because the survey title is not part of the expert paper set.

Organization Bottleneck (Bottom-Up Mode)

Depth Deficit: Models consistently fail to build deep taxonomies. The average expert depth is 4.86, while all 70 model runs (across generations and configurations) average between 2.99 and 3.97. The deepest configuration is still 0.89 levels short.
Over-Segmentation: Models tend to fragment taxonomies into fine-grained clusters (high Homogeneity) rather than consolidating them into broader expert-style categories. For example, GPT-5.6-sol produces 28.3 categories with a 43.5% singleton rate (papers in categories of size 1), compared to the expert's 14.0 categories and 17.0% singleton rate.
The Sem-Path Floor Effect: Raw Sem-Path scores for models cluster tightly (28–30%), barely distinguishable from a "Flat" taxonomy baseline (27.49%). This is due to the lack of chance correction and the depth deficit.
- Depth-Matched Analysis: When models are evaluated against an expert tree collapsed to the model's depth, scores rise to 32–42%, and the spread widens. However, a significant gap remains: humans (annotators) achieve 54.09% Sem-Path on matched surveys, while models average 38.71%.
Prompt Influence: A controlled probe shows that forcing models to match expert depth (via explicit instructions) does not improve organization; instead, it causes further fragmentation (higher singleton rates) and lowers alignment scores (ARI drops, Sem-Path drops). This suggests the depth deficit is a failure to build coherent hierarchical refinement, not an inability to add levels.

Human Baseline

In a matched subset of 10 surveys, two independent human annotators (without seeing the expert reference) achieved a Sem-Path of 53.91% and a depth of ~4.65. They agreed with each other at 50.56% Sem-Path. Models, by contrast, score near the floor (28–30%) and agree with the expert more than humans agree with each other on flat metrics (ARI), indicating ARI is saturated and unreliable for this task.

5. Significance and Claims

The paper claims that TaxoBench identifies retrieval and hierarchical organization as separate, critical bottlenecks in current Deep Research agents.

Retrieval: Agents are currently unable to replicate the literature discovery process of experts, retrieving less than 21% of essential papers.
Organization: Even with perfect retrieval, agents fail to construct deep, coherent taxonomies. They tend to produce shallow, over-segmented structures that lack the progressive refinement found in expert surveys.
Metric Calibration: The authors argue that raw hierarchy metrics (like Sem-Path) are uninterpretable without calibration against depth and floor baselines. They demonstrate that a model generation gaining 3.68 pp in ARI shows almost no movement in raw Sem-Path (0.87 pp), highlighting the metric's insensitivity in the shallow-tree regime.
Future Directions: The work calls for chance-corrected hierarchy metrics, richer agentic workflows (beyond current product interfaces), and multi-reference evaluations to account for the subjectivity of taxonomy design.

The authors conclude that while DR agents can generate text, they currently lack the foundational capabilities to autonomously synthesize expert-level survey structures, necessitating expert oversight rather than full automation.