Technical Summary: From Simple QA to Deep Research

Problem Statement

Deep research tasks require AI agents to move beyond simple fact retrieval to integrate domain knowledge, perform multi-step reasoning, and produce high-quality, open-ended responses. However, constructing reliable benchmarks for these tasks is challenging. Existing benchmarks often rely on expensive expert authoring or pre-existing human materials, limiting scalability. Conversely, fully automatic construction methods struggle to ensure consistent, traceable verification and often lack the task-specific knowledge required for fine-grained evaluation. Current automatic approaches either compare models relatively (ranking based on report similarity) or generate evaluation rubrics that lack guaranteed reliability and professional quality. The core challenge addressed is how to build a diverse collection of professionally deep research tasks with reliable, grounded rubrics without relying on manual expert authoring.

Methodology

The paper introduces a fully automatic benchmark construction pipeline centered on an iterative Explorer–Formalizer–Challenger loop. This pipeline progressively transforms simple questions into complex deep research tasks.

1. Task Representation

The core structural representation of a task is a Directed Acyclic Graph (DAG).

• Nodes: Represent atomic research steps, classified as either evidential (retrieving traceable facts) or analytical (performing inference or comparison).
• Edges: Represent logical dependencies between steps.
• Checkpoints: Each node includes verifiable checkpoints derived from the agent's interaction with the environment.
  This structure allows the solving process to be explicit, decomposable, and traceable.

2. The Construction Pipeline

The pipeline iterates through three roles to evolve a simple query $q_1$ into a deep research task $q_T$:

• Explorer: Solves the current query in an open environment using tools (e.g., search, web scraping). It produces a trajectory of explored information, evidence, and analysis. This step is designed to uncover long-tail, sparse knowledge that becomes discoverable only when search intents are refined.
• Formalizer: Parses the Explorer's trajectory to update the task DAG ($G$) and derive a corresponding set of verifiable rubrics ($R$).
  • It organizes the trajectory into a DAG, ensuring the graph is sufficient to answer the query but minimal (removing irrelevant nodes and merging semantically equivalent ones).
  • It generates pointwise rubrics for each node, grounded in the specific evidence found.
  • It assigns weights to nodes based on the DAG structure (propagating weights from leaves to roots) to reflect the relative importance of different research steps.
• Challenger: Identifies explored but unused clues in the trajectory that remain logically connected to the task. It uses these "unused" directions to generate a harder query for the next round ($q_{t+1}$), effectively expanding the task's scope (width) or reasoning depth. Crucially, it masks specific checkpoints to prevent the new query from revealing the solution path.

Stopping Criterion: The evolution stops when the DAG structure (nodes and edges) remains unchanged for two consecutive rounds, indicating that the task has reached a saturation point where no further relevant knowledge can be integrated.

3. Query Design

From the final converged task, three distinct query forms are generated to probe complementary capabilities:

• Query with Hints ($q^h_T$): The full, complex query containing analysis dimensions and constraints. Tests information processing under rich clues.
• Query without Hints ($q^o_T$): A concise, everyday question derived from the complex query, stripping away constraints. Tests decomposition and planning under hidden constraints.
• Assigned-Topic Query ($q^a_T$): A declarative research title. Tests argument formation under a directional topic without explicit question framing.

Key Contributions

1. A Verifiable Benchmark: The authors constructed a benchmark of 500 deep research tasks spanning 31 topics and 10 major categories. Each task is paired with fact-grounded pointwise rubrics and the three query forms mentioned above.
2. Fully Automatic Construction Pipeline: They introduced a novel pipeline that iteratively expands simple queries into DAGs of atomic steps. This approach enables the query, task structure, and rubrics to evolve together without manual authoring or pre-written expert materials.
3. Fine-Grained Evaluation: The benchmark demonstrates that its rubrics enable stable, human-aligned, and fine-grained evaluation, distinguishing model performance more effectively than binary judgments or relative rankings.

Results

Experiments were conducted on 10 popular models (including GPT-5.6, Claude Sonnet 5, DeepSeek-V4-Pro, and various Qwen versions) under two settings: Agent Mode (with tools) and Model Mode (internal knowledge only).

• Discrimination: The benchmark clearly discriminates between models of varying capabilities. Stronger models (e.g., GPT-5.6 Terra) consistently outperformed weaker ones across all query types.
• Capability Gradient: Scores showed a continuous spread rather than clustering, indicating the rubrics provide fine-grained graded satisfaction.
• Impact of Tools: Removing tools caused a significant drop in performance (average 0.14 decrease) across all models and query types, confirming that the tasks encode long-tail information not present in pre-training data.
• Query Type Sensitivity: Models showed varying strengths across query types. For instance, weaker hints ($q^o_T$) shifted the performance gap toward analytical reasoning, revealing that smaller models struggle more with decomposing broad goals and integrating evidence.
• Human Alignment: Human reviews of 100 sampled tasks showed high quality (no "poor" ratings) and high agreement among reviewers. Furthermore, automated judging by LLMs showed high correlation with human ratings (Pearson $r \approx 0.90$), validating the reliability of the generated rubrics.

Significance

The paper claims that its primary significance lies in addressing the gap between the need for deep research benchmarks and the difficulty of constructing them reliably without human experts. By demonstrating that an iterative, agent-driven process can generate tasks with traceable, verifiable rubrics, the work offers a scalable path to evaluating the next generation of AI agents. The benchmark provides a stable, fine-grained metric for assessing deep research capabilities, revealing specific weaknesses in current models (such as the inability to uncover hidden goals or integrate evidence without explicit guidance) that are not captured by existing benchmarks. The authors view the construction cost (using frontier models for the pipeline) as a necessary investment to build a trustworthy resource for the community.