Technical Summary: Evaluating Deep Data Research on LLMs

Problem Statement

Current evaluations of Agentic Large Language Models (Agentic LLMs) predominantly focus on executional intelligence—the ability to complete predefined tasks or answer specific user queries. This framing conflates execution with investigatory intelligence, which is the capacity to autonomously decide what is worth exploring, formulate hypotheses, and determine when to terminate an investigation. In real-world data science, analysts begin with raw structured data without predefined questions, iteratively refining hypotheses as patterns emerge. Existing benchmarks for data science (LLM4DS) often rely on pre-defined questions, subjective report quality assessments, or "LLM-as-a-Judge" mechanisms that fail to objectively verify factual claims against the source data. Furthermore, current benchmarks typically limit agent interactions to short horizons (dozens of steps), failing to capture the complexity of long-horizon autonomous exploration.

Methodology

Deep Data Research (DDR) Formalization

The authors formalize Deep Data Research (DDR) as an open-ended task $I = \text{DDR}(\text{LLM}, D, T)$, where an agent is given a structured database $D$ and a toolset $T$ (SQL and Python) but no predefined questions, objectives, or interaction limits.

• Input: A minimal start prompt specifying a task entity (e.g., "Start analysing the user with userid=2048") and basic database metadata.
• Process: The agent engages in a ReAct-style interaction loop $(r, t, o)$ over multiple rounds, where $r$ is reasoning, $t$ is tool invocation, and $o$ is the observation. The agent autonomously determines when to stop exploration.
• Output: Two types of insights:
  1. Message-wise Insight ($I_m$): Incremental interpretations of each interaction round.
  2. Trajectory-wise Insight ($I_t$): A comprehensive report synthesizing the entire exploration history.

DDR-Bench Construction

To evaluate DDR, the authors introduce DDR-Bench, a large-scale benchmark comprising three distinct real-world scenarios:

1. MIMIC-IV: A de-identified Electronic Health Record (EHR) database. Agents must reconstruct patient care trajectories and extract clinical insights from structured tables and unstructured clinical notes.
2. GLOBEM: A sports and exercise psychology dataset combining wearable sensor signals with survey data. Agents must identify temporal patterns and link behavioral data to psychological states.
3. 10-K: Financial reports from US public companies, integrating structured XBRL financial statements with unstructured textual disclosures. Agents must synthesize economic conditions and financial trends.

Evaluation Framework:

• Checklist-Based Verification: Instead of subjective scoring, the benchmark uses fact checklists derived from the unstructured components of the databases (e.g., clinical notes, survey responses, financial disclosures).
• Objective Scoring: For each task entity, a checklist of verifiable facts is generated. An LLM-as-a-Checker (GPT-5-mini) evaluates whether the agent's generated insights provide sufficient evidence to support each fact item.
• Novelty Analysis: To address the limitation that checklists cannot cover all valid insights, the authors perform pairwise comparisons of "novel" insights (those not used for checklist verification) to rank models on the usefulness of their unverified discoveries.

Experimental Design

• Minimalist Agent Scaffold: The benchmark uses a lightweight ReAct agent with no external memory, planning modules, or complex scaffolding to isolate intrinsic model capabilities.
• Scaling Analysis: The study examines performance across interaction turns, token consumption, and inference cost.
• Ablation Studies: The authors analyze the impact of training factors (parameter scale, context length, training generations) and agent modules (memory, reasoning budget) on performance.

Key Results

Overall Performance

• Limited Investigatory Agency: Even frontier models struggle with DDR. Only Claude 4.5 Sonnet achieves an average checklist accuracy above 40% (47.73% overall). Other top models (GPT-5.2, DeepSeek-V3.2, GLM-4.6) score significantly lower.
• Discrepancy in Insight Types: Performance varies between message-wise ($I_m$) and trajectory-wise ($I_t$) insights, suggesting that integrating global context is a distinct challenge from incremental analysis.
• Novelty Correlation: There is a strong correlation (Spearman $\rho > 0.72$) between checklist accuracy and the usefulness of novel insights, validating that checklist-based evaluation captures the dominant signal of model capability.

Scaling and Dynamics

• Interaction Scaling: Performance follows a sigmoid curve. High-performing models (e.g., Claude, GLM) often delay rapid improvement, exhibiting a "plan-then-act" dynamic through implicit planning rather than explicit prompts.
• Token and Cost Scaling: Performance gains are disproportionately driven by late-stage, high-value tokens (deep, targeted queries) rather than early broad exploration.
• Exploration Patterns: Successful models exhibit a balanced exploration regime, maintaining high database coverage with low entropy (focused access). Weaker models either explore too narrowly or distribute attention too diffusely.
• Self-Termination: Stronger models demonstrate higher confidence in self-termination, whereas weaker models show fluctuating confidence, often terminating prematurely or looping.

Training and Module Analysis

• Scaling is Not Enough: Increasing parameter size (e.g., within the Qwen family) yields marginal gains (<3% improvement for a 10x parameter increase) without specific agentic training.
• Training Strategy Matters: Models with "agentic-first" training strategies (Qwen3 and Qwen3-Next) significantly outperform their predecessors, even with fewer activated parameters. This suggests that systematic agentic training (pre-training and reinforcement learning) is more critical than scale or context length.
• Agent Modules: Adding external memory or complex planning modules often degrades performance compared to the minimal ReAct baseline, indicating that current models benefit more from intrinsic capabilities than external scaffolding.

Failure Modes and Hallucinations

• Primary Failure: 58% of errors stem from insufficient exploration (lack of breadth or depth), often due to premature self-termination.
• Hallucinations: Hallucination rates are low (<5% for most models) and show no significant correlation with accuracy. This suggests that data contamination is not a primary driver of performance inflation in this setting.

Significance and Claims

The paper claims to establish a rigorous framework for evaluating investigatory intelligence, distinguishing it from executional intelligence. Its primary contributions are:

1. Formalization of DDR: Defining an open-ended setting where agents must autonomously explore structured data without predefined goals.
2. DDR-Bench: Providing the first large-scale, checklist-based benchmark that enables verifiable, objective evaluation of long-horizon data exploration.
3. Empirical Insights: Revealing that current frontier models, while showing emerging agency, are far from mastering deep data research. The transition to effective investigatory intelligence relies less on external scaffolding or simple parameter scaling and more on intrinsic strategies developed through agentic-focused training.

The authors conclude that the field must move beyond query-response alignment toward training paradigms that cultivate end-to-end autonomy, enabling agents to proactively define goals and derive useful insights from large-scale data.