Technical Summary: DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces

1. Problem Statement

Data agents are emerging as natural-language interfaces for organizational data, yet realistic analytical settings present a complex challenge: the evidence required to answer a query is rarely contained within a single, clean table. Instead, information is scattered across relational databases, structured/semi-structured files (CSV, JSON), long documents (PDF, Markdown), and multimedia artifacts (video), often involving cross-language variations.

Existing benchmarks fail to unify three critical properties of realistic data analysis:

1. Workspace Scope: They often isolate structured querying (e.g., Spider) or unstructured retrieval (e.g., HotpotQA), failing to capture tasks requiring the discovery and integration of evidence across heterogeneous modalities within a task-local workspace.
2. Output Contract: Many benchmarks accept factoid answers, open-ended reports, or executable pipelines. Real-world analytics often demand a complete, typed tabular result that can be directly consumed.
3. Evaluation Semantics: Current evaluations often rely on model-based judges or fail to handle equivalent representations (e.g., different column orders, header wording, or numeric precision) deterministically.

The paper argues that without a unified benchmark enforcing a complete tabular output contract over heterogeneous, cross-language workspaces with deterministic evaluation, progress in data agent reliability cannot be accurately measured.

2. Methodology

2.1 DataSpace Benchmark

DataSpace is a benchmark comprising 410 cross-language tasks and 7,439 artifacts totaling 15.01 GB.

• Modalities: Workspaces include CSV, JSON, SQLite, Markdown, PDF, and Video.
• Languages: Tasks involve cross-language scenarios where the question and workspace artifacts may mix Chinese and English.
• Task Formulation: An agent receives a natural language question and a task-local workspace. It must autonomously discover sources, align entities/schemas, execute multi-step computations (filtering, joining, aggregation), and return the complete requested tabular result as a CSV file.
• Domains: Financial analytics (funds, stocks), macroeconomic data, and healthcare analytics.

2.2 DataSpace-Builder: Construction Framework

To construct these complex tasks reliably, the authors propose DataSpace-Builder, an execution-grounded framework that transforms instances from existing Text-to-SQL benchmarks (EHRSQL and BULL) into heterogeneous workspace tasks. The process involves four stages:

1. Cross-Language Transformation (CLT): A joint migration of the question, database state, and executable SQL. It ensures referential consistency by translating entity names and values while preserving identifiers and codes. An LLM judge verifies semantic alignment between the translated question and SQL.
2. Constraint-Aware Relational Sampling: To avoid repetitive workspaces, the framework samples rows from the source database while retaining the full schema. It uses a safeguard set (primary/foreign keys, query predicates) to ensure relational integrity (e.g., preserving join paths) during sampling.
3. Modality Routing & Artifact Rendering:
   • Base Routing: Tables are assigned to renderers (CSV, JSON, SQLite, Markdown, PDF) based on schema properties.
   • Document Rendering: Long-form documents (Markdown/PDF) are generated from table data using LLMs, ensuring "fact-grounded" generation where specific cells are recoverable from the text.
   • Video Rendering: Tabular data is converted into data-insight videos. Two strategies are used: Predicate Abstraction (moving filter conditions into the video narrative) and Answer-Evidence Rendering (distributing result cells across video scenes).
4. Human Review & Task Repair: A panel of 11 domain experts performs blind, independent reviews. They solve the task, verify the gold standard, and author the evaluation configuration. Disagreements trigger evidence-based repairs to the question, workspace, or gold answer.

2.3 Deterministic Evaluator

The evaluation protocol is model-free and header-invariant:

• Column Alignment: The evaluator finds a one-to-one mapping between predicted and reference columns regardless of header wording or order.
• Normalization: Values are normalized based on semantic types (text, number, date, boolean) and precision rules (e.g., decimal places, percentage conventions).
• Row Comparison: Tasks requiring order are compared as sequences; others are compared as unordered multisets.
• Scoring: A task is correct only if the complete tabular prediction matches the reference under these semantics.

3. Key Contributions

1. A Heterogeneous Workspace Benchmark: DataSpace introduces 410 tasks requiring the synthesis of structured files, databases, long documents, and video into a single, verifiable tabular output.
2. An Execution-Grounded Construction Framework: DataSpace-Builder transforms executable Text-to-SQL resources into complex multimodal tasks, ensuring data consistency through sampling safeguards and expert review.
3. A Semantics-Aware Evaluator: A deterministic protocol that tolerates equivalent representations (column reordering, formatting) while rejecting incomplete or erroneous outputs, eliminating reliance on LLM judges.
4. Empirical Baselines: The paper establishes performance baselines across six frontier multimodal models and five agent harnesses, identifying specific bottlenecks in current agent capabilities.

4. Experimental Results

The authors evaluated six multimodal backbones (Grok 4.5, GPT-5.6 Sol, Kimi K3, MiMo-V2.5, Claude Sonnet 5, MiniMax M3) and five agent harnesses (including their own DataSpace-Agent).

• Overall Performance: The best performing configuration (Grok 4.5 with DataSpace-Agent) achieved 66.34% accuracy. This indicates the benchmark is unsaturated, as 76 tasks were missed by all six backbones, while only 56 tasks were solved by all models.
• Harness Impact: The choice of agent harness significantly impacts performance. With the MiMo-V2.5 backbone fixed, accuracy varied by 15.36 percentage points across different harnesses (ranging from 30.98% to 46.34%).
• Key Challenges:
  • Multimodal Integration: Tasks requiring multimodal evidence consistently reduced accuracy across all backbones by 1.8–14.0 points compared to single-modal tasks.
  • Joins: Join operations reduced accuracy by 9.7–19.8 points.
  • Efficiency: GPT-5.6 Sol achieved near-top accuracy (64.63%) using 74.2% fewer tokens and 50.3% fewer actions than the top performer, Grok 4.5.
• Failure Analysis: An audit of 136 failures from Grok 4.5 revealed that 52.2% of errors originated in answer materialization (e.g., adding/omitting columns or rows after the correct internal result was computed), rather than in evidence discovery or extraction. Only 5 of 13 "no-submission" cases were pure termination failures; most stemmed from earlier persistent errors.

5. Significance and Claims

The paper claims that DataSpace establishes a rigorous test bed for advancing reliable data agents. Its primary significance lies in shifting the evaluation paradigm from isolated skills (e.g., SQL generation or document retrieval) to end-to-end workspace solving with a strict, verifiable output contract.

The authors modestly conclude that while current frontier models show promise, significant gaps remain in cross-modality integration and joins. The benchmark highlights that the ability to "find" data is insufficient; agents must also faithfully materialize the exact requested output structure. The results suggest that improving data-agent reliability requires addressing the full pipeline from heterogeneous evidence discovery to precise tabular serialization, rather than focusing solely on reasoning capabilities.

DataSpace serves as the official evaluation benchmark for the KDD Cup 2026 Data Agents for Complex Data Analysis competition, providing a standardized, reproducible environment for future research in this domain.