Technical Summary: Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities

Problem Statement

Current benchmarks for evaluating Large Language Models (LLMs) in data analysis fail to reflect real-world settings. Existing datasets (e.g., WikiTableQuestions, Spider) typically rely on small-scale, single-table environments and focus primarily on fact retrieval via semantic parsing or text-to-SQL. These benchmarks neglect critical practical challenges, including:

• Scale and Complexity: The need to process massive tables (millions of rows) and merge multiple heterogeneous tables.
• Contextual Dependency: The essential role of metadata and external knowledge (e.g., data dictionaries) required to interpret raw data.
• Task Scope: The lack of evaluation for proactive, exploratory insight discovery. Existing tasks focus on answering specific queries, missing the capability of data analysts to generate expert-level findings without explicit user prompts.

Consequently, the "discovery-oriented" skill set remains largely unevaluated, creating a gap between the vision of autonomous data analysis systems and their actual performance in complex, real-world scenarios.

Methodology: DataGovBench

To bridge these gaps, the authors introduce DataGovBench, a benchmark derived from governmental open data repositories (e.g., Data.gov, Data.gov.uk). The benchmark is constructed through a three-stage process:

1. Data Curation: The authors identified 53 open data platforms and filtered datasets based on strict criteria: at least one tabular file with >5,000 rows, available metadata (descriptions), and the presence of external knowledge (e.g., data dictionaries). The final set comprises 178 unique datasets, featuring an average of 210,000 rows and 18 columns, with the largest table containing 11.9 million rows. Notably, over 36% of datasets are multi-tabular, and over 57% include external knowledge in various formats (PDF, XLSX, etc.).

2. Task Definition: The benchmark evaluates LLMs on two complementary tasks:

   • Table QA: Models must answer simple or decomposable questions (multi-step reasoning) with textual or visual outputs. To handle large tables, the authors propose Feature Type-Specific Table Serialization, a technique that creates a compact summary of a table by representing columns according to their data types (e.g., listing unique categories for categorical columns, min/max for numerical) rather than raw rows.
   • Table Insight: Models must perform open-ended exploratory analysis to generate a list of expert-level findings and a summary without a specific user query. Ground truth for this task is derived from official reports accompanying the datasets, ensuring objective evaluation of subjective insights.

3. Annotation Pipeline:

   • QA Pairs: Generated via a human-in-the-loop pipeline using an ensemble of four LLMs (GPT-4o, GPT-4o-mini, Gemini 2.0 Flash, Gemini 1.5 Pro). Questions undergo automated scoring, answer generation (via Python code execution), and rigorous human verification to ensure clarity, correctness, and analytical complexity.
   • Insights: Extracted from domain-expert reports using LLMs (NotebookLM, Gemini) to structure findings into declarative sentences and summaries, followed by manual verification.

Experimental Setup

The authors evaluated a diverse range of models, including open-source LLMs (Llama 3.1, Qwen3, DeepSeek-R1), specialized table models (TableGPT2), and closed-state-of-the-art models (GPT-4o, GPT-5.1, Claude Sonnet 4.6, Gemini 2.5 Flash).

Two evaluation frameworks were tested:

1. Direct LLM: Models generate answers directly.
2. Agentic Frameworks:
   • Answer Agent: A modular agent featuring serialization, code generation with self-correction, and a reflection module (using VLM/MLLM for visual validation).
   • Insight Agent: An extension of the AgentPoirot framework, utilizing a Directed Acyclic Graph (DAG) structure to iteratively generate questions, answers, and insights, aggregating context from all previous nodes rather than a simple tree structure.

Evaluation metrics included Exact Match (EM) for text, MLLM-as-a-judge for visualizations, and LLaMA-3-Eval scores (adapted for GPT-4o) for insight quality, measured at both summary and individual insight levels.

Key Results

Comprehensive experiments reveal significant performance gaps, indicating that current systems are far from satisfying real-world data analytics demands:

• Low Accuracy: Even top-performing closed-source models with agentic support achieved low absolute scores. In the Table QA "Whole" setting (requiring all sub-questions to be correct), the best score was below 0.4. For Table Insight, top models scored below 0.35 for insight-level alignment and below 0.5 for summary-level alignment.

• Agentic Support: While agentic frameworks (Answer Agent) improved performance over base LLMs (e.g., ~27% relative improvement for Gemini 2.5 Flash in QA), the gains were insufficient to overcome the benchmark's difficulty.

• Model Comparison: Open-source models lagged significantly behind closed-source counterparts. Specialized table models (TableGPT2-7B) performed poorly (score < 0.1), suggesting that general reasoning capabilities are currently more critical than table-specific pre-training for these complex tasks.

• Error Analysis:

  • Table QA: The most prevalent errors were Condition Filter Errors (32.4%, e.g., failing to avoid double-counting) and Transformation Errors (23.2%, e.g., parsing datetime formats or converting string numbers). Models also struggled with context handling in multi-turn scenarios and selecting the correct table from metadata in multi-table datasets.
  • Table Insight: While agents performed reasonably well on Topic Relevance, they frequently failed on Narrative Alignment and Qualitative Details Match. Crucially, agents completely failed at Quantitative Details Match, unable to accurately calculate or present specific numerical values in their generated insights.

• Ablation Findings:

  • Serialization vs. Raw Rows: Providing more raw rows (up to 200) degraded performance compared to the baseline of 10 rows. In contrast, the feature type-specific serialization significantly improved performance by providing a compact, reliable representation.
  • SQL vs. Python: Text-to-SQL approaches performed significantly worse than Python code generation for this benchmark, with higher rates of syntax and column selection errors.

Significance and Claims

The paper claims that DataGovBench provides a challenging, realistic benchmark that exposes the limitations of current LLM-based data analytics systems. Its primary contributions are:

1. Realism: It is the first benchmark to holistically combine complex, decomposable QA and insight generation with large-scale, multi-tabular datasets requiring metadata and external knowledge.
2. Performance Gap Identification: It demonstrates that even state-of-the-art models, whether standalone or agentic, struggle with the nuances of real-world data, particularly in narrative-level reasoning and accurate fact retrieval from complex tables.
3. Research Roadmap: Through detailed qualitative analysis, the authors identify specific failure modes (e.g., inability to synthesize core arguments, quantitative calculation errors) and missing capabilities, offering a clear guide for the community to focus future research on building more robust agents capable of handling real-world data complexities.

The authors conclude that while LLMs have shifted the paradigm of human-data interaction, significant advancements are required before these systems can autonomously satisfy the demands of professional data analytics.