Technical Summary: StatABench

Problem Statement

Statistical analysis is a complex domain requiring both deep conceptual knowledge and the ability to operationalize that knowledge through specific software tools (e.g., R, Python, SPSS). While Large Language Models (LLMs) have shown promise in general reasoning and code generation, existing benchmarks for statistical capabilities suffer from significant limitations. Prior works often focus on narrow domain scopes, lack real-world tool integration, rely on static text-only formats, or fail to evaluate the end-to-end workflow of statistical modeling. Consequently, there is a gap in assessing whether LLMs can reliably perform authentic statistical analysis, which involves data handling, method selection, tool invocation, and contextual interpretation of results.

Methodology

To address these limitations, the authors introduce StatABench, a comprehensive benchmark designed to systematically evaluate LLMs' statistical analysis capabilities. The framework consists of two complementary components:

1. Stat-Closed (404 Questions)

This component evaluates fundamental statistical knowledge and tool proficiency across 18 statistical topics (ranging from descriptive statistics to high-dimensional data analysis). It features four question formats:

• Multiple-choice, Fill-in-the-blank, Decision-making, and Practical Application.
• Tool Integration: Models are evaluated using the LangChain MCP Framework integrated with a custom Statistical Analysis Toolkit (SAToolKit). SAToolKit provides 35 callable statistical functions covering all 18 topics.
• Decontamination: A rigorous "Modify-and-Verify" pipeline was employed to prevent data leakage. Questions initially answered correctly by strong models (Qwen2.5-72B, Llama-3.1-70B) were manually perturbed (rewriting stems, adding distractors, shuffling options) until the models failed, ensuring the benchmark tests genuine reasoning rather than memorization.
• Evaluation: Answers are evaluated via direct matching for closed-form questions and an LLM-as-Judge protocol for semantically equivalent or interpretive responses.

2. Stat-Open (30 Tasks)

This component targets complex, open-ended, real-world statistical modeling problems adapted from professional competitions (MCM/ICM, CUMCM, MAS).

• Workflow: Unlike Stat-Closed, these tasks require agents to perform end-to-end analysis: problem understanding, data cleaning, exploratory analysis, model building, code execution, and structured report generation.
• Evaluation: Solutions are assessed using a validated LLM-as-Judge protocol (using Gemini 3 Pro as the judge, selected for high alignment with human raters). The evaluation rubric covers four dimensions:
  1. Structural Coherency: Rigor and completeness of modeling components.
  2. Methodological Quality: Authenticity of data application and depth of reasoning.
  3. Practicality and Scientific Validity: Robustness and real-world applicability.
  4. Result Interpretation and Bias Analysis: Ability to identify and mitigate biases.

Experimental Setup

The authors evaluated a diverse set of LLMs (including GPT-5.1, Claude 4.5, DeepSeek-V3, and various Qwen and Llama models) using both the baseline LangChain MCP setup and four advanced agent frameworks (Qwen Agent, CrewAI, AutoGen, Smolagents).

Key Contributions

• StatABench Framework: The first benchmark to uniquely combine statistical domain focus with agentic tool use, open-ended modeling, and dynamic LLM-as-Judge evaluation.
• SAToolKit: A unified interface of 35 statistical functions that allows for controlled, reproducible evaluation of tool selection and parameter construction.
• Dual-Track Evaluation: A comprehensive assessment covering both closed-form problem solving (Stat-Closed) and realistic, open-ended analytical workflows (Stat-Open).
• Rigorous Data Curation: A multi-source data collection strategy with strict decontamination protocols to ensure benchmark integrity.

Results

Experiments reveal a substantial gap between current LLM capabilities and the requirements for reliable statistical analysis:

• Stat-Closed Performance: Even the strongest model, GPT-5.1, achieved only 68.6% accuracy. The best open-source model, Qwen2.5-72B, reached 60.6%.
• Tool-Integration Tax: All models exhibited a performance drop when transitioning from fundamental statistical questions to practical application tasks requiring tool use. For instance, GPT-5.1's accuracy dropped significantly in the practical setting compared to fundamental tasks.
• Error Analysis: The primary source of failure was identified as statistical errors (e.g., inappropriate tool selection, incorrect result interpretation) rather than engineering errors (e.g., format violations). In a comparative analysis of failure cases, 74% of DeepSeek-V3's errors were statistical in nature.
• Agent Framework Impact: While stronger agent frameworks (e.g., CrewAI) improved absolute performance (e.g., raising DeepSeek-V3 to 85.35% on practical tasks), they did not eliminate cross-model stratification. The underlying statistical capability gap persisted.
• Stat-Open Performance: On open-ended modeling tasks, the top agent framework achieved an average score of 61.86 out of 100, indicating that end-to-end statistical modeling remains far from saturation.

Significance

The paper argues that current LLMs are not yet reliable statistical analysis experts. The results highlight persistent challenges in tool-grounded reasoning, methodological decision-making, and end-to-end statistical modeling. The authors posit that future progress requires more than just stronger base models or orchestration; systems must effectively connect statistical reasoning with executable workflows and defensible interpretation. StatABench serves as a rigorous testbed for diagnosing these weaknesses, distinguishing failure sources, and tracking future progress in the field of statistical analysis with LLMs.