Technical Summary: CausalReasoningBenchmark

Problem Statement

Current benchmarks for automated causal inference systems, particularly those leveraging Large Language Models (LLMs), suffer from a critical evaluation flaw: they conflate identification and estimation. Most existing benchmarks assess performance based solely on a single numerical output, such as an Average Treatment Effect (ATE). This approach fails to distinguish between two distinct phases of causal analysis:

1. Identification: The conceptual exercise of determining whether a causal quantity is recoverable from data under specific assumptions, requiring the formulation of a valid research design (e.g., Instrumental Variables, Regression Discontinuity) and the specification of all necessary components (treatment, outcome, controls, instruments, cutoffs).
2. Estimation: The numerical implementation of the identified design on finite data to compute point estimates and standard errors.

By collapsing these steps into a single score, existing evaluations cannot diagnose whether a model failed due to flawed causal reasoning (invalid design) or errors in numerical execution (coding mistakes). Furthermore, many benchmarks rely on synthetic or simplified data that lack the complexities of real-world empirical research, such as ambiguous variable definitions and missing data.

Methodology

The authors introduce CausalReasoningBenchmark, a comprehensive evaluation suite designed to disentangle identification from estimation.

Dataset Construction

The benchmark comprises 173 queries across 132 unique real-world datasets, curated from:

• 79 peer-reviewed research papers (primarily from political science journals like The Journal of Politics and American Political Science Review), including large-scale reanalysis studies of Instrumental Variable (IV), Regression Discontinuity (RDD), and Difference-in-Differences (DiD) designs.
• Three causal inference textbooks (The Mixtape, The Effect, and Causal Inference: What If), providing pedagogical examples and covering Conditional Exogeneity designs.

Each query includes a natural-language causal question, a CSV dataset, metadata describing variables and study context, and a gold-standard solution.

Evaluation Framework

The system is required to produce two distinct outputs for each query:

1. Structured Identification Specification: A JSON object detailing the research design strategy and specific fields (e.g., instrument names for IV, running variables and cutoffs for RDD, time/group variables for DiD).
2. Estimation Output: A point estimate and standard error, generated via executable code (Python/R).

Scoring Mechanism:

• Identification Metrics: The specification is evaluated field-by-field against the gold standard. Checks include exact matches for strategy and estimand labels, set-based matches for treatment/outcome variables, and strict logic for control variables (must include the minimal sufficient adjustment set and exclude "bad controls" like post-treatment variables or mediators).
• Estimation Metrics: The predicted effect and confidence intervals are compared to gold-standard values using metrics such as point-estimate error, Jaccard overlap of confidence intervals, and null-hypothesis agreement. The evaluator includes an auto-rescaling feature to mitigate errors caused by unit mismatches (e.g., percentages vs. proportions).

Key Contributions

1. Real-World Benchmark: A curated collection of 173 queries derived from peer-reviewed literature and textbooks, ensuring the data reflects the complexity of actual empirical research.
2. Structured Identification Schema: A requirement for agents to output detailed, structured specifications for five canonical designs: IV, RDD, DiD, Conditional Exogeneity, and Randomized Controlled Trials (RCT).
3. Disentangled Evaluation: The first benchmark to separately score research-design correctness and numerical estimation accuracy, enabling granular diagnosis of failure modes.
4. Standardized Gold-Standard Scripts: Provision of reference estimation code for every query to isolate identification failures from implementation errors.

Baseline Results

The authors evaluated a state-of-the-art LLM (GPT-5.3 with reasoning capabilities) on the benchmark. The results highlight a significant gap between high-level strategy recognition and detailed specification:

• Strategy Recognition: The model correctly identified the high-level research strategy (e.g., "Instrumental Variable") in 79.2% of cases.
• Full Identification Correctness: The rate of fully correct identification specifications (including all design-specific fields and control variables) dropped to only 34.1%.
• Estimation: While estimation errors were non-trivial (median relative error of 18.1%), they were secondary to identification errors. In many cases, the model produced reasonable estimates even when the identification specification was flawed, often by using a plausible but non-gold-standard approach.

Error Analysis:

• Control Variables: The most frequent error type (39 cases of missing necessary controls; 18 cases of including bad/post-treatment controls).
• Strategy Confusion: A notable pattern involved misclassifying observational studies requiring covariate adjustment (Conditional Exogeneity) as Randomized Controlled Trials (24 cases), likely due to the model misinterpreting the word "assignment" in the metadata.
• Estimand Confusion: In RDD queries, the model frequently confused Conditional Average Treatment Effects (CATE) with Local Average Treatment Effects (LATE) (29 out of 44 cases).
• Design-Specific Nuances: IV queries showed the largest gap between strategy recognition and full specification, indicating difficulty in identifying valid instruments and distinguishing them from first-stage fitted values.

Significance and Claims

The paper claims that CausalReasoningBenchmark provides a more rigorous and realistic evaluation framework for automated causal inference systems. By separating identification from estimation, the benchmark reveals that the primary bottleneck in current LLM-based causal reasoning is not the ability to recognize broad design categories or perform computation, but rather the nuanced details of research design specification.

The authors argue that a single-score evaluation based on final effect estimates would obscure these critical distinctions, potentially masking fundamental reasoning failures. The benchmark is intended to foster the development of more robust systems capable of handling the intellectual rigor required for valid causal inference in real-world settings. The dataset and evaluation code are publicly available on Hugging Face to support community contributions and future research.