Technical Summary: CausalDS – Benchmarking Causal Reasoning in Data-Science Agents

1. Problem Statement

Current benchmarks for Large Language Models (LLMs) in data science and causal reasoning suffer from a fragmentation that fails to capture the complexity of real-world causal analysis. Existing evaluations typically isolate specific capabilities:

• Data-science benchmarks (e.g., DS-1000, MLE-bench) emphasize coding, tool use, and open-ended analysis but lack a hidden, principled causal data-generating structure.
• Causal reasoning benchmarks (e.g., CLadder, CausalGraph2LLM) test symbolic graph reasoning, intervention, and counterfactual logic but often operate on purely symbolic levels or use curated examples from existing literature.

This separation creates a gap where models may rely on "amortized causal inference" (memorizing patterns from existing datasets) rather than genuine structure-sensitive reasoning, a failure mode termed the "causal parrot." Furthermore, realistic causal data science requires an agent to not only answer causal questions but also to interpret domain descriptions, inspect observational data, determine identifiability, estimate quantities, quantify uncertainty, and recognize when a question is unanswerable (abstention). No existing benchmark jointly evaluates symbolic reasoning, data-science execution, uncertainty quantification, epistemic abstention, and tool use within a single, synthetic yet realistic framework.

2. Methodology: The CausalDS Framework

CausalDS addresses this gap by generating synthetic causal data-science scenes that serve as the benchmark unit. Each scene consists of a hidden Structural Causal Model (SCM), a natural-language story, tabular observational data, and a suite of tasks spanning Pearl's three-rung hierarchy.

2.1 Scene Generation Pipeline

The generation process follows a strict pipeline to ensure ground truth is known while maintaining narrative coherence:

1. Graph Sampling: Directed Acyclic Graphs (DAGs) are sampled from canonical motifs (chains, forks, confounders, colliders, etc.) and can be expanded via "anchor-based grafting," where auxiliary motifs are attached through shared nodes to increase complexity while preserving narrative flow.
2. SCM Instantiation: Each graph is instantiated with a Structural Causal Model using typed mechanism profiles (continuous and binary) drawn from registries that mix empirical biochemical/Boolean rules with synthetic regimes.
3. Observation Layer: A critical design choice distinguishes the conceptual SCM from the data released to the agent. Conceptual variables may be replaced by bundles of noisy measurements (proxies). This layer increases data-analysis difficulty without altering the underlying causal identifiability status.
4. Verbalization: Abstract nodes are mapped to domain-relevant variable names (optionally seeded from CauseNet) and verified by an auditor to ensure the generated natural-language story faithfully reflects the hidden graph without introducing "accidental edges."
5. Task Derivation: From each scene, tasks are derived across Rung 1 (associational/prediction), Rung 2 (interventional/identification/estimation), and Rung 3 (counterfactual/mediation).

2.2 Evaluation Harness

Agents interact with a sandboxed environment (using mini-swe-agent) where they must:

• Read provided files (stories, data schemas, observational data).
• Execute code (Python/bash) to perform analysis.
• Write answer files in a specific format.
• Handle imperfect observations and decide whether to abstain when a target is non-identifiable.

2.3 Scoring and Metrics

Evaluation is fully deterministic and scores five distinct axes:

• Symbolic Causal Reasoning: Graph recovery and identification logic.
• Data-Science Execution: Accuracy of estimates and predictions.
• Uncertainty Quantification: Calibration of confidence intervals.
• Epistemic Abstention: Correctly identifying non-identifiable queries and refusing to answer.
• Tool Use/Efficiency: Token usage and tool-call counts.

A composite score (CausalDSScore) aggregates performance, penalizing both content errors and failures to abstain on non-identifiable tasks.

3. Key Contributions

• SCM-Grounded Synthetic Scenes: The first benchmark to generate hidden SCMs, synthesize observational data, and compute private ground truth for evaluation, while anchoring composition axes (variable types, graph structures) to empirical distributions from real-world datasets to ensure realism.
• Graph-Faithful Free-Form Verbalization: A pipeline that maps abstract graph nodes to coherent domain variables and generates natural-language stories, audited to ensure the narrative does not deviate from the underlying causal structure.
• Separate Observation Layer: A mechanism to vary data-science difficulty (via noisy proxies) independently of causal identifiability, allowing the benchmark to stress-test estimation skills without changing the causal question.
• Broad Task Suite: A comprehensive set of tasks covering all three rungs of Pearl's hierarchy, including prediction, association, graph recovery, identification, effect estimation, bias diagnostics, and counterfactual reasoning.
• Abstention-Aware Deterministic Evaluation: A scoring system that treats abstention on non-identifiable estimands as a first-class outcome, preventing models from being rewarded for hallucinating answers to unanswerable questions.

4. Experimental Results

The authors evaluated six models (three frontier closed models: Claude Opus 4.8, Gemini 3.1 Pro, GPT-5.5; and three open-weight models: Qwen 3.6 35B, Kimi K2.6, Gemma 4 26B) on a 100-scene realistic-composition exam.

• Performance Dissociation: The five evaluation axes do not collapse into a single capability. Models differ significantly in content correctness, uncertainty quantification, abstention recognition, and tool-use efficiency.
• Leaderboard: Claude Opus 4.8 achieved the best overall CausalDSScore (0.278), leading in pass rate, normalized error, and signal-to-noise ratio (SNR). It was the only model to achieve perfect content correctness (100%) on binary tasks while maintaining strong abstention performance.
• Abstention as a Differentiator: The ability to recognize non-identifiable queries was a major differentiator. Frontier models (especially GPT-5.5 and Claude) showed significantly higher abstention pass rates (up to 75%) compared to open-weight models (as low as 18.8% for Gemma 4 26B).
• Uncertainty Quantification: All models exhibited overconfidence. Empirical coverage of nominal 95% ATE intervals collapsed to between 20.0% and 71.4%, with Claude Opus 4.8 performing best (71.4%).
• Tool Use and Efficiency: Frontier models (Claude, GPT-5.5) utilized a "near one-shot" strategy with fewer tool calls (2.1–3.4 per task) and lower token usage, whereas open-weight models iterated heavily (11–18 calls/task) and consumed significantly more tokens.
• Observation Hardness: Under the hardest observation variants (noisy proxies), performance degraded, particularly for open-weight models. GPT-5.5 showed the largest drop in continuous estimate calibration (mean normalized error rising to 3.10), while Claude Opus 4.8 maintained controlled error.

5. Significance and Claims

The paper claims that CausalDS provides a necessary evolution in benchmarking by integrating symbolic reasoning, quantitative estimation, and agentic tool use into a single, realistic setting. The primary empirical finding is that causal data-science competence decomposes: models may master the parts (reading structure, producing estimates) but fail on the whole (knowing the quality of the estimate or whether an answer is licensed at all).

The authors assert that a competent causal data-science agent must clear all five axes simultaneously. The benchmark exposes that current frontier models are approaching this capability, with Claude Opus 4.8 emerging as the closest to a "well-rounded causal data-science agent," while open-weight models and smaller models struggle significantly with epistemic axes (abstention and uncertainty) and tool-use efficiency. The work highlights that "knowing when to abstain" is a separate, advanced skill that distinguishes robust agents from those prone to hallucination.