Technical Summary: CausaLab

Problem Statement

Current evaluations of Large Language Models (LLMs) in causal reasoning often rely on static benchmarks where models answer questions based on pre-existing causal graphs, datasets, or narratives. The authors argue that these benchmarks fail to distinguish between genuine causal reasoning and the "causal parrot" phenomenon, where models succeed by retrieving memorized causal facts or linguistic cues rather than discovering mechanisms. Furthermore, existing interactive benchmarks often focus solely on whether an agent can predict an outcome or recover a graph structure, without verifying if the agent's internal hypothesis about the underlying structural equations is correct. There is a need for an environment that evaluates whether an agent can interactively discover a causal mechanism from scratch and verify that its internal model transfers to new, unseen instances.

Methodology: The CausaLab Environment

The paper introduces CausaLab, a scalable, interactive environment designed to evaluate LLM agents as causal discoverers. The environment is built upon the following core components:

1. Task Formulation

Each episode in CausaLab is a transfer problem governed by a hidden Structural Causal Model (SCM). The SCM consists of:

• A directed acyclic graph (DAG) over physical properties.
• Structural equations (linear or quadratic) defining how variables influence one another.
• Coefficients and exogenous noise terms.

The agent is presented with three distinct entities governed by the same hidden SCM but with different property values:

1. Prior Measurement Records: A batch of observational data.
2. Manipulator Crystal: A controllable instance where the agent can perform interventions on a subset of non-frequency properties.
3. Reactor Crystal: A held-out instance where the agent must predict the resonance frequency based on its observed properties, without direct intervention.

The agent operates under a finite budget of interventions. It must infer the causal graph and structural equations to predict the frequency of the Reactor Crystal.

2. Interaction Loop

The agent engages in a hypothesis-experiment loop:

• Observation: The agent observes prior records and the results of its interventions on the Manipulator Crystal.
• Intervention: The agent chooses to intervene on a controllable property (e.g., setting temperature to a specific value). The environment applies a shift-style intervention (modifying the base term of the structural equation while retaining parent contributions) rather than a hard do(X=v) operation.
• Hypothesis Generation: At every step, the agent emits a Domain-Specific Language (DSL) record. This record includes:
  • Memory: Persistent notes.
  • Thought: Interpretation of evidence.
  • Past Data: Accumulated observations.
  • Hypothesis: A structured claim containing the current causal graph, the structural equation for frequency, and the coefficients.
  • Experiment: The planned next intervention.

3. Evaluation Metrics

CausaLab decouples Task Success from Mechanism Recovery:

• Task Accuracy: The percentage of times the agent correctly predicts the held-out frequency of the Reactor Crystal.
• Mechanism Fidelity: The agent's DSL hypothesis is parsed and compared against the ground-truth SCM using:
  • Graph metrics: Precision, Recall, F1, and Structural Hamming Distance (SHD).
  • Quantitative metrics: Coefficient F1 (accuracy of the structural equation weights).
  • Root identification: Correctly identifying exogenous variables.

Key Contributions

1. A Scalable Interactive Benchmark: CausaLab moves beyond static QA to a dynamic, budget-constrained discovery setting where the ground-truth SCM is sampled per episode, preventing memorization.
2. Separation of Prediction and Understanding: By scoring the trajectory-level DSL hypothesis against the ground truth, the benchmark reveals that high task accuracy does not necessarily imply correct causal understanding.
3. Domain-Specific Language (DSL) for Auditing: The use of a parsable DSL allows for granular inspection of the agent's evolving hypothesis, enabling the detection of "premature commitment" where agents stop exploring before verifying their models against data.
4. Systematic Analysis of Interaction Strategies: The framework supports comparing pure observation, pure intervention, and mixed strategies, as well as offline vs. online intervention choices.

Experimental Results

The authors evaluated models including GPT-5-mini, GPT-5.2-high, and Qwen3.5 (with and without thinking traces) across 3–7 node graphs.

• Prediction vs. Mechanism Gap: A persistent gap exists between task accuracy and mechanism recovery. For example, in a 6-node purely observational setting, GPT-5.2-high achieved 92% task accuracy but only 0.471 all-edge F1. Agents often find plausible parents but fail to recover the correct quantitative equations.
• Interaction Strategies:
  • Mixed Observation-Intervention: This regime provided the best balance. In the 6-node mixed setting, GPT-5.2-high achieved 80% on both task accuracy and all-edge F1.
  • Pure Observation: High accuracy (92%) but poor graph recovery (0.47 F1).
  • Pure Intervention: Performed poorly on both axes, suggesting agents need observational data to narrow the hypothesis space before interventions are effective.
  • Offline vs. Online: Injecting "Golden" (optimal) intervention traces improved task accuracy (e.g., to 90% on 4-node graphs) but degraded graph recovery, indicating that choosing interventions is distinct from merely receiving intervention data.
• Model Scaling:
  • GPT-5.2-high outperformed smaller models, particularly in endpoint accuracy and frequency-weight F1.
  • However, scaling did not uniformly solve all issues; performance dropped significantly on 7-node graphs (64% accuracy, SHD 4.761).
  • "Thinking" traces generally improved structural recovery for Qwen models but did not close the gap to the strongest GPT models.
• Failure Modes: A primary cause of failure is premature stopping. Both successful and failed runs often left half the intervention budget unused. Failed runs frequently ended with hypotheses inconsistent with their own collected data. A single explicit verification step (checking hypothesis consistency with data) improved 4-node accuracy from 48% to 60%.

Significance and Claims

The paper positions CausaLab as a "controlled stress test" for interactive causal discovery rather than a claim that current agents are ready for real-world scientific deployment. Its significance lies in:

• Exposing Limits: It demonstrates that current LLM agents struggle to explore unfamiliar environments, test candidate mechanisms, and revise hypotheses toward causal regularities, even when they can predict outcomes accurately.
• Refining Evaluation: It argues that evaluating causal reasoning requires separating predictive success from structural and quantitative faithfulness.
• Guiding Future Agents: The findings suggest that future AI scientists need better strategies for designing informative interventions and mechanisms for self-verification to avoid premature commitment to incorrect models.

The authors explicitly state that the results should not be overgeneralized to high-stakes real-world settings, as the benchmark uses synthetic 3–7 node SCMs with limited functional forms and shift-style interventions.