🤖 AI

Why LLMs Fail at Causal Discovery and How Interventional Agents Escape

This paper proves that supervised and preference-based learning methods fundamentally fail at causal discovery due to a kernel obstruction theorem, and proposes Agentic Causal Bayesian Optimization (A-CBO) as a provably convergent alternative that uses a frozen language model as an interventional oracle within an external Bayesian loop to outperform existing benchmarks.

Original authors: Amartya Roy, Sonali Parbhoo

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Amartya Roy, Sonali Parbhoo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Why AI Gets Stuck on "Causal" Puzzles

Imagine you are trying to figure out how a machine works just by watching it run. You see that when a red light flashes, a bell rings. You also see that when a blue light flashes, the bell rings.

Now, you have to guess the wiring inside:

Scenario A: The red light triggers a switch that hits the bell.
Scenario B: The blue light triggers a switch that hits the bell.

If you only watch the machine (observational data), both scenarios look exactly the same. The bell rings every time a light flashes. This is what the paper calls a "near-miss" problem. Two completely different internal structures (causal graphs) can produce the exact same outside behavior.

The Paper's Main Discovery:
The authors prove that Large Language Models (LLMs)—the brains behind tools like ChatGPT—have a fundamental mathematical blind spot here. Even if you train them on millions of examples (Fine-Tuning) or ask them to think really hard (Prompting), they cannot reliably tell the difference between these "near-miss" scenarios.

The Analogy: The "Blurry Photo" Limit
Think of an LLM as a photographer taking a picture of a complex machine.

The Problem: When the machine gets bigger and more complex (more variables), the "near-miss" scenarios look 99.9% identical in the photo. The tiny difference that tells you which wire is which is so small it gets lost in the blur.
The Math: The paper proves that for an LLM to see that tiny difference, it would need to "stretch" its internal memory infinitely. Since it can't do that, it just guesses. As the puzzles get harder, the AI's performance crashes, often dropping to the level of random guessing.

The Solution: The "Agent" and the "Oracle"

Since the AI can't solve the whole puzzle at once, the authors built a new system called A-CBO (Agentic Causal Bayesian Optimization).

Instead of asking the AI, "Which of these two complex machines is the real one?" (which it fails at), they change the game. They use the AI as a specialized helper and a mathematical detective as the decision-maker.

Here is how it works, step-by-step:

1. The Frozen Oracle (The AI)

The AI is "frozen," meaning it isn't being retrained or forced to learn new rules. It acts like a truth-telling oracle.

The Question: Instead of asking the AI to solve the whole mystery, the system asks it simple, binary questions: "If I push this specific red button, does the bell ring?"
Why it works: Even if the AI can't tell the difference between the two complex machines, it is very good at answering simple cause-and-effect questions. Pushing a button and seeing a result is easy for the AI to predict.

2. The Bayesian Loop (The Detective)

This is the part that does the heavy lifting. It's a mathematical loop running outside the AI.

The Strategy: The detective starts with a list of all possible machine designs.
The Action: It picks the one question that will eliminate the most wrong answers. It asks the AI: "If I push Button A, does the bell ring?"
The Update:
- If the AI says "Yes," the detective crosses off every machine design where Button A wouldn't ring the bell.
- If the AI says "No," it crosses off the designs where it would.
The Result: With every simple question, the list of possibilities shrinks. The detective doesn't need the AI to be perfect; it just needs the AI to be slightly better than random.

Why This is a Game-Changer

The paper shows that this method works where everything else fails.

No Training Needed: The AI doesn't need to be retrained on millions of new examples. It uses its existing knowledge to answer simple questions.
It Scales: As the puzzles get harder (more variables, more complex machines), the old methods (Fine-Tuning) crash and burn. But the A-CBO system keeps getting better. It can solve puzzles with 24 variables that stump even the most advanced fine-tuned models.
The "Lazy" Advantage: The system keeps the AI in its "lazy" state (where it just predicts the next word) but moves the decision-making to a place where the AI's mathematical limitations don't apply.

Summary in One Sentence

The paper proves that AI is mathematically incapable of distinguishing between complex, look-alike causes on its own, but by turning the AI into a simple "yes/no" question-asker and letting a mathematical loop do the actual detective work, we can solve these puzzles perfectly without ever needing to retrain the AI.

Technical Summary: Why LLMs Fail at Causal Discovery and How Interventional Agents Escape

Problem Statement
Causal discovery—the recovery of causal graph structures from statistical data—is fundamental to scientific reasoning but remains an open challenge for Large Language Models (LLMs). While LLMs excel at natural language tasks, recent benchmarks (e.g., Corr2Cause) indicate they fail to reliably distinguish between causal graphs that generate similar observational data. Specifically, models struggle with "near-miss" hypotheses: Directed Acyclic Graphs (DAGs) that belong to different Markov equivalence classes but produce nearly identical conditional independence statements. Existing approaches, including prompting strategies, supervised fine-tuning (SFT), direct preference optimization (DPO), and in-context learning (ICL), show empirical degradation as graph complexity increases, yet the theoretical reason for this failure has not been established.

Methodology and Theoretical Framework
The authors formalize the failure of LLMs in causal discovery as a Kernel Obstruction Theorem.

Kernel Obstruction Theorem (Theorem 1): The paper proves that SFT, DPO, and ICL all produce predictors operating within a Reproducing Kernel Hilbert Space (RKHS) with bounded norms (the "lazy" or NTK regime). For "near-miss" pairs of causal hypotheses (e.g., a chain $V_1 \to V_2 \to V_3$ vs. a fork $V_1 \leftarrow V_2 \to V_3$ ), the observational data tokens are nearly identical, leading to a kernel similarity approaching 1 ( $\delta \to 0$ ).
- The theorem demonstrates that to separate these hypotheses with a score margin $\gamma$ , the norm of the predictor's weight vector ( $B$ ) must grow as $B \geq \gamma / (\kappa\sqrt{2\delta})$ .
- As graph complexity ( $d$ ) increases, $\delta$ scales as $O(1/d^2)$ . Consequently, the required norm $B$ diverges to infinity. Since training methods like SFT and DPO operate under the constraint of bounded norms (lazy training), they are geometrically incapable of distinguishing near-miss hypotheses as complexity grows.
Agentic Causal Bayesian Optimization (A-CBO): To escape this obstruction, the authors propose a framework that moves the discrete decision-making process outside the LLM's RKHS.
- Interventional Oracle: A frozen LLM acts solely as an oracle answering binary interventional queries (e.g., "Does $V_j$ change under $do(V_i=v)$ ?"). Lemma 2 proves that while observational data may be near-misses, interventional responses are kernel-separated by a constant margin $\rho$ independent of $\delta$ .
- External Bayesian Loop: An external agent maintains a probability distribution over candidate graphs in the probability simplex $\Delta^{n-1}$ . It uses information-theoretic scoring (Information Gain) to select the most discriminative intervention, queries the LLM, and updates the belief distribution via Bayes' rule.
- Convergence: Because the decision logic operates in the simplex rather than the RKHS, the convergence rate is independent of the near-miss parameter $\delta$ . Theorem 2 proves A-CBO converges to the correct graph in $O(\log n)$ rounds, provided the true graph is interventionally distinguishable.

Key Contributions

Theoretical Proof of Failure: The paper establishes that the inability of LLMs to separate near-miss causal hypotheses is a fundamental geometric obstruction intrinsic to kernel-type predictors (SFT, DPO, ICL) in the lazy regime, not a limitation of model scale or data volume.
A-CBO Framework: The introduction of Agentic Causal Bayesian Optimization, which decouples the LLM's role (answering local binary queries) from the causal decision (global hypothesis selection), thereby bypassing the kernel obstruction.
Extended Benchmark: The creation of Extended Corr2Cause, a new benchmark scaling from 7 to 24 variables with 18,000 test samples, designed to stress-test causal discrimination at scale where near-miss gaps shrink to $O(1/d^2)$ .

Experimental Results

Corr2Cause (Original): A-CBO matches or exceeds the performance of fine-tuned baselines (e.g., LLaMA-7B FT) without any gradient updates. For instance, a high-tier A-CBO model (GLM-5.1) achieved an F1 of 93.2, surpassing the fine-tuned LLaMA-7B (92.0).
Extended Corr2Cause (d=7–24):
- Scalability: Fine-tuned models (SFT and DPO) degrade catastrophically as graph size increases, dropping to near-random accuracy (35.1% for SFT at $d=21-24$ ). In contrast, A-CBO maintains robust performance (83.2% accuracy for high-tier models) across all graph sizes.
- Advantage Growth: The performance gap between A-CBO and fine-tuned baselines grows monotonically with graph complexity. On the hardest graphs ( $d=21-24$ ), A-CBO outperforms SFT by +44.7 percentage points.
- Ablation: The agentic loop itself drives the performance. Applying A-CBO to zero-shot models yields massive gains (e.g., +59.3 pp for Qwen3-30B), confirming that the improvement stems from the architectural shift rather than raw model capability.

Significance and Claims
The paper claims that interventional querying is not merely a heuristic but a provably necessary and sufficient escape from the RKHS obstruction. By restricting the LLM to a non-decisional oracle role and delegating hypothesis discrimination to an external Bayesian loop, A-CBO achieves reliable causal discovery at scale without requiring the model to learn new representations or leave the lazy training regime. This work provides a principled foundation for LLM agents in science and policy, arguing that genuine causal intelligence requires moving beyond correlational supervision to interventional reasoning structures that respect the geometric limits of current learning paradigms.