🤖 machine learning

The Predictive-Causal Gap: An Impossibility Theorem and Large-Scale Neural Evidence

This paper establishes a structural "predictive-causal gap" as an impossibility theorem demonstrating that minimizing prediction error in neural networks systematically causes models to encode environmental dynamics rather than the target system's causal structure, leading to catastrophic failure in out-of-distribution generalization.

Original authors: Kejun Liu

Published 2026-05-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kejun Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand a complex machine, like a car engine. The machine has two parts: the engine (the "system" you care about) and the weather outside (the "environment").

The robot's goal is to predict what will happen next. The paper argues that if you just tell the robot, "Minimize your prediction errors," it will fail to learn about the engine. Instead, it will become obsessed with the weather.

Here is the breakdown of the paper's findings in simple terms:

1. The "Easy Target" Problem

Imagine the engine is a wild, jittery thing that changes its mind every second. The weather outside, however, is slow and steady; it takes a long time to change.

If you ask a student to predict the future, they will naturally focus on the thing that is easiest to guess. The weather is predictable; the engine is chaotic. So, the student (the AI) learns a perfect model of the weather but ignores the engine entirely.

The paper calls this the Predictive-Causal Gap. The AI is doing exactly what you asked it to do (predict the future accurately), but it is answering the wrong question. It is tracking the environment, not the system you actually wanted it to understand.

2. The "Wrong Answer" is Actually the "Right Answer"

You might think, "Maybe the AI just needs more brainpower or better training." The paper says no.

The authors proved mathematically that this isn't a mistake or a bug. It is the correct solution to the problem you gave the AI. If the environment is slower or less noisy than the system, the mathematically perfect way to predict the future is to ignore the system and focus on the environment.

Analogy: If you ask a detective to solve a crime by looking for the most obvious clues, they will ignore the subtle evidence inside the house and focus on the traffic outside because the traffic patterns are clearer. The detective isn't "bad"; they are just following the instructions too well.

3. Bigger Models Make It Worse

Usually, in AI, we believe that bigger models (more data, more computing power) lead to better understanding. This paper found the opposite.

When they made the AI models bigger and gave them more complex tasks:

The models got better at predicting the future (lower error rates).
But they got worse at understanding the system (they became "causally blind").

In high-dimensional tests (where the environment is huge), the best AI models became so focused on the environment that they essentially had zero sensitivity to the system they were supposed to model. They were perfect at predicting the weather, but they knew nothing about the engine.

4. The "Grounding" Fix

The researchers tried a trick called Operational Grounding. Instead of letting the AI predict everything, they forced it to only predict the specific parts of the system it cares about (the engine).

Result: This helped. The AI stopped ignoring the engine as much.
Limitation: It didn't fully fix the problem. As long as the environment influences the system, the AI still has an incentive to look at the environment. To truly fix it, you have to explicitly tell the AI, "Here is the system, and here is the environment. Do not mix them." You have to draw a hard line in the sand.

5. Why This Matters for "World Models"

Many modern AI systems (like Large Language Models or "World Models") are built on the idea that if you just predict the next word or the next frame of a video perfectly, the AI will magically learn the "causal structure" of the world (how things actually work).

This paper says: That premise is false.

If you just minimize prediction error, the AI will learn to track whatever is easiest to predict, not whatever is causally important. It will become a master of correlation (what happens together) but a failure at causation (what causes what).

Summary

The Problem: AI trained only to predict the future will ignore the complex things you care about if there are simpler, slower things (like the environment) it can predict instead.
The Cause: It's not a bug; it's a fundamental law of how prediction works. The "easiest" path is not the "causal" path.
The Scale: Making AI bigger doesn't fix this; it just makes the AI better at ignoring the right things.
The Solution: You cannot just "predict your way" to understanding. You must explicitly define what the "system" is and restrict the AI's attention to it.

The paper concludes that we are currently "sharpening the answer to the wrong question." We are building AI that is incredibly good at predicting the noise, but blind to the signal.

Technical Summary: The Predictive-Causal Gap

Problem Statement

The paper addresses a fundamental failure mode in predictive representation learning, a paradigm dominant in fields ranging from world models and neural state-space models to large language models (LLMs) trained via next-token prediction. The prevailing premise is that minimizing prediction error on observations will naturally lead the learned representation to capture the causal structure of the underlying system.

The authors challenge this premise by defining a "system" ( $s$ ) as the degrees of freedom a modeler cares about, and an "environment" ( $e$ ) as all other variables. They demonstrate that when environment modes are slower or less noisy than system modes, the optimal predictive representation does not track the system. Instead, it tracks the environment. This phenomenon is termed the predictive-causal gap: the degradation in causal fidelity (the fraction of encoder sensitivity allocated to system degrees of freedom) that occurs when optimizing a predictive objective rather than a causal one.

Methodology

The authors employ a three-tiered analytical approach to characterize this gap:

Theoretical Proof (Impossibility Theorem):
- The authors prove Theorem 1, establishing that for any model class containing all linear encoders, there exist stable linear-Gaussian dynamics where every minimizer of the population predictive risk is strictly misaligned with the system subspace.
- They demonstrate that the set of dynamics exhibiting this property is open and of positive measure in parameter space, meaning it is not a rare corner case but a generic property.
- Corollary 2 proves that increasing model capacity (enlarging the model class) cannot resolve this issue; it can only lower the risk further, thereby tightening the commitment to the non-causal (environment-dominant) encoder.
Large-Scale Linear-Gaussian Experiments:
- A parameter sweep was conducted over 539 configurations of linear-Gaussian dynamics, resulting in 2,695 independent training runs using two-layer MLP encoders.
- The experiments measured causal fidelity ( $|\partial \phi / \partial s| / (|\partial \phi / \partial s| + |\partial \phi / \partial e|)$ ) and prediction error.
- High-dimensional extensions ( $N=10, 50, 100$ environment modes) were analyzed to test scaling effects.
Nonlinear Dynamics Validation:
- Single-layer GRU predictors were trained on a Duffing oscillator coupled to a hidden Ornstein–Uhlenbeck environment.
- Two training conditions were compared: unconstrained (predicting the full state) and operational grounding (restricting the loss to system observables only).
- Robustness was tested via out-of-distribution (OOD) evaluation under environment distribution shifts.

Key Results

1. Systematic Failure in Linear Systems

In a sweep of 160 deterministic linear configurations, the optimal encoder aligned with the system (New Zealand/NZ encoder) in only 25% of cases (the diagonal, decoupled cases).
In 75% of cases (any nonzero off-diagonal coupling), the optimal encoder departed from the system axis.
The Information Bottleneck (IB) regularization was tested but found to fail; the compression-optimal direction aligned with the prediction-optimal direction, reinforcing the environment bias rather than correcting it.

2. Neural Network Evidence

Across 2,695 training runs, the mean causal fidelity of the optimal neural network encoder was 0.49, with only 2.5% of configurations exceeding 0.70.
The neural network achieved a 99.3% lower prediction error than the optimal linear encoder, yet this improved predictive performance corresponded to a representation further from the causal encoding.
High-Dimensional Collapse: As the number of environment dimensions ( $N$ ) increased to 100, the causal fidelity collapsed to $\sim 10^{-8}$ (effectively zero), while the prediction error gap grew to 92%. The optimal encoder became "causally blind."

3. Nonlinear Confirmation and Operational Grounding

In nonlinear Duffing–GRU tasks, unconstrained predictors learned environment-dominant representations in 55% of tasks, compared to 24% under operational grounding.
OOD Robustness: Under environment distribution shifts, unconstrained models suffered a median Mean Squared Error (MSE) inflation of 1.82 $\times$ , whereas grounded models remained stable at 1.00 $\times$ .
The paper notes that operational grounding (restricting loss to system observables) partially suppresses the gap but cannot fully recover causal fidelity without an explicit system-environment boundary in the architecture.

Significance and Claims

The paper claims to identify the predictive-causal gap as a structural limit of learning, not an optimization artifact, a finite-data issue, or a problem solvable by scaling.

Scaling Laws: The authors argue that empirical scaling laws (smooth power-law decay of loss with parameters/data) quantify the "wrong quantity." They measure the reduction in predictive risk, which can accompany movement away from causal structure. Scaling "sharpens the answer to the wrong question."
Self-Supervised Objectives: The failure is inherent to any objective that computes loss on an unpartitioned observation space. Whether using masked autoencoding, joint-embedding predictive architectures (JEPA), or contrastive learning, the absence of an explicit system-environment distinction in the loss function permits environment-dominant minimizers.
Operational Interface: The paper posits that recovering causal structure requires an "operational interface"—a specification of which degrees of freedom belong to the system and are permitted as encoder inputs. Techniques like RLHF, instruction tuning, and verifiable-reward post-training are framed not as alignment overlays, but as the necessary operational interfaces that convert ill-posed predictive problems into well-posed identification problems.

In conclusion, the paper asserts that a model trained to predict the next step in a composite system will learn to track whatever is easiest to predict (often the environment), not what is causally relevant. This is described as "the right answer to the wrong problem."