Technical Summary: Identifiability Without Gaussianity

Problem Statement

The paper addresses a fundamental limitation in the temporal consistency of World Models. While Joint-Embedding Predictive Architectures (JEPAs) have demonstrated success in recovering latent variables under specific conditions, recent work by Klindt, LeCun, and Balestriero [1] established that linear identifiability (the linear recovery of true latent variables) in JEPAs is achievable if and only if the world's latent dynamics follow a Gaussian, stationary process (specifically an Ornstein-Uhlenbeck transition).

The authors argue that this Gaussian boundary imposes a severe constraint on physical modeling. Most physical systems of scientific interest (e.g., projectiles, phase transitions, turbulent flows) are non-Gaussian and non-mean-reverting. For these systems, the statistical alignment mechanisms used in current World Models result in a representation error that grows monotonically with time. Consequently, statistical World Models possess a finite "temporal horizon" beyond which predictions become unreliable, regardless of model capacity or training data volume. The paper seeks to determine if near-infinite temporal consistency is achievable for non-Gaussian systems and, if so, what architectural properties are required.

Methodology

The paper introduces and formalizes the Physics-Grounded Symbolic Architecture (PGSA) as an alternative to statistical alignment. The methodology relies on three core definitions:

1. Atom Registry: A finite set of deterministic, executable functions ($A$) representing known physical laws with well-defined domains.
2. State Graph: A directed graph where nodes are typed physical variables (mass, velocity, etc.) and edges are labeled by atoms from the registry that compute one variable from another.
3. Causal Basis: The condition where the Atom Registry contains a composition of atoms that exactly replicates the world's true transition function $W$ on a dense subset of the domain.

The authors contrast this with Statistical World Models (including pixel-space, latent-space, and transformer-based models) which learn representations via spectral alignment or variational objectives.

To validate their theoretical claims, the authors:

• Formalize Proofs: The algebraic cores of Theorems 2, 3, 4, and 7 are formalized in Lean 4 using Mathlib4 v4.31.0 with zero "sorry" placeholders.
• Empirical Comparison: They simulate seven physical systems (ranging from conservative harmonic oscillators to chaotic Lorenz attractors) and four engineering systems (solenoid, plate, cantilever, heat sink).
• Metrics: They measure linear identifiability ($R^2$) of the optimal isotropic-Gaussian representation against the true latents and calculate the temporal horizon ($T^*$) where error exceeds a fixed tolerance $\delta$.

Key Contributions

1. Theoretical Proofs of Identifiability and Consistency

The paper proves three primary results regarding the PGSA:

• Theorem 2 (Symbolic Identifiability): If the Atom Registry contains a causal basis for the world generator, the PGSA achieves exact linear identifiability ($h(z)=z$) and zero single-step error ($\epsilon_1=0$) for all physical regimes, regardless of the latent distribution $p(z)$.
• Theorem 3 (Near-Infinite Temporal Consistency): For a PGSA, the accumulated error over $t$ steps is bounded solely by machine precision ($\mu$) and the Lipschitz constant of the system: $\epsilon_t \le t \cdot \mu \cdot \|W\|_{Lip}^t$. For non-chaotic systems, this allows for an unbounded number of transitions before error exceeds a tolerance.
• Theorem 4 (Statistical Temporal Ceiling): Conversely, for any statistical World Model trained via spectral alignment on a non-Gaussian system, there exists a finite time $T^*$ after which the error exceeds any fixed tolerance. This horizon is determined by the representation bias $\kappa(p)$ induced by the Gaussian prior, not by model capacity.

2. The Mechanism of Failure: Representation Bias Propagation

The paper introduces Lemma 1 (Representation Bias Propagation) to explain why statistical models fail over time. It demonstrates that non-identifiable representations install a state-dependent bias $b(z)$ that the model cannot remove. Because the transition operator operates on this biased state ($z + b(z)$) at every step, the error compounds additively (or exponentially in chaotic systems). In contrast, a PGSA pays a one-time perceptual error at $t=0$ but operates on the true state thereafter, preventing compounding bias.

3. Formal Comparison of Paradigms

The authors establish a strict ordering of temporal consistency horizons (Theorem 5):
$$T^*_{\pi} \le T^*_{stat} < T^*_{PGSA}$$

• Pixel-space models fail structurally due to non-injective rendering (Proposition 8).
• Latent-space statistical models (including Transformers with statistically aligned latents) are bounded by the statistical ceiling $T^*_{stat} = \delta / \kappa(p)$.
• PGSA is bounded only by machine precision $\mu$, yielding a horizon ratio of $\kappa(p)/\mu$ (approx. $10^{13}$ for typical systems) compared to statistical models.

4. Approximate Identifiability

Theorem 7 addresses the case of an incomplete Atom Registry. It proves that if the registry covers only part of the dynamics ($W_{known}$), the error is bounded by the magnitude of the uncovered residual ($W_{unknown}$). Crucially, adding new atoms to the registry reduces this error monotonically without retraining, contrasting with statistical models which require full retraining to incorporate new physical knowledge.

Results

• Linear Identifiability: Empirical measurements on seven physical systems confirm that linear identifiability degrades as the system departs from Gaussianity. Non-chaotic systems showed $R^2 \approx 0.90$, while chaotic systems were closer to Gaussian ($R^2 \approx 0.99$) but limited by Lyapunov instability.
• Temporal Horizons:
  • For five non-chaotic systems, the PGSA horizon was the machine-precision bound ($\approx 4.5 \times 10^{13}$ steps).
  • For two chaotic systems, the horizon was limited by the Lyapunov time ($\approx 31$ Lyapunov times), a fundamental physical limit rather than an architectural one.
  • Statistical models (pixel, latent, transformer) were shown to have finite horizons orders of magnitude shorter, dictated by the system's non-Gaussianity.
• Engineering Validation: The PGSA approach was validated on four engineering systems (magnetostatics, modal analysis, static structural, thermal), reproducing closed-form solutions within finite-element tolerance.

Significance and Claims

The paper claims to establish that symbolic grounding in the causal generator of the world's dynamics is the sufficient (and for non-Gaussian regimes, the only) condition for achieving near-infinite temporal consistency.

Key claims include:

1. Statistical Alignment is Insufficient: For non-Gaussian physical systems, no amount of model capacity or data can overcome the structural bias introduced by statistical alignment objectives. The "Gaussian boundary" is a hard limit for statistical World Models.
2. Architecture Determines Horizon: The temporal consistency of a World Model is determined by where error enters the system. Statistical models re-apply representational bias at every step; PGSA amortizes a single bounded perceptual error.
3. Transformers are Not a Panacea: Transformer-based world models, when relying on statistically aligned latents (as is the case for Sora, Genie, Cosmos, and Dreamer), are subject to the same statistical ceiling as latent-space models. Attention mechanisms can increase the constant in the horizon but cannot eliminate the bias $\kappa(p)$.
4. Constructive Route: The work provides a constructive route to long-horizon consistency by decoupling execution from statistical perception, provided a causal basis (Atom Registry) exists.

The authors explicitly state they do not claim symbolic architectures are universally superior to all possible future models, but rather that under the assumptions identified by Klindt et al., statistical identifiability does not guarantee long-horizon consistency, whereas symbolic causal execution does.