🤖 AI

Interpreting Physics in Video World Models

This paper investigates how video world models represent physical properties, discovering that they do not use factorized variables like classical engines but instead rely on a distributed, high-dimensional representation that emerges at a specific intermediate depth within the model's architecture.

Original authors: Sonia Joseph, Quentin Garrido, Randall Balestriero, Matthew Kowal, Thomas Fel, Shahab Bakhtiari, Blake Richards, Mike Rabbat

Published 2026-02-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sonia Joseph, Quentin Garrido, Randall Balestriero, Matthew Kowal, Thomas Fel, Shahab Bakhtiari, Blake Richards, Mike Rabbat

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a movie of a ball rolling across a floor. Even if you’ve never seen a ball before, you intuitively know that if it hits a wall, it should bounce, and if it’s moving fast, it won't stop instantly. You have a "built-in physics engine" in your brain.

For a long time, computer scientists have wondered: Do AI models (like the ones that generate videos) actually have a "built-in physics engine," or are they just really good at guessing what the next pixel should look like based on patterns?

This paper, "Interpreting Physics in Video World Models," goes "under the hood" of these AI models to find out. Here is the breakdown of what they discovered, using some everyday analogies.

1. The "Physics Emergence Zone" (The Teenage Years)

The researchers found that AI models don't understand physics immediately.

Think of an AI model like a student going through school. In the "early grades" (the first few layers of the AI), the model only sees basic things—like colors and shapes. It’s like a toddler who sees a ball but doesn't understand "speed."

But then, at a very specific point—about one-third of the way through its "brain"—something magical happens. The researchers call this the Physics Emergence Zone. It’s like the "teenage years" where the model suddenly goes from seeing static pictures to understanding movement and rules. Suddenly, it can tell if a video is "impossible" (like a ball passing through a wall) or "possible."

2. Speed vs. Direction (The "How Fast" vs. "Where")

The researchers broke motion down into two parts: Speed (how fast) and Direction (where to).

They discovered that the AI is a bit of a specialist:

Speed is easy: The AI learns how fast things are moving almost immediately. It’s like a driver who knows they are pressing the gas pedal, even if they don't know which way the road turns.
Direction is hard: The AI doesn't actually "get" direction until it hits that "Physics Emergence Zone" mentioned above. It has to connect different parts of the video together to realize, "Oh, that object is moving that way."

3. The "Distributed" Brain (The Orchestra vs. The Soloist)

This is the most important finding. There are two ways an AI could represent physics:

The Physics Engine Way (The Soloist): The AI has a specific "folder" in its brain labeled "Velocity" and another labeled "Acceleration." It’s neat, organized, and easy to read.
The Distributed Way (The Orchestra): There is no single "folder." Instead, the concept of "direction" is spread out across hundreds of different tiny signals, all playing together like an orchestra. To change the direction of an object in the AI's mind, you can't just turn one knob; you have to adjust dozens of different "instruments" at once.

The researchers found that AI uses the Orchestra method. It doesn't use neat, tidy math formulas like a human physicist would. Instead, it uses a massive, complex, and "messy" web of connections. It’s not a "physics engine" in the classical sense, but it’s a "distributed" version that works just as well.

4. The "Circular" Secret (The Compass)

When the researchers looked at how the AI stores "direction," they found something beautiful. The AI doesn't store direction as a simple number (like "North = 1"). Instead, it stores it in a circular pattern, much like how biological brains (including ours!) work.

It’s like a compass where every direction is part of a continuous loop. This allows the AI to understand that if you turn 359 degrees, you are almost back to 0 degrees.

Summary: The Big Picture

If you asked a human, "How does that ball move?" they would say, "It has a velocity of 5 meters per second toward the left."

If you ask this AI, it doesn't have an answer in words. Instead, it has a massive, swirling, high-dimensional dance of data happening in its middle layers. It doesn't "calculate" physics; it "feels" the rhythm of motion through a complex web of patterns.

The takeaway: AI isn't thinking like a scientist with a calculator; it's thinking like a master musician who understands the "flow" of the world.

Technical Summary: Interpreting Physics in Video World Models

Authors: Sonia Joseph, Quentin Garrido, Randall Balestriero, Matthew Kowal, Thomas Fel, Shahab Bakhtiari, Blake Richards, Mike Rabbat.

1. Problem Statement

A fundamental question in AI is whether large-scale video world models (like V-JEPA or VideoMAE) represent physical reality using factorized, physics-engine-style representations (e.g., explicit, low-dimensional variables for velocity, mass, and direction) or through distributed, task-specific representations (heuristic-based, high-dimensional patterns). While these models show strong performance on intuitive physics benchmarks, their internal "representational regime" has remained a black box. Understanding this is crucial for scientific modeling (e.g., fluid dynamics) and for determining if these models possess a true "internal physics engine."

2. Methodology

The researchers conducted an interpretability study on two state-of-the-art video encoders: V-JEPA 2 (latent predictive objective) and VideoMAE-v2 (pixel reconstruction objective). They employed a multi-pronged diagnostic approach:

Layerwise Probing: Using linear and attentive-MLP probes to detect when physical information (direction, speed, acceleration, and "possible vs. impossible" physics) becomes accessible across the network's depth.
Synthetic Toy-Ball Dataset: A controlled environment using the Kubric simulator to provide ground-truth variables (velocity, acceleration, speed, and direction) for precise decoding.
Subspace Geometry & Orthogonality Analysis: Measuring the principal angles and projection overlap between different physical subspaces (e.g., does the "direction" subspace overlap with the "physics judgment" subspace?).
Targeted Attention Ablations: Suppressing specific attention heads (local vs. long-range) to identify the computational circuits responsible for spatiotemporal reasoning.
Latent Space Steering: Attempting to causally manipulate motion direction by intervening on the identified representational dimensions to see if the model's "perception" changes.

3. Key Contributions & Results

A. The Physics Emergence Zone (PEZ)
The study identifies a sharp transition at approximately one-third of the model's depth, termed the Physics Emergence Zone.

Scalar vs. Vector: Scalar quantities like speed and acceleration magnitude are available from very early layers. However, directional information only emerges at the PEZ.
Peak Representation: Physical representations peak in the middle layers and actually degrade toward the output layers, suggesting that final layers prioritize task-specific features over raw physical structure.

B. Distributed vs. Factorized Representations
The findings strongly favor distributed, task-specific representations over classical physics engines:

Orthogonality: Despite emerging at the same depth, the subspace for "motion direction" and the subspace for "possible-vs-impossible physics" are nearly orthogonal. This means the model does not "reuse" a single direction variable to make physics judgments; instead, it uses different circuits for different tasks.
High Dimensionality: Unlike language models, where concepts often align with a single direction, motion direction is encoded in a high-dimensional population code (requiring 40–80 dimensions to steer effectively).

C. Biological Parallels & Circuit Mechanisms

Circular Geometry: Direction is represented via a circular population code (similar to direction-selective neurons in the primate visual cortex), where individual neurons exhibit sinusoidal tuning.
Local Attention Heads: The PEZ is characterized by the emergence of spatiotemporally local attention heads. Ablating these local heads destroys the model's ability to reason about direction and physics but leaves static tasks (like ImageNet classification) unaffected.
Local-to-Global Transition: Directional information shifts from being "patch-local" (fragmented) to "globally distributed" (position-invariant) at the PEZ, mirroring the V1 $\to$ MT hierarchy in biological vision.

4. Significance

This paper provides the first direct evidence that modern video world models do not function like classical simulators. Instead, they implement a distributed, high-dimensional heuristic system that is remarkably efficient at making physical predictions.

Implications include:

AI Safety & Interpretability: Understanding that physical reasoning is distributed and high-dimensional warns against assuming we can "steer" or "fix" physical logic by manipulating a few simple variables.
Neuroscience: The alignment between transformer internal structures and biological motion-processing hierarchies suggests that video models could serve as useful computational models for studying the brain.
Scientific Modeling: It highlights the distinction between models that reconstruct pixels and those that predict latents, suggesting that the training objective (like V-JEPA's) is critical for the emergence of structured physical reasoning.