Technical Summary: Do Video Foundation Models Understand Intuitive Physics? A Layerwise Probing Analysis

Problem Statement

While video foundation models trained on internet-scale data demonstrate strong performance in tasks ranging from action recognition to text-conditioned generation, it remains unclear whether these models encode genuine intuitive physical understanding or merely rely on superficial visual statistics and dataset-specific shortcuts. Recent literature suggests that modern video architectures are motivated as "world models" capable of capturing structured latent evolution, yet evaluating whether they truly encode physical constraints (e.g., permanence, solidity, spatio-temporal continuity) is difficult. Success on standard benchmarks may reflect generic motion sensitivity rather than robust physical reasoning. This paper investigates whether pretrained video representations inherently encode physics-relevant structure and how the accessibility of this information varies across pretraining paradigms, network depths, and readout mechanisms.

Methodology

The authors employ a frozen-feature probing framework, adhering to the paradigm established by El Banani et al. (2024) and Joseph et al. (2026). Rather than fine-tuning models for specific tasks, the study extracts representations from frozen backbones and trains lightweight probes to predict physical plausibility.

Models Evaluated

Three distinct pretraining paradigms are compared using their largest available backbones:

1. Predictive Joint-Embedding (V-JEPA family): Includes V-JEPA, V-JEPA 2, and V-JEPA 2.1. These models predict target representations in a latent space rather than reconstructing raw pixels.
2. Masked Video Reconstruction (VideoMAE family): Includes VideoMAE and VideoMAE-v2. These models are trained to recover masked spatio-temporal content.
3. Diffusion-Based Generation (LTX-Video): A large-scale text-to-video generator optimized via progressive denoising.

Datasets

Two benchmarks targeting physical understanding are utilized:

• IntPhys2 (Bordes et al., 2025): Tests sensitivity to four core principles (permanence, immutability, spatio-temporal continuity, solidity) using a "Violation of Expectation" (VOE) metric. A scene is correct only if all possible clips are ranked higher than impossible ones.
• Minimal Video Pairs (MVP) (Krojer et al., 2025): Designed to prevent shortcut-based performance by presenting visually similar video pairs with opposite answers. The metric is pair consistency. The authors adapt the original text-conditioned QA format into a binary physical-plausibility task to align with frozen-feature probing.

Probing Protocols

Representations are extracted from four relative depths (approx. 25%, 50%, 75%, 100%) of each architecture. For diffusion models, probing is also conducted across 10 noise levels. Three probe types are evaluated:

1. Linear Probe: A single layer mapping pooled embeddings to the output, testing for linearly decodable information.
2. MLP Probe: A multi-layer perceptron with non-linear activations, testing for information present but not linearly separable.
3. Temporal Attentive Probe: Operates directly on the sequence of token embeddings using self-attention and cross-attention layers to explicitly model temporal interactions.

Control Conditions

To distinguish genuine temporal physics sensitivity from static appearance cues, two control conditions are applied:

• Frame-Shuffled: Frames are randomly permuted to disrupt motion trajectories while preserving frame content.
• Single-Frame: A single frame is repeated for the clip duration to eliminate temporal variation entirely.

Key Results

1. Pretraining Paradigm Performance

The V-JEPA family consistently achieves the strongest results across both benchmarks, particularly when evaluated with temporally expressive readouts.

• V-JEPA 2.1 leads on IntPhys2 with 66.67% VOE accuracy.
• Base V-JEPA leads on MVP with 94.03% pair consistency.
• VideoMAE remains competitive on MVP (92.01%) but shows a wider gap on IntPhys2.
• LTX-Video (diffusion) recovers weaker but non-trivial signals, trailing by ~10% on MVP and >19% on IntPhys2 compared to the best V-JEPA models.
• Note: VideoMAE-v2 showed extremely low performance on IntPhys2 (15.69%), attributed by the authors to training instability.

2. Layerwise Emergence

Physics-relevant information is not uniformly distributed; it is weakest in early layers and becomes most accessible at intermediate-to-late depths.

• MVP: Shows a clear late-layer trend. Performance generally improves toward the final layers (e.g., V-JEPA pair consistency rises from ~59% at 25% depth to ~87% at the final layer).
• IntPhys2: Exhibits a less monotonic profile, often peaking between 50% and 75% depth before potentially declining at the final layer.
• LTX-Video: The strongest signals are concentrated in the middle of the backbone and across the denoising trajectory, rather than at the extremes.

3. Readout Accessibility

The explicitness of physics information depends heavily on the task and probe capacity.

• MVP: Information is largely not directly readable by linear probes. Pair consistency for V-JEPA jumps from ~48% (linear) to ~87% (MLP) and ~94% (Temporal Attentive). This indicates that MVP-relevant physics requires modeling non-linear or explicit temporal interactions.
• IntPhys2: A larger fraction of the signal is linearly decodable (e.g., V-JEPA linear probe reaches ~51% VOE). However, more expressive probes (MLP and Temporal Attentive) still provide significant gains, especially for the strongest models.

4. Temporal Controls

Disrupting temporal structure substantially reduces performance, confirming that the signal is not purely static.

• MVP: Temporal disruption causes a catastrophic collapse. Frame shuffling reduces pair consistency by 39–61%, and single-frame repetition reduces it by 86–100%.
• IntPhys2: Results are mixed. While single-frame repetition is highly damaging (e.g., -85% for V-JEPA linear), frame shuffling sometimes results in negligible drops (0.00% for V-JEPA temporal attention). This suggests that some IntPhys2 VOE performance may rely on order-insensitive plausibility cues or unordered multi-frame evidence rather than robust causal trajectory modeling.

Significance and Claims

The paper claims that intuitive-physics knowledge emerges reliably in pretrained video representations, but its accessibility is contingent on three factors:

1. Pretraining Objective: Predictive joint-embedding learning (V-JEPA) yields the most accessible and robust physics-relevant features compared to masked reconstruction or diffusion objectives.
2. Representational Depth: Physical understanding is not present in early layers; it emerges and becomes most accessible at intermediate-to-late depths.
3. Readout Mechanism: The ability to decode physics information depends on the probe's capacity to model non-linearities and temporal dynamics. While some signals are linearly decodable (particularly in IntPhys2), complex physical reasoning (as in MVP) requires expressive, temporally aware readouts.

The authors conclude that while video foundation models encode physical structure, they do not necessarily reason about physics in a human-like, robust manner. The study highlights that benchmark success can be driven by specific architectural inductive biases and that evaluating physical understanding requires careful disentanglement of temporal dynamics from static appearance. Future work is suggested to disentangle pretraining objectives from confounding factors like model scale and architecture.