Technical Summary: Causal Physics Steering in Video World Models via Concept Activation Vectors

Problem Statement

Video world models, which learn representations of physical dynamics for tasks like model-based reinforcement learning and autonomous driving, currently lack mechanisms to control their physical expectations at inference time. While recent interpretability research identified a Physics Emergence Zone (PEZ) within the middle transformer layers of VideoMAE—where physical plausibility is represented separately from other visual features—it remained unclear whether this structure could be directly manipulated to alter the model's reasoning. Existing controllable generation methods (e.g., ControlNet) rely on training-time conditioning or post-hoc adapters that require gradient-based optimization and paired supervision, failing to address the underlying question of where and how physical knowledge is represented internally and whether it can be directly intervened upon without retraining.

Methodology

The authors propose physics steering, a training-free, inference-time intervention method that utilizes Concept Activation Vectors (CAVs) to manipulate physical reasoning.

1. Physics Emergence Zone (PEZ) Identification:
   Following prior work, the authors train linear logistic regression probes at each layer of a pretrained VideoMAE to predict physical plausibility (possible vs. impossible). They identify the PEZ as the cluster of layers (specifically layers 0–5, peaking at layer 5) where probe accuracy is near-peak.

2. CAV Extraction:
   For each layer within the PEZ, the weight vector ($w_l$) of the trained linear probe is extracted and normalized to form the physics CAV ($v_l$). This vector defines a direction in the representation space associated with "physically impossible" scenarios. To handle cases where the number of training samples is smaller than the feature dimension, the authors apply PCA and a direction-flip check to ensure the vector aligns correctly with the concept.

3. Inference-Time Intervention:
   During inference, the method modifies the hidden states of patch tokens at a target layer $l^*$ by adding a scaled version of the CAV:
   $$\tilde{H}_{l^*}(x)_i = H_{l^*}(x)_i + \alpha \cdot v_{l^*}$$
   where $\alpha$ is a steering strength parameter.

   • $\alpha > 0$: Shifts the representation toward "physically impossible."
   • $\alpha < 0$: Shifts the representation toward "physically possible."
   • $\alpha = 0$: Baseline (no change).

   The intervention is applied to the encoder's hidden states; since VideoMAE is an encoder, the method alters downstream physics-sensitive predictions rather than generating pixel-space video frames directly.

4. Evaluation Metrics:
   The efficacy of steering is measured using:

   • Flip Rate (FR): The fraction of test videos whose classification changes relative to the baseline.
   • Score Delta ($\Delta P$): The mean change in the probability of being "impossible."
   • Directional Purity (DP): The cosine similarity between the observed representation shift and the intended CAV direction.
   • Representation Drift (RD): The magnitude of the shift in representation space.

Key Contributions

1. Physics Steering with CAVs: The authors demonstrate that linear probe weights at PEZ layers provide interpretable steering directions. Injecting these vectors allows for bidirectional control of the model's physical judgment (driving $P(\text{impossible})$ to 1.0 or 0.0) without updating model weights.
2. Layer-Specific Intervention Analysis: By injecting the same vector at different layers, the authors show that the steering effect is strictly localized to the PEZ and earlier layers. Interventions at layers 6–11 result in a 0.00 flip rate, confirming that the relevant physics representation is established and accessible within the PEZ.
3. Subspace Orthogonality: The study measures the angle between the physics CAV and the motion direction subspace, finding an orthogonality of 90.0°. This supports the hypothesis that physics and motion are encoded in separate, non-interfering subspaces.
4. Intrinsic Dimensionality Estimation: Through iterative orthogonal probe training, the authors find that the physics subspace at the PEZ is dominated by approximately 2–3 directions. This low dimensionality explains why a single-vector steering approach is highly effective.
5. Principle-Specific Steering: Separate CAVs trained for distinct intuitive physics principles (Object Permanence, Continuity, and Solidity) are found to be nearly orthogonal (e.g., 86.1° between Continuity and Solidity), suggesting that specific physical principles can be steered independently.

Results

Experiments were conducted on the IntPhys benchmark using a frozen VideoMAE-base model.

• Steering Efficacy: At the primary PEZ layer (Layer 5), steering with $|\alpha| \ge 5$ reliably saturates the model's judgment. Positive steering drives $P(\text{impossible})$ to 1.0, while negative steering drives it to 0.0. The directional purity reaches 1.00 at the injection layer, indicating a highly structured intervention.
• Localization: The effect is strictly layer-dependent. Injecting the PEZ-trained CAV into layers 6–11 produces no change in the physics judgment measured at Layer 5, confirming that the physics representation is localized to the PEZ and earlier layers.
• Geometry: The physics CAV is orthogonal to the motion direction (90.0°), a higher degree of separation than previously reported (69–83°). This suggests that steering physics does not inherently disrupt motion representations.
• Disentanglement: Different physics principles (O1, O2, O3) occupy distinct directions within the PEZ space, with O2 and O3 being nearly orthogonal, enabling potential principle-specific control.

Significance and Claims

The paper claims that physical reasoning in VideoMAE is not only readable (via linear probes) but also directly steerable at inference time. By identifying a localized Physics Emergence Zone and utilizing CAVs, the authors provide a method to control a model's physical expectations without retraining, fine-tuning, or architectural changes.

The significance lies in:

• Causal Control: Moving beyond correlation to demonstrate that intervening in specific representation layers causally changes the model's physical plausibility judgments.
• Interpretability to Action: Bridging the gap between mechanistic interpretability (finding where knowledge is) and representation engineering (using that knowledge to control behavior).
• Safety and Evaluation: The method offers a way to systematically stress-test world models by generating physically impossible scenarios without requiring new dataset collection, which is valuable for AI safety evaluation.

The authors maintain a modest scope, noting that while the method controls the encoder's internal representations, producing steered pixel-space video frames requires coupling the method with a decoder, which is left for future work. They also emphasize that the existence of a PEZ in other architectures (e.g., DINOv2, Stable Video Diffusion) remains an open empirical question.