Technical Summary: Distilling Physical Priors into Streaming World Models (PhyS)

Problem Statement

Streaming world models aim to predict future visual states through online rollouts while maintaining physically coherent dynamics over long horizons. However, current state-of-the-art approaches, which typically distill pretrained bidirectional Diffusion Transformers (DiTs) into few-step causal generators, suffer from two fundamental limitations:

1. Limited Physical Priors: Generic bidirectional teachers acquire only limited physical understanding from visually oriented pretraining, often failing to capture the nuances of real-world interactions.
2. Distillation Loss: The process of distilling bidirectional knowledge into a causal student further degrades these limited priors. Causal models struggle to preserve the long-range dependencies essential for physical causality, leading to error accumulation and violations of basic physical constraints (e.g., gravity, collision, deformation) during autoregressive rollouts.

Existing methods often rely on synthetic data or coarse language-guided signals, which fail to provide the fine-grained, observation-grounded supervision necessary for learning complex physical dynamics like fluid flow, soft-body deformation, and phase transitions.

Methodology: The PhyS Framework

The authors propose PhyS, a three-stage framework designed to inject physical priors into streaming world models and align them for causal generation.

1. Data Construction: PhyS-120K

To address the lack of real-world physical data, the authors constructed PhyS-120K, a dataset of 120,804 real-world physical-interaction videos.

• Taxonomy: Videos are categorized by observable state evolution into four domains: fluid dynamics, soft-body deformation, phase transitions, and rigid-body mechanics.
• Annotation: Unlike standard captions, each clip is annotated with structured physical descriptions generated by a vision-language model (Qwen3-VL-235B). These annotations include object properties (e.g., stiffness, mass density) and causal event sequences (e.g., "press head exerts force," "ball yields").
• Curation: The pipeline involves filtering for motion quality and safety, followed by rebalancing to ensure diverse process coverage.

2. Three-Stage Training Pipeline

Stage 1: Physics-Aware Supervised Fine-Tuning (SFT)
A bidirectional 14B DiT (Wan2.1-14B) is fine-tuned on PhyS-120K using LoRA. The model learns from the structured video-caption pairs to internalize real-world physical dynamics, resulting in a physics-aware teacher model.

Stage 2: Bidirectional-to-Causal Distillation
The 14B teacher is distilled into a lightweight 1.3B causal DiT using DMD (Distribution Matching Distillation).

• The student performs autoregressive self-rollouts (few-step generation) conditioned on its own history.
• The objective minimizes the Kullback-Leibler (KL) divergence between the noisy student rollout distribution and the teacher distribution, transferring the learned physical priors to a model capable of low-latency, block-wise streaming.

Stage 3: Online Policy Optimization with Temporal Credit Routing (TCR)
To further align the causal generator with physical consistency, the authors introduce an online reinforcement learning stage.

• The Challenge: Standard video-level rewards are too coarse; a single reward for a whole video cannot distinguish between a physically plausible sequence and one containing a localized violation.
• The Solution (TCR): TCR aggregates feedback over overlapping temporal windows.
  1. Window-Level Scoring: A frozen reward model (V-JEPA2 predictor combined with an optical-flow reward from RAFT) evaluates the physical consistency of overlapping windows within a generated rollout.
  2. Group Normalization: Rewards are normalized within each window across a group of rollouts to prevent easy or difficult temporal regions from dominating updates.
  3. Overlap Routing: Advantages are routed to specific denoising actions based on the overlap between the reward window and the causal generation block. This localizes the physical feedback signal, allowing the model to correct specific temporal deviations rather than applying a global penalty.

Key Contributions

1. PhyS-120K Dataset: A large-scale, real-world dataset of physical interactions with structured annotations of object properties and causal state transitions, covering diverse physical processes.
2. PhyS Framework: A three-stage training pipeline that successfully transfers physical priors from a bidirectional teacher to a lightweight causal generator for streaming video generation.
3. Temporal Credit Routing (TCR): A novel optimization strategy that addresses the temporal credit assignment problem in video generation by routing window-level physical rewards to aligned denoising actions, improving physical consistency without regenerating the entire sequence.

Experimental Results

The authors evaluate PhyS on PhysicsIQ and other physics-aware benchmarks (VideoPhy, VideoPhy2, PhyGenBench).

• PhysicsIQ: PhyS improves the baseline Wan2.1-14B teacher by 18.2%.
• Causal Generators: When applied to causal forcing methods, PhyS yields significant gains:
  • Self Forcing: +23.7%
  • Rolling Forcing: +14.8%
  • Causal Forcing: +31.4%
• Generalization: The method demonstrates consistent improvements across physical-consistency metrics on VideoPhy, VideoPhy2, and PhyGenBench.

Significance and Claims

The paper claims that PhyS represents a significant step toward building world models that adhere to the laws of physics. By moving beyond generic visual pretraining and introducing a dedicated pipeline for acquiring and distilling physical priors, the authors demonstrate that streaming world models can achieve long-horizon coherence without violating fundamental physical constraints. The introduction of TCR specifically addresses the "temporal coarseness" of existing reward mechanisms, enabling fine-grained alignment of causal generation with physical reality. The authors emphasize that their approach relies on real-world data grounding rather than synthetic simulation or coarse language prompts, offering a more robust foundation for physically plausible video generation.