DiReCT: Disentangled Regularization of Contrastive Trajectories for Physics-Refined Video Generation

1. Problem Statement

Current flow-matching video generators produce high-fidelity, temporally coherent videos but frequently violate elementary physical laws (e.g., objects passing through each other, incorrect gravity, or impossible kinematics).

• Root Cause: Standard reconstruction objectives (like pixel-level $\ell_2$ or latent distances) penalize frame deviations without distinguishing between physically consistent dynamics and impossible ones.
• The Semantic-Physics Entanglement Obstacle: While contrastive learning (specifically Contrastive Flow Matching or $\Delta$FM) has shown promise in separating trajectories for different conditions, it fails in text-conditioned video generation.
  • In text-to-video, a single prompt couples semantic content (what is depicted) with physical behavior (how it moves).
  • Naive negative sampling (randomly picking a different prompt) often results in "negative" conditions that share substantial velocity-field structure with the "positive" condition due to semantic proximity.
  • Gradient Conflict: When the positive and negative conditions are semantically similar, the contrastive gradient (pushing trajectories apart) directly opposes the flow-matching gradient (pulling the trajectory toward the target). This conflict degrades training performance rather than improving it.

2. Methodology: DiReCT

The authors propose DiReCT (Disentangled Regularization of Contrastive Trajectories), a lightweight post-training framework that resolves the gradient conflict through entanglement-aware multi-scale contrastive learning.

A. Theoretical Foundation

The authors formalize the gradient conflict, deriving a precise alignment condition. They prove that contrastive learning is beneficial only when the velocity gap ($\delta$) between positive and negative targets is large enough to dominate the self-interference term. If $\delta$ is small (semantic overlap), the gradients oppose each other.

B. Two-Scale Contrastive Strategy

DiReCT decomposes the contrastive objective into two complementary terms to handle different regimes of the gradient conflict spectrum:

1. Macro-Contrastive Learning (MaNS):

   • Goal: Global trajectory separation without interference.
   • Mechanism: The training prompts are partitioned into $K$ semantic clusters using a frozen text encoder. For a given positive sample, negatives are drawn exclusively from different semantic clusters.
   • Effect: This ensures the negative samples are semantically distant, guaranteeing a large velocity gap ($\|\delta\|$) and a cooperative gradient signal.

2. Micro-Contrastive Learning (MiNS):

   • Goal: Fine-grained physical distinction between semantically similar scenes.
   • Mechanism: Constructs "hard negatives" that share the exact scene semantics but differ along a single, controlled physics axis.
   • Implementation: An LLM (Qwen2.5-7B-Instruct) perturbs the original prompt by modifying exactly one of five physics dimensions:
     • Kinematics (speed/acceleration profile)
     • Forces (direction/type of force)
     • Material (viscosity, elasticity, friction)
     • Interaction (bounce vs. stick, shatter vs. dent)
     • Magnitude (intensity/volume)
   • Hard Negative Generation: The base model generates a video from this perturbed prompt. The model is then trained to push its velocity field away from this physically inconsistent trajectory while maintaining the scene's visual structure.

C. Distributional Anchoring

To prevent catastrophic forgetting (where the model loses visual quality or temporal coherence while learning physics), DiReCT includes a velocity-space regularizer. It penalizes the divergence between the fine-tuned model's velocity field and that of a frozen reference model ($\theta_{ref}$), acting as a KL-divergence equivalent in velocity space.

D. Combined Objective

The total loss function combines:

1. Standard Flow Matching (Reconstruction).
2. Macro-Contrastive Loss (MaNS).
3. Micro-Contrastive Loss (MiNS).
4. Distributional Anchoring Loss (preventing drift).

3. Key Contributions

1. Formalization of Gradient Conflict: The paper identifies and mathematically proves that naive contrastive flow matching fails in text-conditioned settings due to semantic-physics entanglement, establishing a precise condition under which negative sampling harms training.
2. DiReCT Framework: A novel post-training method that decouples the contrastive signal into macro (semantically distant) and micro (physics-perturbed hard) scales, effectively resolving the gradient conflict without requiring additional training time compared to standard Supervised Fine-Tuning (SFT).
3. Physics-Aware Negative Generation: A pipeline using LLMs to generate hard negatives that isolate specific physical violations while preserving scene semantics, enabling fine-grained physics learning.

4. Experimental Results

The method was evaluated on Wan 2.1-1.3B (a 1.3B parameter model) against baselines and state-of-the-art (SOTA) models.

• Benchmarks:
  • VideoPhy: Measures physical commonsense and semantic adherence.
  • WorldModelBench: Evaluates instruction following, common sense, and adherence to specific physical laws (mass conservation, penetration, fluid dynamics).
• Performance:
  • VideoPhy: DiReCT improved the Physical Commonsense (PC) score by 16.7% over the baseline and 11.3% over SFT.
  • WorldModelBench: DiReCT achieved the highest total score (5.68), surpassing CogVideoX-5B (5.33) and Mochi (4.91).
  • Efficiency: DiReCT achieved these results with only 1.3B parameters, significantly outperforming models with 3.8x to 7.7x more parameters.
• Training Dynamics: Unlike SFT, which saturates after ~5k steps, DiReCT continues to improve throughout 15k steps, indicating the contrastive signal provides sustained learning value.
• Qualitative Improvements: Visual comparisons show DiReCT correctly handles forward kinematics (cars moving forward), maintains object integrity (surfers not merging with sails), and simulates buoyancy (wood floating downstream), whereas baselines fail these physics constraints.

5. Significance

• Efficiency over Scale: The paper demonstrates that targeted physics-aware fine-tuning is more effective for improving physical realism than simply scaling model size.
• Solving the Entanglement Problem: DiReCT provides a generalizable solution to the "semantic-physics entanglement" problem in text-to-video generation, offering a blueprint for how to apply contrastive learning in continuous, entangled conditioning spaces.
• Practicality: The method adds no significant inference cost and minimal training overhead (comparable to SFT), making it a viable upgrade for existing video generation pipelines.
• World Modeling: By improving physical commonsense, DiReCT moves video generators closer to functioning as reliable "world models" capable of predicting physically plausible outcomes, which is critical for downstream applications like autonomous driving or robotics simulation.