Technical Summary: Proprio: Latent Self-Scoring and Inference-Time Refinement for Physically Plausible Video Generation

1. Problem Statement

Modern video generative models, particularly diffusion-based architectures, have achieved remarkable visual fidelity. However, they frequently violate basic physical principles, producing videos with unrealistic collisions, impossible object interactions, or inconsistent dynamics. These failures limit their utility as reliable world models for downstream applications like robotics and simulation.

Existing solutions to this problem typically rely on external feedback mechanisms:

• Training-time interventions: Injecting physical priors or using curated datasets (e.g., [35, 24]).
• Inference-time external evaluators: Using Vision-Language Models (VLMs) or pretrained latent world models (e.g., V-JEPA [37, 41]) to score or guide generation.

The authors identify two critical limitations in the external-feedback paradigm:

1. Bias and Misalignment: VLMs often conflate physical plausibility with text-image alignment or visual aesthetics. Pretrained world models operate in their own representation spaces, creating a mismatch with the generator's latent space.
2. Lack of Intrinsic Consistency: External evaluators are trained with different objectives and do not align with the generator's specific denoising dynamics, potentially missing generator-specific inconsistencies across sampling steps.

The paper posits a central question: Can a video generator leverage its own denoising dynamics as a self-contained signal to evaluate and improve the physical plausibility of its outputs?

2. Methodology: Proprio

The authors propose Proprio, a training-free, inference-time framework that enables a frozen video generator to assess and refine its own outputs. Inspired by biological proprioception (the sense of one's own movement), the method treats the model's internal flow residual under controlled perturbations as a self-scoring signal.

2.1 Core Concept: Flow Residuals as Self-Scoring

The method relies on the observation that samples consistent with the generator's learned distribution (i.e., natural dynamics) should induce smaller and more stable residuals when perturbed.

• Perturbation: For a generated latent video $z_0$, the method constructs perturbed latents $z_t$ at various scheduler timesteps $t$ using controlled noise $\epsilon$.
• Residual Calculation: The model predicts a flow field $\hat{v}_\theta(z_t, t, c)$. The target transport is $v^*(z_0, \epsilon) \approx \epsilon - z_0$. The flow residual is the discrepancy:
  $$r(z_0, \epsilon, t; c) = \hat{v}_\theta(z_t, t, c) - v^*(z_0, \epsilon)$$
• Scoring: The squared norm of this residual serves as a local score. Lower residuals indicate the model can better "explain" the sample under perturbation, implying higher physical plausibility.

2.2 Signal Aggregation and Refinement

To create a robust score, Proprio aggregates residuals across multiple dimensions:

1. Multi-Perturbation & Multi-Timestep: The score is averaged over multiple noise realizations and timesteps to reduce sensitivity to specific noise instances.
2. Inverse-Variance Weighting: Timesteps with high variance in residuals (unstable estimates) are downweighted, while stable timesteps are prioritized.
3. Dynamic Spatiotemporal Masking: A mask $\mathcal{M}$ is applied to focus the score on motion-relevant regions, filtering out static background noise that is less informative for physical plausibility. Three masking strategies are explored:
   • Motion Mask: Based on temporal differences in decoded frames.
   • Text-aware Mask: Based on cross-attention changes between latent and prompt tokens.
   • Attention-based Mask: Based on self-attention feature changes.

2.3 Inference-Time Strategies

Proprio utilizes the computed score $S(z_0)$ in two primary modes:

1. Best-of-N Search: Generate $N$ candidate latents and select the one with the lowest self-score.
2. Self-Refinement: Instead of retraining the generator, the method optimizes the initial sampling noise. It parameterizes the noise as $\zeta = \mu + \sigma \odot \zeta_0$ (where $\zeta_0$ is base noise) and uses gradient descent to update $\mu$ and $\sigma$ to minimize the Proprio score. A KL-divergence regularizer ensures the optimized noise remains close to the original Gaussian prior.

3. Key Contributions

1. Generator-Native Self-Scoring: Introduction of a training-free framework that reuses a frozen generator's denoising residual as an intrinsic proxy for physical plausibility, eliminating the need for external reward models.
2. Inference-Time Self-Alignment: A mechanism to turn this self-score into an optimization objective, refining outputs by updating only the initial noise parameters while keeping the generator weights fixed.
3. Empirical Validation: Demonstration of consistent improvements on both Text-to-Video (T2V) and Image-to-Video (I2V) benchmarks, showing that a single frozen generator can act as both sampler and self-evaluator.

4. Experimental Results

The authors evaluate Proprio on VideoPhy2 (T2V) and Physics-IQ (I2V) benchmarks, comparing against random selection, VLM-based scorers (Qwen-7B), and external world-model baselines (WMReward).

• Performance Gains:

  • On TurboWan2.2, Proprio improves Physics-IQ from 32.2 to 37.5 (+16.5%) and VideoPhy2-hard physical commonsense from 45.6 to 55.0 (+20.6%).
  • Proprio consistently outperforms VLM-based scoring and often surpasses WMReward, particularly in physical commonsense (PC) metrics.
  • A hybrid scoring approach (combining global and motion-aware scores) provides the most robust balance between semantic adherence and physical consistency.

• Human Evaluation:

  • In pairwise comparisons, human raters preferred Proprio-selected or refined videos for physical plausibility in approximately 67% of cases.
  • Visual quality was largely preserved, with raters showing no strong preference against the baseline in many cases, indicating the improvements are not at the expense of aesthetics.

• Ablation Insights:

  • Noise Levels: The most informative scoring signals arise from mid-level noise (0.3–0.6 of sampling steps). Low noise is too local, while high noise loses sample-specific structure.
  • Variance Weighting: Inverse-variance weighting significantly improves performance on non-distilled models by prioritizing stable timesteps.
  • Search vs. Refinement: Search (Best-of-N) is highly effective for T2V due to high scene diversity. For I2V, where candidates are anchored to the same image, refinement plays a more dominant role.

5. Significance and Claims

The paper claims that frozen video generators contain actionable internal signals for evaluating and improving physical plausibility. By reinterpreting the model's own flow residuals as a likelihood-preference signal, Proprio demonstrates that:

• External evaluators are not strictly necessary for physical alignment; the generator's internal dynamics are sufficient.
• Inference-time scaling (search and refinement) can yield substantial gains without retraining.
• The method offers a shift in perspective: future systems may increasingly exploit generator-internal signals for self-evaluation rather than relying solely on external supervision or auxiliary reward models.

Limitations Acknowledged:

• Proprio is limited by the knowledge encoded in the base generator; it cannot recover dynamics the model has not learned.
• It increases inference cost due to the need for multiple candidate scoring and refinement steps.
• Gradient-based refinement currently relies on distilled models for computational tractability, as backpropagating through long denoising trajectories in large models remains expensive.