Technical Summary: Self-Refining Video Sampling

Problem Statement

Modern video generators, particularly those based on diffusion and flow matching models, have advanced significantly as early-stage world models. However, they still struggle with complex physical dynamics, often failing to produce physically realistic outputs. Common issues include inconsistencies in motion, implausible object interactions, and temporal artifacts.

Existing approaches to mitigate these issues face significant limitations:

• External Verifiers: Methods using rejection sampling with external models (e.g., LLMs or physics engines) are computationally expensive due to low acceptance rates and often lack domain generality.
• Additional Training: Post-training strategies involving synthetic data or fine-tuning on augmented datasets require substantial computational resources and high-quality, domain-specific data.
• Fine-Grained Motion: Current methods struggle to capture fine-grained motion dynamics accurately, limiting their applicability to real-world tasks like robot manipulation.

Crucially, unlike Large Language Models (LLMs) which can iteratively critique and revise their own token outputs, video generators lack an explicit internal feedback signal for critique and correction, especially given the high dimensionality and temporal coupling of video data.

Methodology: Self-Refining Video Sampling

The authors propose Self-Refining Video Sampling, a training-free inference-time method that utilizes a pre-trained video generator as its own self-refiner. The core approach reinterprets the flow matching objective as a time-conditioned Denoising Autoencoder (DAE) and introduces an iterative inner-loop refinement process.

1. Predict-and-Perturb (P&P)

The method leverages the connection between flow matching and DAEs. At any fixed noise level $t$ during inference, the model acts as a denoiser. The authors introduce a two-step iterative loop:

• Predict: The model predicts a clean endpoint latent $\hat{z}_1$ from the current noisy state $z_t$ using the learned vector field.
  $$\hat{z}_1 = z_t + (1-t)u_\theta(z_t, t)$$
• Perturb: The predicted clean latent is perturbed back to the same noise level $t$ by adding noise $\epsilon$.
  $$z_{t}^{(k+1)} = t \hat{z}_1^{(k)} + (1-t)\epsilon$$

This Predict-and-Perturb (P&P) cycle creates a pseudo-Gibbs Markov chain that iteratively steers the latent representation toward higher-density regions of the learned video distribution. Conceptually, this pulls the generation toward temporally coherent and physically plausible states without requiring external models. The refinement is applied primarily during the early stages of the sampling trajectory (low noise levels), where coarse motion and structure are determined.

2. Uncertainty-Aware Refinement

A naive application of multiple P&P iterations can lead to over-saturation and artifacts, particularly in static regions (e.g., backgrounds) due to the accumulation of classifier-free guidance (CFG) effects. To address this, the authors propose Uncertainty-aware P&P:

• Self-Consistency Measure: The method computes an uncertainty map by comparing the reconstructed predictions between consecutive P&P iterations. Regions with high variance (low self-consistency) are identified as uncertain.
• Selective Refinement: An uncertainty mask is generated based on a confidence threshold $\tau$. P&P refinement is applied only to uncertain regions (typically moving objects), while confident regions (static backgrounds) are preserved.
• Efficiency: This strategy is implemented without additional Network Function Evaluations (NFEs) by reusing predictions computed during the standard sampling steps.

Key Contributions

1. Training-Free Self-Refinement: The paper introduces a novel paradigm where a pre-trained video generator refines its own outputs at inference time without external verifiers, additional training, or domain-specific data.
2. Predict-and-Perturb (P&P) Framework: It establishes a theoretical link between flow matching and DAEs to enable iterative inner-loop refinement, effectively acting as a local search mechanism that improves motion coherence and physical plausibility.
3. Uncertainty-Aware Strategy: The authors develop a mechanism to prevent over-refinement artifacts by selectively refining only uncertain, motion-related regions based on the model's internal self-consistency signals.
4. Broad Applicability: The method is demonstrated to be compatible with various state-of-the-art flow matching models (e.g., Wan2.1, Wan2.2, Cosmos-2.5) and can be integrated into existing ODE solvers in a plug-and-play manner.

Experimental Results

The authors evaluated their method on state-of-the-art video generators across multiple benchmarks:

• Motion Coherence (Dynamic-bench & VideoJAM-bench):
  • On the Dynamic-bench (challenging motions like gymnastics and multi-object interactions), the method achieved a 73.57% win rate in human preference over the default sampler and 70.57% over the training-free guidance method FlowMo.
  • Automated metrics (VBench) showed significant improvements in motion coherence and consistency scores.
• Physical Realism (Robotics & PAI-Bench):
  • In image-to-video (I2V) robotics tasks, the method improved grasp success rates by +11.0% (Cosmos) and +8.4% (Wan) compared to the base ODE sampler.
  • It outperformed verifier-based rejection sampling (best-of-4) while incurring significantly lower computational overhead (1.5x vs 4.0x inference time).
• Physics Alignment (VideoPhy2 & PhyWorldBench):
  • Human evaluation showed an 84% preference for the method's outputs over the default sampler regarding physical commonsense.
  • The method generated more accurate free-fall trajectories in PisaBench experiments compared to baselines.
• Spatial Consistency:
  • The method significantly improved spatial consistency in videos with large camera motions (e.g., 360-degree rotations), preserving scene content better than standard solvers.
• Visual Reasoning:
  • The method improved success rates in graph traversal tasks (from 0.1 to 0.8) but showed limited gains in discrete semantic tasks like maze solving, highlighting the local-search nature of the refinement.

Significance and Claims

The paper claims that Self-Refining Video Sampling provides a practical and broadly applicable approach to enhance existing pre-trained video generators. By reusing the generator's own learned priors as a self-refiner, the method achieves significant improvements in motion coherence, physical plausibility, and spatial consistency without the computational burden of additional training or external verification systems.

The authors emphasize that this approach complements recent advances in large-scale video generative models, offering a principled sampling strategy to make video generation more reliable for real-world applications such as robot manipulation and physical simulation. They note that while the method acts as a local search effective for correcting temporal inconsistencies and motion errors, it is not a substitute for global planning in tasks requiring discrete semantic correctness.