Technical Summary: Stable Velocity: A Variance Perspective on Flow Matching

1. Problem Statement

While Flow Matching (FM) and Conditional Flow Matching (CFM) offer elegant frameworks for generative modeling by learning probability flow without explicitly simulating forward SDEs or PF-ODEs, they suffer from a fundamental limitation: the high variance of their training targets. In CFM, the training objective relies on a single-sample Monte Carlo estimate of the true marginal velocity field, $v_t(x_t | x_0)$, to approximate the marginal velocity $v_t(x_t)$.

The authors identify that this single-sample estimator exhibits significant variance, particularly at timesteps where the marginal distribution remains close to the prior (high-variance regime). This high variance leads to:

1. Unstable optimization: Noisy gradients that slow convergence.
2. Inference inefficiency: The inability to exploit the deterministic nature of the flow in specific regimes, leading to unnecessarily small integration steps.
3. Ineffective auxiliary supervision: Existing representation alignment methods (like REPA) often apply supervision uniformly, failing to recognize that semantic alignment is ill-posed in high-variance (noisy) regimes but highly effective in low-variance regimes.

2. Methodology

The paper proposes the Stable Velocity framework, which addresses these issues by explicitly characterizing the variance of conditional velocity targets and adapting both training and sampling strategies accordingly.

2.1 Variance Analysis

The authors analyze the variance of the CFM target, $V_{CFM}(t)$, across different data distributions (GMMs, CIFAR-10, ImageNet). They reveal a two-regime structure:

• High-Variance Regime ($t \in [\xi, 1]$): Near the prior (noise), the posterior $p(x_0|x_t)$ is multimodal, causing the conditional velocity to fluctuate significantly.
• Low-Variance Regime ($t \in [0, \xi]$): Near the data distribution, the posterior concentrates on a single reference sample, causing the conditional velocity $v_t(x_t | x_0)$ to nearly coincide with the true marginal velocity $v_t(x_t)$.

2.2 Training: Stable Velocity Matching (StableVM)

To reduce variance without introducing bias, the authors propose Stable Velocity Matching (StableVM).

• Mechanism: Instead of using a single reference sample $x_0$, StableVM constructs a composite conditional path using a batch of $n$ reference samples $\{x_0^i\}_{i=1}^n$.
• Target: The training target is a self-normalized importance-weighted average of conditional velocities over the reference batch:
  $$\hat{v}_{StableVM}(x_t) = \frac{\sum_{k=1}^n p_t(x_t | x_0^k) v_t(x_t | x_0^k)}{\sum_{j=1}^n p_t(x_t | x_0^j)}$$
• Properties: Theoretically proven to be unbiased (sharing the same global minimizer as CFM) while strictly reducing the variance of the training target by a factor of $O(1/n)$.
• Class-Conditional Extension: To handle sparsity in class-conditional generation, a memory bank is used to maintain a diverse set of reference samples per class, ensuring the mixture remains effective even with low per-batch class frequency.

2.3 Training: Variance-Aware Representation Alignment (VA-REPA)

Building on the two-regime insight, the authors propose VA-REPA to optimize auxiliary semantic supervision.

• Insight: Representation alignment (e.g., aligning latent features with pretrained encoders) is only informative in the low-variance regime where semantic information is preserved. In the high-variance regime, the signal is noisy and ill-posed.
• Strategy: A weighting function $w(t)$ is applied to the alignment loss, restricting supervision to the low-variance regime ($t < \xi$). This prevents the model from learning from noisy, uninformative signals.

2.4 Inference: Stable Velocity Sampling (StableVS)

The authors leverage the low-variance regime to accelerate sampling without fine-tuning.

• Insight: In the low-variance regime, the flow dynamics are effectively deterministic, and the trajectory approximates a straight line with constant velocity.
• Mechanism: StableVS derives closed-form simplifications for the reverse SDE and PF-ODE in this regime. For linear interpolants, the update rule simplifies to:
  $$x_\tau = x_t + (\tau - t)v_t(x_t)$$
• Implementation: The sampler uses standard solvers (e.g., Euler, DPM-Solver++) in the high-variance regime but switches to the closed-form StableVS update in the low-variance regime. This allows for large integration steps (fewer total steps) without degrading sample quality.

3. Key Contributions

1. Variance-Theoretic Perspective: The paper provides a principled analysis revealing a dimensionality-dependent two-regime structure (high vs. low variance) in stochastic interpolants, governing both training dynamics and inference behavior.
2. StableVM: An unbiased, variance-reduced training objective that aggregates multiple reference samples to stabilize optimization, outperforming standard CFM and existing variance-reduction techniques (like STF) in convergence speed and sample quality.
3. VA-REPA: A regime-aware adaptation of representation alignment that selectively applies auxiliary supervision in the low-variance regime, improving training efficiency and stability.
4. StableVS: A finetuning-free acceleration strategy that exploits the deterministic nature of the low-variance regime to achieve >2x sampling speedups on state-of-the-art models.

4. Experimental Results

The framework was validated on ImageNet 256×256 and various large-scale text-to-image (T2I) and text-to-video (T2V) models (SD3.5, Flux, Qwen-Image, Wan2.2).

• Training Efficiency:
  • StableVM + VA-REPA consistently outperformed vanilla REPA and other baselines (REG, iREPA) across model scales (SiT-B to SiT-XL) and training durations.
  • On ImageNet, the method achieved competitive FID and IS scores with significantly fewer training epochs compared to baselines requiring end-to-end fine-tuning (e.g., REPA-E).
  • Ablation studies confirmed that restricting alignment to the low-variance regime yields better FID than uniform application.
• Sampling Speed:
  • StableVS achieved >2x inference acceleration (e.g., matching 30-step baselines with ~17-20 total steps) on SD3.5, Flux, and Wan2.2.
  • Crucially, this acceleration did not degrade sample quality. Reference metrics (PSNR, SSIM, LPIPS) and generative metrics (GenEval, T2V-CompBench) remained comparable to or better than standard solvers with reduced steps.
  • Qualitative results showed that StableVS outputs were visually indistinguishable from 30-step baselines, whereas naive step reduction caused noticeable artifacts.

5. Significance and Claims

The paper claims that explicitly modeling the variance structure along the generative trajectory provides a principled foundation for designing more efficient generative models.

• Unified Framework: It unifies variance reduction in training and regime-aware acceleration in sampling under a single theoretical perspective.
• Practical Impact: The methods are modular and compatible with existing architectures (e.g., SiT, DiT) and solvers, offering immediate improvements in training stability and inference speed without requiring architectural changes or expensive fine-tuning.
• Theoretical Insight: The work challenges the view of diffusion/flow trajectories as uniform processes, demonstrating that distinct regimes (high vs. low variance) require distinct optimization and sampling strategies.

The authors emphasize that their approach complements existing techniques (like dimensionality reduction or improved solvers) by addressing the specific source of variance induced by reference-sample stochasticity in flow matching objectives.