Technical Summary: DynaForcing

1. Problem Statement: Dynamic Collapse in Self-Forcing

The paper addresses a critical failure mode in audio-driven avatar generation known as dynamic collapse. While recent self-forcing distillation methods (leveraging Distribution Matching Distillation, or DMD) have enabled real-time, infinite-length streaming avatar generation, they suffer from a systematic degradation of temporal dynamics.

Despite achieving high perceptual quality scores (sharp images, good aesthetics), the student models converge to a "near-static optimum." In this state:

• Lip-sync accuracy and expression diversity are severely suppressed.
• Temporal dynamics (mouth movement, micro-expressions, head motion) vanish, resulting in videos that appear visually frozen despite the presence of driving audio.

The authors trace this collapse to two specific mechanisms inherent in the standard self-forcing paradigm:

1. Mode-Seeking Reverse KL Objective: DMD minimizes the reverse KL divergence $D_{KL}(p_\theta \| p_{data})$. This objective is "mode-seeking," penalizing the student for generating data outside the teacher's support but imposing no cost for ignoring specific modes (such as high-motion dynamics). Consequently, the model gravitates toward the "easiest" mode: static content that satisfies visual quality metrics without requiring complex temporal generation.
2. Unanchored Self-Conditioning: Unlike Ground-Truth (GT)-anchored approaches (e.g., CausVid) that use noisy GT latents as input for all blocks, self-forcing builds the entire Key-Value (KV) cache exclusively from the student's own previous outputs. If the student generates a low-motion block, all subsequent blocks condition on this static history, creating a compounding feedback loop that reinforces the collapse across the entire sequence.

2. Methodology: DynaForcing

To resolve the quality–dynamics trade-off, the authors propose DynaForcing, a training framework that intervenes at three distinct levels of the pipeline: Data, Loss, and Conditioning.

A. Hybrid Forcing (Data Level)

To break the feedback loop of unanchored self-conditioning, the authors introduce Hybrid Forcing.

• Mechanism: At the start of each training rollout, the starting point for the denoising trajectory is probabilistically chosen. With probability $p_{data}$, the model starts from a noised ground-truth latent ($\Psi(B_{GT}, t_{start})$) rather than pure Gaussian noise.
• Effect: This injects ground-truth dynamics (motion patterns, lip movements) as structural information that the student must refine. It provides an "anchor" that prevents the model from collapsing into a zero-motion basin, offering a gradient direction toward dynamically rich outputs.
• Optimization: The probability of sampling early noise steps ($t_{start}$) is redistributed to ensure the model traverses sub-trajectories where dynamic structure is still recoverable.

B. Dynamics-Aware Reward Regularization (Loss Level)

The authors leverage the Reinforcement Learning (RL) interpretation of DMD to introduce explicit constraints against static outputs.

• Mechanism: An explicit dynamics reward ($R_{dyn}$) is defined based on two metrics:
  1. Sync-C: Lip-sync confidence from a pretrained SyncNet.
  2. ExpVar: Standard deviation of 3DMM expression coefficients (measuring expression variance).
• Integration: This reward is used to exponentially reweight the DMD gradient: $\nabla_\theta L \propto \exp(\alpha \cdot R_{dyn}) \cdot (s_{real} - s_{fake})$.
• Effect: This amplifies the gradient for samples with rich dynamics and suppresses it for static outputs, effectively steering the optimization toward reward-rich modes without requiring backpropagation through the reward model.

C. Reference Perturbation (Conditioning Level)

The authors identify that standard training allows models to rely on static details from the reference image (identity, background, pose) rather than generating audio-driven motion.

• Mechanism: Reference images are perturbed using an off-the-shelf image editing model to alter background, lighting, and viewpoint while preserving identity (verified via ArcFace similarity).
• Effect: This decouples identity from extraneous visual details. The model is forced to rely on the audio signal for temporal dynamics rather than copying static content from the reference frame.
• Curriculum: Perturbed references are bucketed by structural similarity (SSIM) to ensure a balanced training distribution.

D. Efficient Training Optimizations

To address the high memory cost of self-forcing (which requires retaining the full rollout graph for backpropagation), the authors introduce:

1. Computation Graph Pruning: Detaching the KV cache from the computation graph, as cross-block gradients were found to have negligible impact on training outcomes.
2. Block-wise Gradient Replay: A two-phase process where the sequence is generated without gradients, stored, and then replayed block-by-block with gradients enabled.

• Result: These optimizations reduce GPU memory requirements by over 10x (from 128 to 8 H100s) with negligible impact on quality.

3. Key Contributions

1. Formalization of Dynamic Collapse: The paper systematically characterizes dynamic collapse in self-forcing distillation, attributing it to the combination of reverse-KL mode-seeking and unanchored self-conditioning.
2. DynaForcing Framework: A novel training framework combining Hybrid Forcing, Dynamics-Aware Reward Regularization, and Reference Perturbation to recover dynamics while preserving visual quality.
3. Efficiency Gains: Introduction of computation graph pruning and gradient replay, achieving a ~10x reduction in GPU-hour requirements for self-forcing training.
4. Empirical Validation: Demonstration that DynaForcing recovers dynamics to teacher-comparable levels on both short-form (~10s) and long-form (>5min) benchmarks without early stopping.

4. Experimental Results

Evaluated on the GenBench datasets (ShortVideo and LongVideo) using a 14B parameter teacher model (WanS2V):

• Dynamic Recovery:
  • Dyn-Deg (Dynamic Degree): Improved from 0.31 (standard self-forcing) to 0.73 (DynaForcing), matching the teacher's 0.72.
  • ExpVar (Expression Variance): Increased from 0.69 to 2.02 (surpassing the teacher's 1.81).
  • Sync-C (Lip-sync): Improved from 7.03 to 7.68.
• Visual Quality:
  • Visual quality metrics (ASE, IQA) improved alongside dynamics, resolving the typical trade-off. DynaForcing achieved ASE 3.55 and IQA 4.58, outperforming the standard self-forcing baseline (3.44/4.51).
• Real-Time Performance:
  • Maintained 45.2 FPS real-time streaming capability.
• Long-Form Stability:
  • Unlike baselines where dynamics degrade over time (e.g., LiveAvatar's ExpVar dropping from 0.69 to 0.57 in long videos), DynaForcing maintained strong dynamics (ExpVar 1.93) throughout >5 minute generations.
• User Study:
  • In a pairwise study with 25 participants, DynaForcing was preferred over real-time baselines 80.6% of the time for motion naturalness.

5. Significance and Claims

The paper claims that DynaForcing successfully resolves the fundamental conflict between real-time streaming efficiency and temporal expressiveness in avatar generation. By addressing the root causes of dynamic collapse—specifically the lack of ground-truth anchoring and the bias of the reverse KL objective—the method enables the training of high-fidelity, real-time avatars that do not require early stopping to preserve motion.

The authors emphasize that their approach is training-side only, meaning it does not alter the inference architecture or speed. They modestly note limitations, including a reliance on pretrained models (SyncNet, 3DMM) for reward calculation and the need for future validation of the data-level components on tasks beyond avatar generation. The work provides a systematic solution to a failure mode that has hindered the deployment of high-quality, interactive digital humans.