Technical Summary: Live Avatar

1. Problem Statement

Audio-driven avatar generation aims to synthesize photorealistic human face videos where motion is driven by an input audio stream. While diffusion models have achieved state-of-the-art visual fidelity, deploying them for interactive, real-time, and infinite-length generation faces two fundamental, conflicting challenges:

1. Long-Horizon Consistency: Existing methods suffer from compounding errors during autoregressive rollout, leading to identity drift, color artifacts, and temporal instability within minutes. Current solutions either fail to scale to large models (e.g., 14B parameters) or degrade rapidly under minute-level generation.
2. Real-Time–Fidelity Trade-off: Large-scale diffusion models (e.g., 14B) offer superior quality but are inherently slow due to sequential multi-step denoising. Conversely, real-time approaches typically rely on non-iterative generation or small models, sacrificing visual fidelity.

The paper posits that no existing method simultaneously achieves streaming, real-time throughput, and infinite-length stability using a large-scale diffusion model.

2. Methodology

Live Avatar is an algorithm-system co-designed framework built upon a 14-billion-parameter diffusion model. It addresses the challenges through a two-stage training pipeline and a novel system-level parallelism strategy.

A. Algorithmic Innovations

The framework employs a two-stage pipeline to adapt a pre-trained bidirectional diffusion model into a causal, few-step streaming model:

• Stage 1: Diffusion Forcing Pretraining with Motion-Frame Scaffolding:
  • The model is pre-trained using block-wise independent noise scheduling and causal attention masks.
  • Motion-Frame Scaffolding: To accelerate convergence, noisy motion frames (context frames preceding the current clip) are injected as auxiliary context. This acts as a "scaffold" to teach dynamics–identity decoupling without the high cost of self-forcing rollouts. This stage yields a 5× speedup in convergence compared to training without scaffolding.
• Stage 2: Self-Forcing Distribution Matching Distillation (DMD) with History Corrupt:
  • The bidirectional teacher is distilled into a causal, few-step student model.
  • History Corrupt: A critical design choice where the Key-Value (KV) cache stores noisy representations rather than clean history.
    • Mechanism: Gaussian noise acts as a low-pass filter, suppressing high-frequency accumulated artifacts while preserving low-frequency motion dynamics.
    • Benefit: This forces the model to rely on the persistent "sink frame" for identity and appearance, while the noisy cache handles motion. This effectively decouples identity from motion, preventing error accumulation and enabling stable extrapolation beyond 10,000 seconds despite training on ~3-second clips.
  • Adaptive Attention Sink (AAS): To prevent distribution drift caused by the mismatch between the real reference image and the model's generation manifold, the sink frame is replaced by the model's own first generated latent after the initial block.
  • Rolling RoPE: To mitigate test-time conditioning drift, the RoPE (Rotary Positional Embedding) index of the sink frame is dynamically adjusted to maintain a consistent relative position to the current noisy states, preventing attention degradation over long horizons.

B. System Innovations: Timestep-forcing Pipeline Parallelism (TPP)

To break the sequential sampling bottleneck of diffusion models, the authors propose Timestep-forcing Pipeline Parallelism (TPP):

• Concept: Instead of sequential denoising on a single device, the $T$ denoising steps are partitioned across $T$ GPUs. Each GPU is assigned a fixed timestep $t_i$ and repeatedly performs the transformation $t_i \to t_{i-1}$.
• Mechanism:
  • The sequential diffusion chain is converted into an asynchronous spatial pipeline.
  • Because the KV cache is noisy (History Corrupt), each GPU maintains its own local rolling KV cache without needing inter-GPU communication for cache updates.
  • GPUs only pass the compact latent representation to the next device.
• Result: Throughput is determined by a single denoising forward pass rather than the sum of all steps, achieving ideal linear speedup proportional to the number of denoising steps.

3. Key Contributions

1. Causal, Streamable Adaptation Framework: A two-stage training method that distills a bidirectional model into a causal, few-step streaming model, utilizing motion-frame scaffolding to achieve a 5× convergence speedup.
2. Long-Horizon Stability: Demonstrates that storing only noisy history in the KV cache (History Corrupt) enables train-short-infer-long capabilities, allowing stable generation beyond 10,000 seconds. This is complemented by AAS and Rolling RoPE to address distribution and positional drift.
3. Timestep-forcing Pipeline Parallelism (TPP): A system-level optimization that breaks the sequential bottleneck by assigning fixed timesteps to GPUs, enabling high-throughput inference for large models.
4. Standardized Benchmark (GenBench): Introduction of a benchmark including long-video test cases exceeding five minutes to evaluate long-horizon performance.

4. Experimental Results

The system was evaluated on 5 NVIDIA H100 GPUs with a 14B parameter model:

• Performance: Achieves 45 FPS with a Time-to-First-Frame (TTFF) of 1.21 s.
• Scale: To the authors' knowledge, this is the first practical real-time streaming of a 14B diffusion model for infinite-length avatar generation.
• Quality Metrics (GenBench):
  • Short Video: Outperforms state-of-the-art methods (e.g., Ditto, OmniAvatar, WanS2V) in visual quality (IQA: 4.51, ASE: 3.44) and identity consistency (Dino-S: 0.96), while maintaining competitive audio-visual synchronization.
  • Long Video: Maintains near-identical quality across durations (e.g., IQA 4.51 $\to$ 4.76, Dino-S 0.96 $\to$ 0.97), whereas baselines like OmniAvatar and WanS2V show significant degradation in visual quality and identity consistency over time.
• Ablation Studies:
  • Removing History Corrupt causes severe degradation in identity consistency (Dino-S drops to 0.81) and visual quality.
  • Removing AAS leads to progressive color desaturation.
  • Removing Rolling RoPE triggers identity drift (Dino-S drops to 0.86).
  • TPP is shown to be essential for achieving high FPS; without it, the system runs at ~5 FPS on 4 GPUs.

5. Significance and Claims

The paper claims that Live Avatar resolves the long-standing trade-off between model scale, visual fidelity, and real-time performance in audio-driven avatar generation. By co-designing the algorithm (noisy KV caching, distillation) and the system (TPP), it enables the first practical deployment of a 14B diffusion model for infinite-length, real-time streaming.

The authors emphasize that their long-horizon stability strategies are grounded in the static-scene prior inherent to avatar interactions (where the subject and background remain largely unchanged). They acknowledge that as native long-duration training becomes feasible, these priors may become unnecessary. Additionally, they note that while the system achieves real-time throughput (45 FPS), the end-to-end latency (approx. 3s including network transport) does not yet meet the stringent requirements for seamless bidirectional conversation, though it represents a significant step forward for interactive digital communication.