Technical Summary: Vorch-Streamer

Problem Statement

Real-time, long-form avatar audio–video generation faces two primary dilemmas when adapting pretrained bidirectional diffusion models to a streaming setting:

1. Exposure Bias and Visual Drift: Long-form streaming requires autoregressive generation where the model's own predictions serve as context for subsequent blocks. Small errors in appearance, motion, or synchronization accumulate over time, creating a train–test mismatch between clean training data and self-generated inference histories. This leads to compounding artifacts and temporal drift, destabilizing long-horizon generation.
2. Global Speech vs. Causal Context Mismatch: In Text-to-Audio-Video (T2AV) tasks, the input prompt describes a complete utterance. However, a causal generator operating in a streaming window can only attend to limited local history. Without explicit planning, the model struggles to determine which portion of the global transcript should be spoken next, often resulting in audio that starts mid-sentence, falls out of temporal alignment, or generates incorrect speech content.

Existing models are typically designed for short, offline clips with bidirectional attention, making them incompatible with streaming systems that must generate the next segment before future content is observed. Furthermore, many existing streaming avatars rely on externally supplied driving audio or reference frames, failing to jointly synthesize speech and video from text in a truly causal manner.

Methodology

Vorch-Streamer is a post-training framework designed to extend the pretrained bidirectional LTX2.3 audio–video foundation model into a causal, real-time, long-form streaming generator. The approach consists of three main stages:

1. Synthetic Corpus Construction

The authors constructed a synthetic corpus of 80,000 high-quality avatar audio–video clips (12–21 seconds duration) using the pretrained bidirectional LTX2.3 model. This ensures the training data is consistent with the model's initialization, providing model-consistent supervision for the subsequent causal training.

2. Causal Audio–Video Streaming (Stage 2)

To convert the bidirectional model into a block-autoregressive streaming generator, the authors employ a mixed training strategy:

• Mixed Teacher Forcing and Diffusion Forcing: The model is trained with a sample-level mixture where 10% of samples use clean ground-truth history (Teacher Forcing) and 90% use corrupted history (Diffusion Forcing). This reduces the train–test discrepancy by exposing the model to imperfect, self-generated contexts during training.
• Causal Attention Masking: The model preserves bidirectional attention within each synchronized audio–video block (approx. 1 second) to model fine-grained interactions but enforces causal attention across blocks.
• Bounded Context: To maintain constant memory usage, the model attends to a bounded window consisting of the first three blocks (persistent prefix) and the most recent preceding block.

3. Long-Horizon Self Forcing (Stage 3)

To address the accumulation of errors over long sequences, the framework applies Self Forcing with Distribution Matching Distillation (DMD):

• The causal student model generates full 12–21 second sequences block-by-block from its own predictions.
• A frozen bidirectional LTX2.3 model acts as a "real-score" teacher, representing the target distribution.
• A trainable "fake-score" model estimates the student's evolving distribution.
• The student is optimized to minimize the difference between its distribution and the teacher's distribution, effectively transferring the quality of the pretrained bidirectional model to long causal trajectories while exposing the model to its own inference-time rollout distribution.

4. LLM-Based Speech Planning

To resolve the mismatch between global text prompts and local causal generation, Vorch-Streamer introduces an explicit speech-planning pathway:

• An external Large Language Model (Fun-CosyVoice) predicts discrete speech-planning tokens at 25 Hz (40 ms units).
• These tokens are converted into continuous features and injected into the audio diffusion branch via a dedicated cross-attention module.
• A gated fusion operator adaptively combines these planning features with the original text-conditioned audio features.
• This decouples speech planning (what to say and when) from audio generation (how to synthesize it), allowing for interruption, switching, and the inclusion of a learnable silence token for interactive listening.

Key Contributions

1. Framework: Introduction of Vorch-Streamer, a post-training framework that adapts a pretrained bidirectional audio–video diffusion model for causal, real-time, long-form streaming.
2. Training Strategy: Construction of an 80K synthetic corpus and a novel training pipeline combining mixed Teacher/Diffusion Forcing with long-horizon Self Forcing and teacher distillation to align causal training with streaming inference.
3. Speech Planning: Proposal of an LLM-based speech-planning pathway that supplies continuous planning features to the audio branch, enabling explicit control over speech progression, interruption, and silent listening without fixing the entire future utterance at the stream's start.
4. Performance: Demonstration of real-time, long-form T2AV streaming at 27.12 FPS (exceeding the 24 FPS real-time threshold) while maintaining competitive synchronization and speech accuracy.

Experimental Results

The authors evaluated Vorch-Streamer on continuous long-form rollouts (approx. 2 minutes) against native T2AV baselines (LTX2.3, JoyAI-Echo, OmniForcing, Hallo-Live) and conditional TIA2V references (LiveAvatar, SoulX-FlashTalk).

• Speed: Vorch-Streamer achieves 27.12 FPS on a single NVIDIA H200 GPU, making it the only evaluated native T2AV method to exceed real-time playback. It is significantly faster than bidirectional LTX2.3 (1.83 FPS) and other streaming baselines.
• Speech Accuracy: The model achieves a Word Error Rate (WER) of 7.92%, comparable to the offline bidirectional LTX2.3 (7.59%). In contrast, baselines without explicit speech planning (OmniForcing, Hallo-Live) suffer catastrophic WERs (>90%) when extrapolated to long durations.
• Synchronization: The model maintains strong audio–lip synchronization (Sync-C: 6.62, Sync-D: 8.95), comparable to the much slower bidirectional LTX2.3.
• Long-Horizon Stability: Over a two-minute rollout, Vorch-Streamer exhibits minimal degradation in identity preservation (ArcFace similarity remains stable around 0.73–0.77) and scene consistency (CLIP similarity around 0.92–0.94). Other native T2AV methods show significant drift and identity loss over similar durations.
• Ablation Studies:
  • Removing the LLM speech planner results in a WER of 184%, confirming the necessity of explicit planning for causal T2AV.
  • Removing Stage 2 (causal initialization) leads to motion collapse (Dynamic Degree drops from 0.2706 to 0.0824).
  • Removing Stage 3 (long-horizon self forcing) results in higher WER (9.17%) and increased drift.
  • A context window of 3+1 (3 persistent prefix blocks + 1 recent block) is shown to be optimal for balancing identity preservation and error accumulation.

Significance

The paper claims that Vorch-Streamer establishes a practical route for adapting powerful bidirectional foundation models to real-time, interactive avatar generation. By solving the exposure bias problem through long-horizon self forcing and the speech progression problem through explicit LLM-based planning, the framework enables native Text-to-Audio-Video generation that is both causal and long-form. Unlike conditional pipelines that rely on external audio or reference frames, Vorch-Streamer generates both modalities from scratch, maintaining stability and synchronization over extended durations without requiring memory growth proportional to sequence length.