💻 computer science

OmniForcing: Unleashing Real-time Joint Audio-Visual Generation

OmniForcing introduces a novel framework that distills high-latency bidirectional audio-visual diffusion models into a real-time streaming autoregressive generator by employing asymmetric block-causal alignment, audio sink tokens, and joint self-forcing distillation to overcome training instability and achieve 25 FPS generation with high fidelity.

Original authors: Yaofeng Su, Yuming Li, Zeyue Xue, Jie Huang, Siming Fu, Haoran Li, Ying Li, Zezhong Qian, Haoyang Huang, Nan Duan

Published 2026-03-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yaofeng Su, Yuming Li, Zeyue Xue, Jie Huang, Siming Fu, Haoran Li, Ying Li, Zezhong Qian, Haoyang Huang, Nan Duan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to conduct a massive orchestra where the violinists (video) play very slowly, taking a long time to finish a note, while the drummers (audio) are playing a frantic, rapid-fire solo.

The Problem:
Currently, the best AI models for creating video and sound together work like a perfectionist conductor who refuses to start the next measure until they have heard the entire song from start to finish. They look at the whole timeline at once to make sure the drums hit exactly when the violin swells. This creates amazing quality, but it's incredibly slow. If you want to generate a 5-second clip, the computer might take 3 minutes to think about it before showing you the first frame. It's like ordering a pizza and having to wait for the chef to bake the whole loaf of bread before slicing off your first piece.

The Solution: OmniForcing
The researchers behind OmniForcing wanted to build a system that could generate video and audio in real-time (like a live stream) without losing that perfect synchronization. They didn't want to build a new orchestra from scratch; they wanted to teach the slow, perfectionist conductor to improvise on the fly.

Here is how they did it, using some simple analogies:

1. The "Macro-Block" Strategy (The Time Zones)

The video and audio move at different speeds. Video is like a slideshow (3 frames per second), while audio is a fast stream (25 frames per second).

The Old Way: Trying to match them frame-by-frame is like trying to zip a jacket where the teeth on one side are huge and the other side is tiny. It just doesn't fit.
The OmniForcing Way: They created "Time Zones" (called Macro-Blocks). Instead of matching every single frame, they group the content into 1-second chunks. In every 1-second chunk, the video has 3 frames and the audio has 25 frames. They treat this 1-second chunk as a single "unit" of time. This allows the AI to jump forward in time smoothly without getting stuck trying to align mismatched pieces.

2. The "Global Prefix" (The Anchor)

When you start a story, you need a beginning. In AI, the very first second of video and audio is special because it sets the tone for everything that follows.

The Analogy: Imagine a ship setting sail. The first few seconds are the "anchor" that holds the ship in place while the engine revs up. OmniForcing creates a special "Global Prefix" block at the very start. This block is visible to the entire future of the video. It acts like a system prompt or a "theme song" that the AI remembers constantly, ensuring that the video and audio stay in sync right from the very first millisecond.

3. The "Audio Sink" (The Shock Absorber)

This is the most clever part. Because the audio is so fast and the video is so slow, when the AI tries to predict the next sound based only on what it has heard so far (without looking ahead), it gets confused.

The Problem: Imagine you are a drummer trying to keep a beat, but you can only see the last 2 seconds of music. If you miss a beat, the whole rhythm collapses into chaos. In math terms, this causes "gradient explosions" (the AI gets angry and crashes).
The Fix: The researchers added "Sink Tokens." Think of these as shock absorbers or cushions placed at the very front of the audio stream. They don't represent real sound; they are just "dummy" tokens that soak up the confusion. They act as a safety net, giving the AI a stable place to rest its attention so it doesn't panic when the data is sparse. They also use a special "Identity RoPE" (a type of position tag) that tells these cushions: "You don't have a time; you are just here to help."

4. The "Self-Correcting" Loop (The Rehearsal)

Usually, when AI generates video, it makes a small mistake, then uses that mistake to make the next frame, leading to a snowball of errors (like a game of "Telephone").

The Fix: OmniForcing uses a technique called Joint Self-Forcing. Imagine a student rehearsing for a play. Instead of just reading the script, the student acts out the whole scene, makes a mistake, and then immediately corrects it while the teacher (the original slow AI) watches. The AI learns to fix its own errors in real-time, ensuring that the video and audio stay perfectly synchronized even over long periods.

The Result

By combining these tricks, OmniForcing transforms a slow, offline model into a real-time streaming engine.

Before: Generating a 5-second clip took 197 seconds (over 3 minutes).
After: It takes 5.7 seconds to generate, and it can stream at 25 frames per second (like a live TV broadcast).

In a nutshell: OmniForcing is like taking a slow, perfectionist chef who needs to cook a whole banquet before serving, and teaching them to be a master street-food vendor who can serve you a hot, delicious, perfectly synchronized meal the moment you order it.

1. Problem Statement

Current state-of-the-art joint audio-visual generation models (e.g., LTX-2, Veo 3) rely on bidirectional full-sequence attention within Diffusion Transformers (DiTs). While these models achieve high-fidelity synchronization and quality, they suffer from critical limitations for real-world deployment:

High Latency: They require processing the entire physical timeline simultaneously, resulting in a massive Time-To-First-Chunk (TTFC) (e.g., ~197 seconds for a 5-second clip), making them unsuitable for interactive or streaming applications.
Decoupled Alternatives Fail: Existing workarounds either use cascaded pipelines (generating video then audio, or vice versa), which breaks the joint temporal distribution and prevents true streaming, or adapt video-only causal models, which fail when extended to dual-stream architectures due to extreme temporal asymmetry between modalities.
Training Instability: Naively applying causal distillation to dual-stream models causes Softmax collapse and gradient explosions. This is due to the severe token sparsity in the audio stream (25 tokens/sec) compared to the video stream (hundreds of tokens/sec), leading to an information density gap that destabilizes the attention mechanism.

2. Methodology: OmniForcing

OmniForcing is a framework that distills a heavy, offline, bidirectional teacher model (LTX-2) into a high-fidelity, streaming autoregressive generator. The approach involves a three-stage distillation pipeline and specific architectural innovations to handle modality asymmetry.

A. Asymmetric Block-Causal Alignment

To bridge the frequency gap between video (3 FPS latent) and audio (25 FPS latent), the authors propose a Macro-block Alignment strategy:

Global Prefix ( $B_0$ ): The initial latent frames ( $V_0, A_0$ ) are anchored at $t=0$ as a zero-truncation Global Prefix. This block is unconditionally bidirectional, acting as a global semantic anchor (similar to a system prompt in LLMs) visible to all future tokens.
1-Second Macro-Blocks: Subsequent generation is divided into 1-second blocks. Each block contains exactly 3 video latent frames and 25 audio latent frames. This ensures integer alignment without fractional remainders.
Four-Way Masking: A custom attention mask is applied:
- Intra-block: Bidirectional (tokens within the same 1-second block can see each other).
- Inter-block: Strictly causal (tokens can only see previous blocks).
- Cross-modal: Synchronized via physical time boundaries.

B. Architectural Stabilizers (Solving Instability)

To prevent the training collapse caused by sparse audio tokens in early causal blocks, OmniForcing introduces:

Audio Sink Tokens: Learnable tokens prepended to the audio sequence and anchored in the Global Prefix. They act as a "soft global memory buffer," artificially expanding the attention denominator for early audio tokens to prevent entropy collapse.
Identity RoPE Constraint: Standard Rotary Position Embeddings (RoPE) are disabled for Sink Tokens ($RoPE(x) = x$). This prevents spurious temporal biases from being applied to these abstract memory tokens, ensuring they remain position-agnostic anchors.

C. Three-Stage Distillation Pipeline

Stage I (Bidirectional DMD): Uses Distribution Matching Distillation (DMD) to compress the teacher's multi-step sampling into a few-step bidirectional student model.
Stage II (Causal ODE Regression): Adapts the model weights to the new causal masks by regressing the ODE trajectories of the Stage I teacher. This stage specifically utilizes the Audio Sink Tokens to stabilize the transition.
Stage III (Joint Self-Forcing): Addresses exposure bias (error accumulation during long rollouts). The model is trained to unroll its own autoregressive predictions (using its own KV-cache) and corrects them against the teacher's distribution. This forces the video and audio streams to dynamically self-correct cross-modal drifts.

D. Inference Optimization

Modality-Independent Rolling KV-Cache: Exploits the decoupling between video and audio self-attention layers. The KV-cache is updated independently for each modality, reducing per-step complexity to $O(L)$ .
Parallel Execution: Enables concurrent execution of video and audio streams on a single GPU, achieving true real-time streaming.

3. Key Contributions

First Real-Time Joint AV Generator: OmniForcing is the first framework to successfully distill a bidirectional joint audio-visual foundation model into a streaming autoregressive engine while preserving exact multi-modal synchronization.
Stabilization Mechanisms: Introduces Asymmetric Block-Causal Alignment and Audio Sink Tokens with Identity RoPE, providing a robust solution to the Softmax collapse and gradient explosion caused by multi-modal token density mismatches.
Efficient Inference Architecture: Proposes a Modality-Independent Rolling KV-Cache and Joint Self-Forcing Distillation, enabling state-of-the-art streaming generation at ~25 FPS on a single GPU with a TTFC of ~0.7s.

4. Experimental Results

Evaluated on JavisBench and VBench, OmniForcing demonstrates:

Latency Reduction: Achieves a ~35x speedup over the bidirectional teacher (LTX-2).
- Teacher: 197s TTFC.
- OmniForcing: 0.7s TTFC, ~25 FPS sustained throughput.
Quality Preservation:
- Visual/Audio Fidelity: Achieves an FVD of 137.2 and FAD of 5.7, nearly matching the teacher (125.4 / 4.6) and significantly outperforming all cascaded baselines.
- Text Consistency: Achieves the highest CLIP score (0.322) among all methods.
- Synchronization: Maintains strong AV-Synchrony (JavisScore 0.208, DeSync 0.392), closely tracking the teacher and far surpassing decoupled methods.
Ablation Studies: Confirmed that without Sink Tokens (or with insufficient size), training fails due to NaN gradients (Softmax collapse). Replacing Identity RoPE with standard RoPE leads to noisy outputs and higher loss.

5. Significance

OmniForcing breaks the latency barrier for joint audio-visual generation, making high-fidelity, synchronized multimodal generation feasible for interactive, real-time, and streaming applications (e.g., live virtual avatars, real-time video games, and interactive media). It solves the fundamental technical challenge of distilling bidirectional multimodal models into causal architectures without sacrificing synchronization or stability, paving the way for the deployment of large-scale foundation models in latency-sensitive scenarios.