Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling

Technical Summary: Talker-T2AV

Talker-T2AV is a novel autoregressive diffusion framework designed for the joint generation of synchronized talking-head audio and video from text. It addresses the structural inefficiencies of existing models by decoupling high-level semantic planning from low-level signal rendering.

1. The Problem: Limitations of Existing Models

The authors identify two primary flaws in current state-of-the-art "dual-branch diffusion transformer" (dual-DiT) models:

• Suboptimal Coupling (Entanglement): Existing models use pervasive cross-modal attention at every denoising step. This forces high-level semantic modeling (e.g., "what is being said") to be entangled with low-level signal rendering (e.g., "acoustic textures" or "visual pixels"). Since these two processes follow fundamentally different physical laws, continuous coupling leads to modeling inefficiency.
• Fixed-Length Constraints: Most diffusion-based models are non-autoregressive and require a predetermined output duration. This makes it impossible to adapt to varying speaking rates or long text inputs without truncating content or compressing speech, which degrades intelligibility.

2. Methodology: A Hierarchical, Decoupled Approach

Talker-T2AV adopts a "coarse-to-fine" architecture that factorizes the generation process into two distinct stages:

Stage 1: High-Level Cross-Modal Modeling (The Planner)

• Shared Autoregressive Backbone: A causal language model (initialized from Qwen3) acts as a temporal planner. It processes a unified sequence consisting of a text prefix followed by joint audio-video "patch tokens."
• Element-wise Summation Fusion: Instead of interleaving tokens (which increases sequence length) or using complex cross-attention, the model encodes audio and video into temporally aligned latent sequences (at 25 Hz) and sums their embeddings at each position. This creates a single, unified joint token that represents both modalities simultaneously.
• Patch Transformer Encoder: To handle the computational complexity of long sequences, multiple frames are compressed into a single "patch token," significantly accelerating inference.
• Stop Predictor: An MLP predicts when the sequence should end, allowing the model to generate variable-length outputs that naturally match the input text.

Stage 2: Low-Level Modality-Specific Refinement (The Renderers)

• Independent Diffusion Heads: The shared hidden states from the backbone are passed to two separate, lightweight Diffusion Transformer (DiT) heads—one for audio and one for video.
• Specialized Rendering: These heads use Optimal-Transport Conditional Flow Matching (OT-CFM) to denoise the specific latents of their respective modalities. This allows the audio head to focus on acoustic waveforms and the video head to focus on facial motion without interference.

Multi-Task Training:
The model is trained end-to-end using a mixture of Text-to-Audio-Video (T2AV) data and large-scale Text-to-Speech (TTS) data. This allows the backbone to learn robust linguistic patterns from the massive TTS datasets, which in turn improves the accuracy of the lip-syncing in the video branch.

3. Key Contributions

1. Decoupled Architecture: Proposes a framework that separates high-level semantic coordination from low-level signal rendering.
2. Unified Multi-Task Capability: Due to the element-wise summation design, a single model can perform three distinct tasks without any architectural changes or fine-tuning:
   • Text-to-Audio-Video (Joint generation).
   • Audio-to-Video (Talking head synthesis).
   • Video-to-Audio (Video dubbing).
3. Variable-Length Generation: Introduces an autoregressive mechanism that supports natural, context-dependent speech duration.

4. Results and Significance

• Superior Performance: In joint T2AV tasks, Talker-T2AV outperforms dual-branch DiT baselines (like MoVA and Ovi) in speech intelligibility (lower WER/CER), video fidelity (lower FVD), and lip-sync accuracy (higher SyncNet Confidence).
• Competitive Conditional Generation: Even though it is primarily a joint model, it matches or surpasses dedicated, single-purpose systems in audio-driven synthesis and video dubbing.
• Significance: This work shifts the paradigm for talking-head generation from "continuous entanglement" to "semantic coordination followed by specialized rendering." It demonstrates that a unified, autoregressive approach is more efficient, more flexible, and more coherent than traditional cascaded or fully-coupled diffusion pipelines.