MoCoTalk: Multi-Conditional Diffusion with Adaptive Router for Controllable Talking Head Generation

Technical Summary: MoCoTalk

Problem Statement
Talking-head generation aims to animate static portraits using driving signals (motion or speech) while preserving source identity and ensuring natural, audio-visually aligned dynamics. Existing approaches face two primary limitations:

1. Limited Control: Most methods address only a subset of critical factors (e.g., audio-only or video-only driving), failing to jointly model identity, head pose, facial expression, and mouth dynamics within a unified framework.
2. Fusion Challenges: When multiple conditions are involved, current methods typically rely on fixed-weight or heuristic fusion strategies. This often leads to destructive interference among heterogeneous signals (e.g., audio, keypoints, 3D meshes), resulting in drifting features, mismatched geometry, or a lack of independent attribute control. Furthermore, parametric models like 3DMMs often underfit fine-grained mouth dynamics, causing temporal jitter and structural mismatches in lip articulation.

Methodology
The authors propose MoCoTalk, a multi-conditional video diffusion framework built upon Stable Video Diffusion (SVD). The system integrates four complementary control signals: a reference image, facial keypoints, 3DMM-rendered shading meshes, and speech audio.

The core methodology consists of four key components:

1. Multi-Conditional Adapters:
   Four parallel adapters lift each conditioning modality into spatially and channel-wise aligned latent feature maps compatible with the shallow layers of the U-Net backbone.

   • Reference Adapter: Extracts identity and appearance.
   • Motion Adapter: Encodes head pose and coarse expression from keypoint maps.
   • Shading Adapter: Captures facial geometry and lighting from 3DMM meshes.
   • Audio Adapter: Unlike prior works that inject audio via cross-attention tokens, this module projects audio features into a spatial latent feature map. It utilizes a sliding window approach to preserve local temporal context, allowing audio to participate in the same feature-level fusion as visual controls.

2. Adaptive Multi-Condition Router:
   To resolve destructive interference, the authors introduce a lightweight gating module positioned after the U-Net's input convolution. Instead of assigning fixed scalar weights, this router computes channel-wise, timestep-aware gates. It takes global summaries of the backbone features, timestep embeddings, and all condition streams as input, outputting a softmax-normalized gating vector. This allows the fusion strategy to dynamically shift emphasis based on the feature subspace and the noise level during the denoising trajectory. It also supports robust dropout for missing modalities.

3. Mouth-Augmented Shading Mesh:
   Addressing the limitation of standard 3DMMs (e.g., DECA) in capturing fine-grained lip movements, the authors propose a fusion pipeline. They combine the global identity and lighting priors of DECA with the temporally consistent lip geometry recovered by SPECTRE. This "Mouth-Augmented Mesh" decouples head motion, mouth motion, expression, and lighting, providing a temporally consistent geometric prior that resolves lower-face artifacts and enables flexible recombination of attributes at inference.

4. Lip Consistency Loss:
   To tighten audio-visual alignment, a specialized loss function is introduced. It supervises mouth-region semantics in pixel space by minimizing the cosine distance between features extracted from the generated frames and ground truth frames using a frozen lip-reading encoder.

Key Contributions

• Unified Framework: A novel multi-conditional diffusion framework that integrates four heterogeneous signals (audio, keypoints, 3D mesh, reference image) for talking-head generation.
• Adaptive Router: An adaptive gating mechanism that dynamically weights and combines control signals in the latent space, preventing destructive interference and enabling attribute-level controllability.
• Mouth-Augmented Mesh: A representation that decouples and recombines motion sources to render 3DMM meshes tightly aligned with speaking dynamics, offering a temporally consistent geometric prior.
• Lip Consistency Loss: A pixel-space supervision method that improves speech-related lip dynamics and audio-visual synchronization.

Experimental Results
Extensive experiments were conducted on the HDTF dataset under both self-reenactment and cross-reenactment settings, comparing MoCoTalk against state-of-the-art methods (VideoReTalking, Diff2Lip, SadTalker, FLOAT, Hallo2, LCVD).

• Quantitative Performance: MoCoTalk achieved state-of-the-art performance on the majority of metrics, including structural (SSIM, PSNR), motion (AED, APD, AKD), and perceptual (FVD, VQA) indicators. Notably, it reduced the FVD score by 37% compared to the second-best method in self-reenactment, indicating superior temporal coherence and distributional similarity to real video.
• Ablation Studies:
  • Removing the Adaptive Multi-Condition Router caused a drastic collapse in performance (e.g., FVD increased from 40.75 to 592.5), confirming that adaptive gating is essential to prevent destructive interference.
  • Removing the Mouth-Augmented Mesh led to significant degradation in motion fidelity and identity preservation, particularly in cross-reenactment settings, highlighting the importance of the temporally consistent geometric prior.
  • Removing the Lip Consistency Loss resulted in a slight drop in synchronization metrics but marginally improved per-frame texture smoothness (VQA), suggesting a trade-off between fine-grained lip dynamics and global texture smoothness.
• Qualitative Analysis: Visual comparisons demonstrate that MoCoTalk preserves reference appearance better than baselines while accurately transferring specified motion. Unlike methods that suffer from frame-level stuttering or neck-boundary discontinuities, MoCoTalk maintains smooth temporal continuity.

Significance and Claims
The paper claims that MoCoTalk addresses the critical gap in jointly modeling diverse control factors for talking-head generation. By introducing an adaptive routing mechanism and a mouth-augmented geometric prior, the framework achieves a balance between high-fidelity reconstruction, motion accuracy, and attribute-level controllability that single-condition methods lack. The authors emphasize that their approach allows for the independent specification of expression, head motion, mouth dynamics, and lighting at inference, enabling customizable animation scenarios previously unattainable with fixed-weight fusion strategies.

Limitations
The authors modestly acknowledge two limitations:

1. There remains room for improvement in identity preservation and lip-sync accuracy compared to models specifically engineered for single objectives.
2. The current backbone generates only 8 frames per forward pass. Longer sequences are created by stitching overlapping segments, which limits scalability and may introduce subtle discontinuities at segment boundaries. Future work aims to migrate to an autoregressive backbone for continuous generation.