Hierarchical Codec Diffusion for Video-to-Speech Generation

1. Problem Statement

Video-to-Speech (VTS) generation aims to synthesize natural, lip-synchronized speech from silent videos without any auditory input. Despite its potential for applications like silent film dubbing and assistive communication, existing VTS methods face a fundamental bottleneck: information asymmetry between visual and acoustic modalities.

• The Core Issue: Current approaches typically treat speech as a flat, holistic sequence. They inject visual features (lip motion, identity, emotion) into a single generative representation, ignoring the inherent hierarchical nature of speech.
• The Consequence: Speech spans from coarse, speaker-aware semantics (content/timbre) to fine-grained prosodic details (emotion/rhythm). By failing to align specific visual cues with the corresponding hierarchical level of speech, existing models struggle to achieve precise cross-modal alignment, resulting in speech that lacks naturalness, expressiveness, or accurate lip-sync.

2. Methodology: HiCoDiT

The authors propose HiCoDiT (Hierarchical Codec Diffusion Transformer), the first discrete diffusion framework for VTS that explicitly integrates a speech hierarchy prior.

A. Core Architecture

HiCoDiT leverages a Residual Vector Quantization (RVQ) codec to tokenize speech into 12 discrete levels. The model decomposes these tokens into two distinct components based on their semantic role:

1. Low-Level Tokens ($x_{low}$): Levels 1–2. These encode coarse speaker-aware semantics (linguistic content and speaker identity/timbre).
2. High-Level Tokens ($x_{high}$): Levels 3–12. These encode fine-grained prosodic details (emotional dynamics and rhythm).

B. Disentangled Visual Conditioning

To bridge the gap between video and speech, HiCoDiT employs a disentangled conditioning strategy, mapping specific visual features to the appropriate token levels:

• Lip Motion & Identity $\rightarrow$ Low-Level Blocks:
  • Lip Motion: Extracted via AV-HuBERT, these features are concatenated with low-level tokens to ensure precise content and lip-synchronization.
  • Speaker Identity: Extracted via ArcFace, these features are used to modulate low-level tokens to preserve timbre.
• Facial Emotion $\rightarrow$ High-Level Blocks:
  • Emotion: Extracted via Poster2, these features guide the high-level blocks to modulate prosody and expressiveness.

C. Dual-Scale Adaptive Instance Layer Normalization (AdaLN)

A key innovation in the high-level blocks is the Dual-Scale AdaLN. Standard normalization often fails to capture both global style and local dynamics simultaneously. HiCoDiT introduces:

• Channel-wise Normalization: Models the global vocal style (speaker identity).
• Temporal-wise Normalization: Captures local prosody dynamics (emotional fluctuations).
  This mechanism allows the model to inject global style and local emotional variations into the generation process simultaneously.

D. Training and Inference

• Model: A Masked Discrete Diffusion Model (DDM). The forward process masks tokens, and the reverse process (score network) predicts concrete scores to unmask them.
• Loss Function: Uses Denoising Score Entropy (DSE) across all 12 RVQ levels, plus an auxiliary $L_1$ loss to align visual identity embeddings with acoustic identity embeddings (GE2E).
• Inference: Uses Euler sampling with predictor-free guidance to generate high-fidelity speech from masked tokens.

3. Key Contributions

1. First Discrete Diffusion for VTS with Hierarchy: HiCoDiT is the first work to explicitly integrate speech hierarchy priors into a discrete diffusion framework, moving beyond flat sequence modeling.
2. Disentangled Hierarchical Conditioning: It proposes a novel transformer architecture that separates visual conditioning based on the speech token hierarchy (Lip/ID $\to$ Semantics; Emotion $\to$ Prosody), effectively addressing the information asymmetry problem.
3. Dual-Scale AdaLN: Introduces a normalization layer that jointly captures global vocal style and local prosodic dynamics, significantly enhancing speech expressiveness.
4. State-of-the-Art Performance: Demonstrates superior performance in naturalness, synchronization, and expressiveness compared to existing continuous and discrete baselines.

4. Experimental Results

The model was trained on VoxCeleb2 and evaluated on LRS2 and LRS3 (in-the-wild datasets) without specific training on them.

• Objective Metrics:
  • Naturalness & Quality: Achieved the highest scores in UTMOS (3.50) and DNSMOS (3.84) on LRS3, outperforming baselines like FTV and AlignDiT.
  • Synchronization: Achieved the best Lip Sync Error (LSE-C) score (7.15), indicating superior alignment with lip movements.
  • Intelligibility: Low Word Error Rate (WER) of 29.41 on LRS3.
  • Expressiveness: High Emotion Accuracy (79.41%).
• Subjective Evaluation (MOS):
  • HiCoDiT received the highest Mean Opinion Scores for Naturalness (3.17) and Synchronization (3.50), surpassing Ground Truth in synchronization metrics in user studies.
  • In A/B testing, users preferred HiCoDiT over AlignDiT (57.0%) and FTV (52.1%).
• Qualitative Analysis: Spectrograms generated by HiCoDiT show clearer acoustic details and less noise compared to continuous diffusion models or GAN-based approaches.
• Ablation Studies:
  • Removing hierarchical modeling caused significant performance drops across all metrics, validating the necessity of the hierarchy prior.
  • Removing the Dual-Scale AdaLN reduced prosody modeling capabilities (lower EmoAcc).

5. Significance

HiCoDiT represents a paradigm shift in Video-to-Speech generation. By recognizing that speech is not a flat sequence but a hierarchical structure, the paper demonstrates that aligning visual cues to specific levels of this hierarchy yields significantly better results.

• Theoretical Impact: It establishes that discrete token modeling with hierarchical priors is a viable and superior alternative to continuous mel-spectrogram generation for VTS.
• Practical Impact: The model's ability to generate high-fidelity, expressive, and lip-synced speech from silent video makes it a strong candidate for real-world applications such as dubbing, accessibility tools for the voiceless, and interactive avatars in privacy-sensitive environments.

The code and demos are available at the provided GitHub repository, marking a significant step forward in multimodal generative AI.