SLD-L2S: Hierarchical Subspace Latent Diffusion for High-Fidelity Lip to Speech Synthesis

1. Problem Statement

Lip-to-Speech (L2S) Synthesis aims to generate audible speech solely from visual lip movements. Despite recent progress, existing State-of-the-Art (SOTA) methods face significant limitations:

• Intermediate Representations: Most methods rely on generating intermediate representations like mel-spectrograms or discrete Self-Supervised Learning (SSL) tokens. These often fail to capture the fine-grained acoustic details necessary for high-fidelity waveform generation.
• Information Loss: The mapping from visual cues to audio is an ill-posed, one-to-many problem. Discrete tokens (semantic) lack acoustic nuance, while mel-spectrograms lack the flexibility to model diverse prosody and emotion.
• Inefficiency of End-to-End Approaches: Direct end-to-end training often requires massive datasets to retrain vocoders and still yields limited quality.
• Token Prediction Challenges: Recent attempts to use audio codecs (e.g., predicting discrete residual tokens) struggle because the sparse visual input cannot sufficiently support complex language modeling tasks required for categorical token prediction.

Core Question: Can we identify a representation that captures detailed acoustic characteristics for high-fidelity L2S without the information loss of intermediate steps or the complexity of discrete token prediction?

2. Methodology: SLD-L2S

The authors propose SLD-L2S, a framework that directly maps visual lip movements to the continuous latent space of a pre-trained neural audio codec (X-Codec) using a Hierarchical Subspace Latent Diffusion Model.

A. Architecture Overview

The framework consists of three main stages:

1. Visual Frontend:
   • Uses a pre-trained AV-Hubert Large model to extract visual features rich in phonetic content from input video frames.
   • Features are temporally upsampled to match the resolution of the target audio codec's latent space.
2. Hierarchical Subspace Decomposition:
   • Instead of a monolithic mapping, the upsampled visual features are decomposed into multiple parallel subspaces (e.g., 8 subspaces).
   • Each subspace processes features through layer normalization and 1D convolution to learn distinct representations.
3. Diffusion Backbone (DiCB):
   • The core network is built using Diffusion Convolution Blocks (DiCB) rather than standard Diffusion Transformers (DiT).
   • DiCB Design: Replaces self-attention with specialized convolutional modules (depthwise convolution with kernel size $t \times k$) to capture local temporal and cross-subspace dependencies efficiently.
   • Conditioning: Uses AdaLN-SOLA (Adaptive Layer Normalization with Shared Low-rank Adjustment) to inject time steps and speaker embeddings, reducing computational overhead while stabilizing training.
4. Subspace Recomposition:
   • Processes the refined features from the DiCB backbone.
   • Uses 1D convolution, reshaping, and ConvNeXt blocks to fuse the parallel subspace representations into a single unified latent vector matching the codec's output dimension.

B. Training Objectives

The model employs a Reparameterized Flow Matching technique combined with auxiliary losses:

1. Reparameterized Flow Matching Loss ($L_{FM}$):
   • Instead of predicting the velocity field $v_\theta$ (which can be unstable), the model is reparameterized to directly predict the target data point $x_1$ (the latent vector).
   • This formulation improves training stability and allows for the direct application of auxiliary losses in the data space.
2. Semantic Consistency Loss ($L_{sem}$):
   • Enforces that the generated latent vectors retain the same core semantic content as the target.
   • Uses a pre-trained semantic decoder ($D_s$) to reconstruct semantic features (from HuBERT/WavLM) from the latents and minimizes the L2 distance between predicted and target reconstructions.
3. Speech Language Model Loss ($L_{SLM}$):
   • Applied on the final synthesized waveform.
   • Decodes the predicted latent to a waveform, extracts features using a pre-trained SLM, and minimizes the distance between the SLM features of the generated and target speech. This ensures high-level perceptual consistency.

Total Loss: $L = L_{FM} + \lambda_1 L_{SLM} + \lambda_2 L_{sem}$

3. Key Contributions

1. Novel Framework: Proposes the first L2S framework to directly generate continuous latent vectors of a pre-trained audio codec from visual inputs, bypassing intermediate mel-spectrograms or discrete tokens.
2. Hierarchical Subspace Architecture: Introduces a subspace decomposition and recomposition mechanism that allows for robust cross-modal mapping by processing visual features in parallel pathways.
3. Diffusion Convolution Block (DiCB): Designs a new backbone leveraging convolutional inductive biases to capture localized and hierarchical dependencies more effectively than Transformer-based DiT architectures for this specific task.
4. Reparameterized Flow Matching with Auxiliary Losses: Develops a stable training objective that enables the principled inclusion of semantic and SLM losses, significantly boosting synthesis quality and intelligibility.

4. Experimental Results

The method was evaluated on LRS3-TED and LRS2-BBC datasets against SOTA methods (DiffV2S, LipVoicer, V2SFlow, etc.).

• Objective Metrics:
  • Quality: SLD-L2S achieved SOTA scores in non-intrusive quality metrics (UTMOS: 4.21, SCOREQ: 4.61 on LRS3), significantly outperforming V2SFlow (UTMOS 3.69).
  • Intelligibility: Achieved competitive Word Error Rates (WER) and high D-BERT scores, balancing intelligibility with high fidelity.
  • Speaker Similarity: Achieved the second-best Speaker Embedding Cosine Similarity (SECS), demonstrating effective speaker conditioning without explicit speaker loss.
• Subjective Metrics (MOS):
  • Naturalness: Scored 4.17/5, nearly matching ground truth and surpassing all competitors.
  • Efficiency: Requires only 10 Number of Function Evaluations (NFE) at inference, making it significantly faster than other diffusion-based methods.
• Ablation Studies:
  • Replacing DiCB with DiT caused a significant performance drop, validating the convolutional design.
  • Removing the reparameterized objective degraded quality.
  • The 8-subspace configuration was found to be optimal; fewer or more subspaces reduced performance.

5. Significance

• Paradigm Shift: SLD-L2S demonstrates that continuous latent diffusion is superior to discrete token prediction or intermediate spectrogram generation for L2S tasks, effectively solving the "information loss" bottleneck.
• High-Fidelity Synthesis: It sets a new benchmark for perceptual quality, producing speech that is not only intelligible but also natural and emotionally expressive.
• Practical Applicability: The high efficiency (low NFE) and robustness on unseen datasets (LRS2) make it a viable solution for real-world applications like automated video dubbing, assistive technologies, and robust communication in noisy environments.
• Methodological Innovation: The introduction of DiCB and the reparameterized flow matching strategy with semantic/SLM losses provides a robust blueprint for future high-fidelity audio generation tasks involving cross-modal inputs.