VIB-AVSR: Variational Information Bottleneck for Noise-Robust LLM-Based Audio-Visual Speech Recognition

Technical Summary: VIB-AVSR

Problem Statement
Audio-Visual Speech Recognition (AVSR) leverages visual cues (lip movements) to complement acoustic signals, particularly in noisy environments. While recent Large Language Model (LLM)-based AVSR paradigms have achieved state-of-the-art performance by connecting pre-trained audio-visual encoders to an LLM backbone, they exhibit significant limitations regarding noise robustness. Unlike traditional end-to-end models trained from scratch, LLM-based AVSR models utilize a backbone pre-trained exclusively on text. Consequently, the LLM parameters are never exposed to noisy audio-visual speech, and only a small fraction of parameters (via LoRA) are updated during fine-tuning. This architecture lacks an explicit mechanism to generate stable representations when input audio is corrupted, causing performance to degrade significantly in noisy conditions. Existing approaches often rely on data augmentation or encoder-only modifications, leaving the LLM backbone itself ill-equipped to handle the domain shift induced by acoustic noise.

Methodology: VIB-AVSR
To address this, the authors propose VIB-AVSR, a framework that integrates Variational Information Bottleneck (VIB) layers directly into the LLM backbone. The core objective is to regularize the internal representations of the model to discard noise-induced variance while retaining task-relevant information.

• Architecture: The system builds upon the Llama-AVSR framework, which uses pre-trained Whisper-medium (audio) and AV-HuBERT (video) encoders, projecting their outputs into a Llama-3.2-1B backbone. VIB modules are inserted at targeted transformer layers within the LLM.
• The VIB Objective: The method applies the Information Bottleneck principle specifically to the audio hidden states ($H_a$). The objective is to maximize the mutual information between the compressed representation ($Z$) and the transcription target ($Y$) while minimizing the mutual information between $Z$ and the noisy input ($H_a$).
  • Selective Application: The bottleneck is applied only to audio tokens. Visual ($H_v$) and text ($H_t$) tokens are preserved unchanged, as they are not corrupted by acoustic noise and provide essential complementary context.
  • Variational Formulation: Since mutual information is intractable to estimate directly, the authors use a variational lower bound. The posterior distribution $p(Z|H_a)$ is parameterized by a two-layer MLP (predicting mean and variance), and a learnable diagonal Gaussian serves as the prior.
  • Loss Function: The training objective combines the standard autoregressive transcription likelihood with a KL-divergence regularization term weighted by a hyperparameter $\beta$.
• Implementation Details:
  • Interpolation: To prevent the sampled representation from being too lossy for the LLM, the bottleneck output ($\tilde{Z}$) is interpolated with the pre-bottleneck representation ($H_a$) using a coefficient $\alpha$ (set to 0.5): $\hat{Z} = \alpha H_a + (1-\alpha)\tilde{Z}$.
  • Inference: During inference, the reparameterization trick is bypassed, and only the mean of the posterior is used.
  • Training: The LLM and video encoder are fine-tuned using LoRA, while the audio encoder remains frozen. VIB modules are trained jointly with the LoRA adapters.

Key Contributions

1. Novel Framework: Introduction of VIB-AVSR, a lightweight method that inserts VIB layers into the LLM backbone to improve noise robustness at the representation level without requiring architectural redesign or additional training data.
2. Generalization: Demonstration that variational compression promotes generalization to noisy conditions even when the model is trained on clean data, suggesting the mechanism is distinct from simple data augmentation.
3. Empirical Analysis: Extensive ablation studies identifying the optimal configuration: inserting VIB layers at transformer layers 4 and 8 with a regularization strength $\beta = 0.1/H$ and an interpolation coefficient $\alpha = 0.5$.
4. Performance Gains: Achievement of Word Error Rate (WER) reductions over the Llama-AVSR baseline across multiple noise types (babble, speech) and Signal-to-Noise Ratio (SNR) levels, with gains widening under extreme noise conditions.

Experimental Results
The authors evaluated VIB-AVSR on the LRS2 dataset under both "Noisy" (noise added during training) and "Clean" (no noise during training) paradigms.

• Noisy Training: VIB-AVSR consistently reduced WER compared to the baseline, with the most significant improvements observed at lower SNRs (e.g., -10 dB) and under babble noise. Notably, it also outperformed the baseline in noise-free evaluation ($\infty$), indicating effective regularization against nuisance variability.
• Clean Training: Even without noise augmentation during training, VIB-AVSR yielded substantial WER reductions when tested on noisy data. This suggests the bottleneck forces the model to learn representations that are inherently robust to unseen noise.
• Ablation Findings:
  • Placement: A dual configuration at layers 4 and 8 proved superior to single or triple bottleneck setups. Placing bottlenecks too early (at the encoder output) or too late was less effective.
  • Hyperparameters: $\beta = 0.1$ offered the best trade-off; higher values ($\beta=1$) caused excessive compression and performance degradation. An interpolation coefficient of $\alpha=0.5$ was critical, as setting $\alpha=0$ (full replacement) led to the worst performance.

Significance and Claims
The paper claims that VIB-AVSR offers a principled approach to noise robustness by directly regularizing the LLM's internal representations, addressing a vulnerability that prior work has largely overlooked. The method is significant because it achieves robustness without architectural overhaul or the need for massive noisy datasets, making it a practical addition to existing LLM-based AVSR pipelines. The results suggest that variational compression can act as a powerful regularizer, enabling models trained on clean data to generalize effectively to noisy environments, thereby narrowing the performance gap between clean and noisy conditions.