Robust Audio-Visual Target Speaker Extraction with Emotion-Aware Multiple Enrollment Fusion

This paper proposes a robust Audio-Visual Target Speaker Extraction framework that leverages emotion-aware multiple enrollment fusion, demonstrating that training with high modality missing rates significantly enhances performance stability against real-world signal loss while achieving optimal results by fusing single-frame facial images with frame-level lip features.

The Gist

Imagine you are at a loud, chaotic party (the "cocktail party"). You want to hear your friend, Alice, clearly, but there are dozens of other people talking, music playing, and noise everywhere. Your brain is amazing at this; it can focus on Alice's voice and tune out the rest. This paper is about teaching a computer to do the exact same thing.

The task is called Audio-Visual Target Speaker Extraction. In simple terms, it's a computer program that listens to a noisy recording and tries to isolate just one person's voice.

Here is the breakdown of how this paper solves the problem, using some everyday analogies:

1. The Problem: "Blind" Computers

To help the computer find Alice, we usually give it a "hint" or a "clue" about what she looks or sounds like.

• The Audio Clue: A recording of Alice's voice.
• The Visual Clues:
  • Lip Movements: Watching her lips move (very precise, like reading lips).
  • Face: A photo of her face (tells us who she is).
  • Expressions: Her facial emotions (tells us how she feels).

The Catch: In the real world, things go wrong. Alice might turn her head, someone might walk in front of the camera, or the video might glitch. If the computer relies only on watching her lips, and the video freezes or gets blocked, the computer goes "blind" and fails to find her voice.

2. The Solution: The "Swiss Army Knife" Approach

The authors realized that relying on just one type of clue is risky. Instead, they built a system that uses multiple clues at once, like a Swiss Army knife. If one tool is broken, the others can still do the job.

They tested four types of clues:

1. Lip Reading: Watching the mouth (Frame-level).
2. Face ID: A steady photo of the face (Utterance-level).
3. Emotions: Watching facial expressions (Frame-level).
4. Voiceprint: A sample of her voice (Utterance-level).

3. The Secret Sauce: "Training in the Rain"

This is the most important part of the paper.

• The Old Way: Imagine training a soccer player only on a perfect, sunny day on a pristine field. When they finally play a match in the pouring rain with mud everywhere, they slip and fall because they never practiced in bad conditions.

  • In the paper: If you train the computer on perfect videos where the face is always visible, it fails miserably when the video gets blocked.

• The New Way (This Paper): The authors decided to train the computer in the rain. They deliberately covered up (occluded) 80% of the video frames during training. They forced the computer to learn how to find Alice even when she was mostly hidden.

  • The Result: When they tested the computer on real-world messy videos, it didn't panic. It knew how to use the little bit of lip movement it could see, combined with the photo of her face, to figure out who was speaking.

4. The Best Combination: "The Photo + The Lip"

The researchers found that the perfect team wasn't necessarily all the clues. The winning combination was:

• One steady photo of the face (to know who it is).
• The moving lips (to know what they are saying).

They found that adding "emotions" (expressions) didn't help much, but adding the photo was a game-changer. It acted as a safety net. If the lip video got blocked, the computer could still remember, "Oh, that's Alice's face," and keep the voice clear.

5. The Takeaway

This paper teaches us that to build a robust AI for the real world, you can't just train it on perfect data. You have to simulate disasters during training.

By teaching the computer to handle missing information (like a blocked camera), they created a system that is:

1. Strong: It works great when everything is perfect.
2. Resilient: It keeps working even when the video is glitchy, blocked, or incomplete.

In a nutshell: They built a "super listener" that doesn't just listen; it watches, but it's smart enough to keep listening even when it can't see perfectly, because it was trained to expect the unexpected.

Technical Summary

Here is a detailed technical summary of the paper "Robust Audio-Visual Target Speaker Extraction with Multiple Enrollment Fusion":

1. Problem Statement

Audio-Visual Target Speaker Extraction (AVTSE) aims to isolate a specific speaker's voice from a noisy mixture (the "cocktail party effect") using prior information about the target. While leveraging multiple cues (utterance-level features like face images or voice, and frame-level features like lip motion) improves performance, real-world applications face a critical challenge: intermittent signal loss.

• The Issue: Frame-level visual cues (e.g., lip movements) are highly susceptible to occlusion, head movement, or signal interruption.
• The Gap: Existing models often degrade sharply when tested on unseen modalities or missing data, especially if they were trained only on pristine, complete data. Current research lacks a systematic investigation into how to fuse multiple enrollment cues (voice, face, lip, expression) to maintain robustness under varying degrees of modality missing.

2. Methodology

The authors propose a novel AVTSE system that integrates Multiple Enrollment Fusion and a High-Missing-Rate Training Strategy.

A. System Architecture

The model consists of a speech encoder, multiple target speaker encoders, a modal fusion block, separator blocks, and a decoder (see Figure 1 in the paper).

• Input: Noisy mixed speech and various enrollment cues.
• Encoders:
  • Utterance-Level:
    • Enrollment Speech: Uses a Universal Speaker Embedding Free (USEF) block with cross-attention to extract voice embeddings.
    • Face Image: Uses InceptionResNetV1 to extract static face embeddings (duplicated across the time dimension).
  • Frame-Level:
    • Lip Motion: Uses a pre-trained ResNet-18 (lip-reading model) to extract dynamic lip embeddings.
    • Facial Expression: Uses ResEmoteNet to extract emotion embeddings, hypothesizing that paralinguistic cues provide context beyond identity.
• Fusion & Processing:
  • Visual embeddings pass through a Visual Temporal Convolutional Network (V-TCN) attractor to project them onto a unified feature plane.
  • All embeddings are channel-wise concatenated with the complex spectrogram of the mixed speech.
  • The Separator uses six TFGridBlock modules (containing intra-frame, sub-band, and cross-frame attention mechanisms) to estimate the target spectrogram.
  • The Decoder uses a 2D transposed convolution and iSTFT to reconstruct the waveform.

B. Training Strategy: Modality Occlusion

To address robustness, the authors introduce a specific training regime:

• Occlusion Function: A custom function zeros out consecutive frames (15–25 frames) in the visual and audio cues to simulate missing data.
• Training Conditions: Models are trained under two regimes:
  1. 0% Occlusion: Standard training on pristine data.
  2. 80% Occlusion: Training with severe, random frame missingness to force the model to rely on available cues rather than overfitting to complete data.
• Testing: Evaluated under 0%, 40%, and 80% occlusion to test generalization.

3. Key Contributions

1. Systematic Analysis of Multi-Modal Fusion: The paper investigates four distinct speaker cues (Lip, Face, Expression, Enrollment Speech) and their correlations. It identifies that while full fusion is optimal under ideal conditions, specific combinations offer better robustness.
2. High-Missing-Rate Training: The authors demonstrate that training with a high rate of modality missingness (80%) fundamentally changes model behavior, preventing over-reliance on specific cues and ensuring stable performance even when test-time data is severely degraded.
3. Optimal Fusion Strategy (Lip + Face): The study identifies that fusing one frame of face image (utterance-level) with frame-level lip features achieves the best trade-off between high performance and robustness, outperforming systems that rely solely on lip movements or complex multi-modal stacks in missing-data scenarios.
4. Open Source: The model and code are shared to facilitate further research.

4. Experimental Results

Experiments were conducted on the AVSEC-3 dataset (Audio-Visual Speech Enhancement Challenge).

• Ideal Conditions (0% Occlusion):

  • Full fusion (Lip + Face + Expression + Speech) achieved the highest performance (SISDR: 13.137).
  • Face embeddings provided significant gains over lip-only models.
  • Expression embeddings showed limited added value compared to static face images in this framework.

• Robustness under Occlusion (The Core Finding):

  • Standard Training (0% occlusion during training): Performance degraded sharply as test-time occlusion increased. For example, a Lip-only model dropped from SISDR 12.22 (0% test) to 3.04 (80% test).
  • Robust Training (80% occlusion during training): Models maintained stable and high performance across all test conditions.
    • Example: The "Lip-Face" model trained with 80% occlusion maintained an SISDR of 12.298 even with 80% test-time occlusion (compared to 5.031 for the same model trained without occlusion).
  • Conclusion: High-occlusion training forces the model to effectively utilize complementary cues (like the static face) when dynamic cues (lips) are missing.

• Comparison with State-of-the-Art (SOTA):

  • On the AVSEC-3 Leaderboard, the proposed Lip-Face system achieved competitive results (SISDR: 13.259, PESQ: 3.087), outperforming or matching top systems like Smiip try2 and ict avsu 1 in key metrics.
  • Unlike other SOTA systems that excel only in ideal conditions, this model demonstrated superior reliability in scenarios with intermittent missing cues.

5. Significance

This paper shifts the paradigm in AVTSE from simply maximizing performance under ideal conditions to engineering robustness for real-world deployment.

• Practical Impact: It proves that by intentionally training with missing data, models can become resilient to the inevitable signal losses (occlusions, network drops) found in real-world "cocktail party" scenarios.
• Efficiency: It suggests that a simple fusion of static face images and dynamic lip movements is a highly efficient and robust strategy, potentially reducing the computational cost of processing complex expression or full-video streams while maintaining high accuracy.
• Generalizability: The findings on training with missing data are likely applicable to other multi-modal tasks where sensor reliability varies.