Technical Summary: PS4 – Proxy-Supervised Joint Training for Real Target Speaker Extraction

Problem Statement

Target Speaker Extraction (TSE) aims to isolate a specific speaker's voice from a multi-talker mixture using a short enrollment utterance as a reference. While state-of-the-art models achieve strong performance on standard benchmarks (e.g., VoxCeleb, WSJ0-2mix), these datasets rely on artificially mixed clean recordings. In contrast, real-world conversational data exhibits complex characteristics such as reverberation, background noise, device distortions, and irregular overlap patterns.

A critical bottleneck exists in deploying TSE to real scenarios: real conversational recordings lack clean reference signals for individual speakers. Consequently, conventional signal-level supervision methods (e.g., SI-SNR loss) cannot be applied directly. Existing systems, including the official REAL-T baseline, are trained on simulated mixtures and often degrade when deployed in real environments. The challenge is how to effectively leverage large-scale real conversational recordings for TSE training without access to clean target speech.

Methodology: The PS4 Framework

The authors propose PS4, a proxy-supervised training framework designed to train TSE models directly on real conversational mixtures. Instead of requiring clean target speech, PS4 utilizes multiple "proxy" supervision signals derived directly from the raw data.

1. Corpus Construction (REAL-PS4)

The authors constructed REAL-PS4, a large-scale training corpus containing 71,771 samples derived from four public datasets: AISHELL-4, AliMeeting, AMI, and CHiME-6. The construction pipeline involves:

• Enrollment Extraction: Identifying single-speaker regions (>5s) from diarization annotations to create enrollment utterances.
• Mixture Generation: Extracting overlapping speech regions (≥2 speakers) from the original recordings. Adjacent overlaps are merged if gaps are <0.5s.
• Quality Filtering: Retaining samples where the target speaker occupies >20% of the duration and the transcript contains ≥2 valid characters. Audio is truncated to 30s for the mixture and 10s for enrollment to manage computational constraints.
• Annotations: Each sample includes the mixture, enrollment audio, ground-truth transcripts, and frame-level Voice Activity Detection (VAD) labels derived from diarization.

2. Model Architecture and Training Strategy

The system builds upon a pre-trained BSRNN-ECAPA model (a BSRNN separator with an ECAPA-TDNN speaker encoder). During fine-tuning, only the BSRNN separator is updated, while the speaker encoder remains frozen.

The core innovation is a joint optimization strategy using four complementary, differentiable proxy objectives:

1. Linguistic Supervision ($L_{CE}$): A frozen Whisper large-v3 model acts as a differentiable teacher. The model minimizes the cross-entropy between the predicted token distribution of the extracted speech and the ground-truth transcript.
2. Speaker Supervision ($L_{SIM}$): A hinge ranking loss ensures the similarity between the extracted speech and the enrollment utterance exceeds the similarity between the original mixture and the enrollment by a predefined margin. This utilizes a frozen ResNet34 speaker encoder.
3. Temporal Supervision ($L_{VAD}$): A binary cross-entropy loss minimizes the difference between the predicted frame-level energy of the extracted speech and the ground-truth VAD labels derived from diarization.
4. Perceptual Supervision ($L_{DNSMOS}$): A differentiable implementation of DNSMOS is used to maximize the overall perceptual quality score of the extracted speech, without requiring a clean reference.

The total loss function is a weighted sum:
$$L = \lambda_{ce}L_{CE} + \lambda_{sim}L_{SIM} + \lambda_{dns}L_{DNSMOS} + \lambda_{vad}L_{VAD}$$

Key Results

The authors evaluated PS4 on the REAL-T benchmark, which consists of real conversational recordings from five datasets (AISHELL-4, AliMeeting, AMI, CHiME-6, DipCo).

• Development Set Performance: PS4 consistently outperformed the official BSRNN-based baselines (BSRNN-TFMAP and BSRNN-EMB) across all five sub-corpora and all four metrics (Token Error Rate, Speaker Similarity, DNSMOS OVRL, and Timing F1).
  • Notably, PS4 achieved significant improvements in perceptual quality (DNSMOS OVRL > 3.1 vs. baselines < 2.0) and speaker similarity.
  • AMI recordings yielded the best results (TER: 0.388, SIM: 0.714), attributed to cleaner lapel microphone channels.
• Leaderboard Performance: On the official REAL-T challenge leaderboard (validation set), PS4 ranked 2nd overall with a composite score of 3.25.
  • It achieved the best Speaker Similarity (0.565) and best Timing F1 (0.871) among all submitted systems.
  • While its DNSMOS-P808 score (3.128) and TER were slightly lower than the top-ranked system (MERL's), it decisively outperformed the leader in identity preservation and timing accuracy.

Significance and Claims

The paper claims that PS4 provides an effective and practical solution for training TSE models directly on real conversational recordings, bridging the gap between simulated benchmarks and real-world deployment.

• Novelty in Supervision: The work demonstrates that "proxy supervision"—using linguistic, speaker, temporal, and perceptual signals available in raw data—can effectively replace the need for clean target speech references.
• Scalability: By constructing the REAL-PS4 corpus from public datasets, the authors show that large-scale training data for real-world TSE can be generated without manual annotation of clean speech.
• Performance: The results validate that proxy-supervised joint training yields models that are more robust to real-world acoustic conditions than those trained solely on simulated mixtures, particularly regarding speaker identity preservation and temporal alignment.

The authors conclude that proxy supervision is a viable alternative to conventional signal-level training for TSE in real conversational scenarios.