Position-invariant Fine-tuning of Speech Enhancement Models with Self-supervised Speech Representations

Technical Summary: Position-Invariant Fine-Tuning of Speech Enhancement Models with Self-Supervised Speech Representations

Problem Statement
Integrating front-end speech enhancement (SE) models with self-supervised learning (SSL) representations (e.g., wav2vec 2.0, HuBERT) has proven effective for downstream tasks in noisy environments. The standard approach involves fine-tuning the SE model using a Mean Squared Error (MSE) loss between the SSL representations of enhanced speech and clean speech (SSL-MSE). However, this paper identifies a critical limitation: the MSE objective is prone to exploiting positional embeddings inherent in SSL models. Instead of learning meaningful content similarity, the optimization process minimizes loss through positional correlations. This phenomenon, analogous to "positional collapse" observed in SSL pre-training, reduces the generalization and robustness of the SE frontend, particularly in unseen noisy conditions.

Methodology
The study investigates this issue through a representation-guided SE framework where a pre-trained SE model is fine-tuned to align its output with the representations of a frozen SSL encoder. To mitigate the exploitation of positional embeddings, two strategies are proposed and evaluated:

1. Positional Perturbation via Random Zero-Padding (SSL-MSE-PAD):
   Adapted from the SPIRAL framework used in SSL pre-training, this method introduces random zero-padding to the clean reference waveform before it is passed through the SSL encoder. The padding length is randomly sampled (2–5% of the signal duration) and aligned with the SSL model's frame hop size. To maintain alignment with the unpadded enhanced signal, an equivalent number of frames are trimmed from the padded SSL representations before computing the MSE. This disrupts absolute positional alignment, forcing the model to rely more on content.

2. Speed Perturbation with Soft-DTW Loss (SSL-SoftDTW):
   This approach combines speed perturbation (varying the playback speed of the clean signal) with a soft-Dynamic Time Warping (soft-DTW) loss. Unlike zero-padding, speed perturbation introduces continuous, local temporal distortions that simulate realistic speech timing variability. Since the enhanced and perturbed clean signals differ in length, standard frame-wise MSE is inapplicable. Instead, soft-DTW—a differentiable extension of DTW using a soft-min operator—aligns the sequences based on content rather than absolute position. The loss is normalized by sequence length to ensure stability.

Experimental Setup
The experiments utilize the master64 SE model (a deep convolutional time-domain network) fine-tuned on a noise-augmented subset of LibriSpeech (train-clean-100) mixed with DEMAND indoor noise. The SSL backbone is HuBERT-BASE. The fine-tuned SE models are subsequently evaluated by integrating them as front-ends for downstream tasks: Automatic Speech Recognition (ASR) and Phoneme Recognition (PR), following the SUPERB benchmark protocol. Evaluation includes clean, seen-noise (indoor), and unseen-noise (outdoor) conditions.

Key Results

• Downstream Performance: The SSL-SoftDTW approach consistently outperformed the baseline SSL-MSE and the SSL-MSE-PAD variant. In ASR tasks, SSL-SoftDTW achieved the lowest Word Error Rate (WER) in unseen noise conditions (9.06% vs. 9.19% for SSL-MSE). Similarly, for Phoneme Recognition, SSL-SoftDTW yielded the best Phoneme Error Rate (PER) (5.11% vs. 5.17% for SSL-MSE).
• Convergence Speed: A significant finding is the convergence rate. The SSL-SoftDTW approach converged to the final performance level of the SSL-MSE baseline in approximately 60k steps, whereas the SSL-MSE baseline required roughly 200k steps.
• Zero-Padding Limitations: While SSL-MSE-PAD showed faster convergence than the baseline, it offered only marginal performance gains. The authors attribute this to the artificial discontinuities introduced by zero-padding, which may interfere with SSL feature extraction.

Significance and Claims
The paper claims that the exploitation of positional embeddings is a general limitation in SSL-based fine-tuning that adversely affects the robustness of SE frontends. The primary contribution is the demonstration that position-invariant fine-tuning strategies, specifically those utilizing soft-DTW alignment, can significantly improve downstream task performance and training efficiency.

Crucially, the authors emphasize that these improvements were achieved by fine-tuning only the SE module, without modifying the SSL encoder or the downstream task heads. This suggests that mitigating positional dependence at the component level is sufficient to yield measurable gains. The work underscores the importance of moving beyond simple MSE alignment in favor of content-based, position-invariant objectives when integrating SE models with SSL representations. Future work is suggested to explore applying these techniques during the computationally expensive SSL pre-training stage itself.