Learning task-specific subspaces via interventional post-training of speech foundation models

Technical Summary: Learning Task-Specific Subspaces via Interventional Post-Training of Speech Foundation Models

Problem Statement
Speech foundation models, pre-trained on vast corpora of unlabelled data, generate general-purpose representations that exhibit strong task generalisability. However, these representations encode information about various speech variables (e.g., speaker identity, content, channel conditions) in a distributed, entangled manner. Downstream tasks, such as Automatic Speech Recognition (ASR) or Speaker Identification (SID), typically rely on only a fraction of this variability. For instance, an ASR system should be invariant to the speaker, while a speaker verification system should be invariant to the spoken content. Existing approaches to learning disentangled representations often rely on assumptions of statistical independence or Gaussianity, which have proven insufficient for guaranteeing true disentanglement. Furthermore, standard contrastive learning typically optimizes for a single embedding space invariant to nuisance variables, rather than explicitly separating multiple sources of variability into distinct subspaces.

Methodology
The authors propose a post-training refinement framework termed Interventional Contrastive Learning. This approach aims to transform the entangled representation space of a frozen speech foundation model into separate, task-specific subspaces (specifically content and speaker) using a multi-part contrastive loss.

• Interventional Dataset: To provide the necessary supervision for disentanglement, the authors synthesize a dataset using a state-of-the-art zero-shot Text-to-Speech (TTS) system (F5-TTS). This dataset allows for "dense interventions" where every speaker is paired with every utterance text. This synthetic setup enables precise control over causal variables: the content (text) and the speaker (reference audio).
• Architecture: The model consists of a frozen pre-trained backbone (e.g., wav2vec 2.0, HuBERT, or WavLM) that extracts features from the input speech. These features are aggregated via mean pooling into a single utterance embedding. A trainable Subspace Network (a 3-layer MLP without a bottleneck) then projects this embedding into a higher-dimensional space partitioned into two equal-sized subspaces: a content subspace ($z_c$) and a speaker subspace ($z_s$).
• Loss Function: The training objective combines two components:
  1. Multi-Positive Contrastive Loss: Extended to handle multiple subspaces, this loss encourages embeddings within the same subspace to be close if they share the same causal variable (e.g., same speaker for the speaker subspace) and far if they differ. It utilizes interventional labels to define positive and negative pairs based on whether the causal variable was intervened upon.
  2. Orthogonality Loss: A regularization term based on the Frobenius norm of the cross-covariance between subspaces, penalizing similarity between the content and speaker representations to enforce disentanglement.

Key Contributions

1. Novel Contrastive Loss: The paper introduces a loss function capable of learning multiple embedding spaces associated with distinct causal variables simultaneously, moving beyond single-space invariance.
2. Synthetic Interventional Dataset: The authors construct a dataset with exhaustive combinations of speakers and texts, facilitating dense interventions on causal variables that are difficult to obtain from natural speech data.
3. Empirical Validation: The approach is evaluated on Speaker Verification (SV) and Keyword Spotting (KS) tasks, demonstrating the separation of information across learned subspaces.

Results
The experiments were conducted on three backbone models (wav2vec 2.0, HuBERT, WavLM) using the synthetic dataset.

• Speaker Verification (Out-of-Domain): The full model, learning both subspaces jointly, achieved significantly lower Equal Error Rates (EER) on the VoxCeleb1 test set compared to the baseline (mean-pooled backbone) and single-subspace variants. Notably, the best models achieved an EER of approximately 24.7% (WavLM) and 27.2% (HuBERT), outperforming non-expert human baselines (26.51%) despite being trained on only 32 speakers and clean read speech, while tested on wild spontaneous speech.
• Keyword Spotting: The learned content subspace showed performance comparable to, though slightly lower than, the baseline pooled representations. The authors attribute this to a mismatch between the sentence-level similarity enforced by the contrastive pretext task and the word-level requirements of keyword spotting.
• Disentanglement Evidence: The gap in performance between the "matched" subspace (e.g., speaker subspace on SV) and the "mismatched" subspace (e.g., speaker subspace on KS) provides evidence that the intervention successfully separated speaker and content information. However, the gap was less pronounced for the KS task, suggesting potential information leakage or that KS requires less content information than anticipated.
• Joint vs. Separate Learning: The results indicated no significant performance advantage for jointly learning the subspaces compared to learning them separately, although the joint model utilized parameters more efficiently.

Significance and Claims
The paper claims that interventional contrastive learning is a viable strategy for post-training speech foundation models to separate task-relevant information from nuisance variables. The primary significance lies in the ability to learn a data-agnostic transformation that improves Out-of-Domain Speaker Verification performance using a small, synthetic dataset, without requiring extensive labelled data for the downstream task. The authors modestly note that while the approach successfully separates speaker information, the separation of content information for keyword spotting remains imperfect, likely due to the granularity mismatch between the pretext task and the downstream task. They conclude that further work is needed to scale the approach to larger, real-world datasets and to refine the loss functions to better penalize information leakage.