Causally Disentangled Contrastive Learning for Multilingual Speaker Embeddings

Technical Summary: Causally Disentangled Contrastive Learning for Multilingual Speaker Embeddings

Problem Statement
Self-supervised speaker embeddings, widely used in speaker verification systems, have been shown to inadvertently encode sensitive demographic attributes such as gender, age, and accent. This "demographic leakage" poses significant risks to fairness, privacy, and ethical compliance, potentially leading to systematic biases or the unauthorized inference of protected attributes. While prior work has addressed this in supervised settings, there is a lack of systematic investigation into how demographic leakage manifests in self-supervised contrastive learning frameworks (e.g., SimCLR) and whether it can be mitigated without severely degrading speaker verification performance. Furthermore, existing mitigation strategies often rely on correlation-based approaches, whereas the underlying mechanisms may be causal, necessitating architectures that explicitly disentangle these factors.

Methodology
The study employs a structured framework to investigate and mitigate demographic leakage in speaker embeddings trained via SimCLR (Simple Contrastive Learning of Representations).

1. Data and Preprocessing: The primary dataset is the Mozilla Common Voice (English, Scripted Speech 23.0), filtered to include only validated clips with binary gender labels (Male/Female), categorized age groups (Young, Adult, Senior), and standardized accent groups. An external validation is performed using the Sonos Voice Control Bias Assessment Dataset.
2. Baseline (SimCLR): A baseline model is trained using SimCLR to learn speaker-discriminative representations. The training involves stochastic waveform-level augmentations (cropping, noise, amplitude scaling) and an NT-Xent loss. The resulting embeddings are evaluated using linear and non-linear (MLP) probing classifiers to quantify the extent and structure of demographic leakage.
3. Adversarial Debiasing: To suppress demographic information, an adversarial framework is introduced. Gradient Reversal Layers (GRL) are placed between the encoder and auxiliary classifiers (adversaries) tasked with predicting gender, age, and accent. The encoder is trained to minimize speaker verification loss while maximizing the adversaries' loss, effectively removing demographic cues. The strength of this debiasing is controlled by a hyperparameter $\lambda$.
4. Causal Bottleneck Architecture: A novel architectural intervention is proposed to explicitly separate demographic and residual information. The encoder output is split into two branches:
   • Demo Branch: A low-dimensional linear projection supervised to capture demographic attributes.
   • Residual Branch: A projection intended to carry speaker-discriminative features independent of demographics.
   • Constraints: The residual branch is subjected to adversarial debiasing (GRL) to suppress demographic leakage. Additionally, a covariance penalty is applied to minimize the covariance between the demo and residual branches, enforcing independence.

Key Contributions

• Systematic Quantification of Leakage: The paper provides a comprehensive analysis of how SimCLR-trained embeddings encode gender, age, and accent, distinguishing between linear and non-linear leakage patterns.
• Comparative Evaluation of Mitigation: It contrasts standard adversarial debiasing with a causally motivated bottleneck architecture, evaluating their efficacy in suppressing demographic information across multiple axes.
• Characterization of Trade-offs: The study rigorously maps the trade-off between demographic fairness (leakage reduction) and speaker verification utility (ROC-AUC and EER) under different debiasing regimes.
• External Validation: Findings are validated on an external dataset (Sonos), demonstrating the robustness of the observed leakage hierarchies and mitigation trade-offs across different data distributions.

Results

• Leakage Structure: In the baseline SimCLR embeddings, gender information is strongly and linearly encoded (linear probe accuracy >90%). Age is encoded more diffusely and non-linearly (linear probe near chance, MLP probe significantly higher), while accent leakage is the weakest and most unstable.
• Adversarial Debiasing: Increasing the adversarial strength ($\lambda$) successfully reduces gender leakage, particularly for linear probes. However, it has limited effect on age and accent leakage. Crucially, this reduction comes with a clear trade-off: speaker verification performance degrades steadily as $\lambda$ increases (e.g., ROC-AUC drops from ~0.83 to ~0.62 at high $\lambda$).
• Causal Bottleneck: The causal bottleneck architecture further suppresses demographic information in the residual branch, achieving lower leakage than adversarial debiasing alone (e.g., gender leakage reduced from 90% to ~57% in linear probes). However, this approach incurs a substantial performance cost, with speaker verification ROC-AUC dropping to near-chance levels (0.50–0.60).
• External Validation: The Sonos dataset validation confirms the hierarchy of leakage (Gender > Age > Accent) and the persistence of the fairness-utility trade-off, although accent prediction was notably weaker on this dataset.

Significance and Claims
The paper claims that demographic leakage in self-supervised speaker embeddings is a fundamental representation-learning problem where task-relevant and sensitive attributes are entangled. The findings highlight that:

1. Linear vs. Non-Linear: Linear probing alone underestimates demographic leakage, particularly for age, necessitating non-linear evaluation methods.
2. Inherent Trade-offs: Mitigating demographic leakage, especially for attributes strongly correlated with acoustic features like gender, inherently compromises speaker verification performance.
3. Limitations of Current Methods: While causal bottlenecks offer stronger control over leakage than adversarial objectives, they do not eliminate the entanglement without sacrificing utility. The study concludes that demographic leakage can be reduced but not entirely removed without significant degradation in speaker verification performance, underscoring the inherent limitations of current debiasing approaches in self-supervised settings.