Where Do Self-Supervised Speech Models Become Unfair?

1. Problem Statement

Self-supervised speech models (S3Ms) have demonstrated superior performance in tasks like Automatic Speech Recognition (ASR) and Speaker Identification (SID). However, it is well-established that these models exhibit significant performance disparities across different Speaker Groups (SGs) (e.g., based on gender, age, native language, or dialect). Despite numerous attempts to mitigate these gaps through fairer training algorithms, the performance gap persists.

The core problem addressed by this paper is the lack of understanding regarding the technological origin and propagation of bias within S3Ms. Specifically, the authors seek to answer:

• At which specific layers of the neural network does bias emerge?
• How does the relationship between overall task performance and bias against specific SGs evolve layer-by-layer?
• Can fine-tuning (with or without fairness-enhancing algorithms) remove biases established during pretraining?

2. Methodology

The authors propose a layerwise fairness analysis framework to probe pretrained S3Ms.

• Probing Architecture: Instead of using complex decoders, the authors train lightweight, single-layer linear "probes" (decoders) on the latent embeddings of every layer of the encoder. This ensures that any observed bias originates from the pretrained S3M embeddings rather than the decoder's learning capacity.
• Tasks: Two downstream tasks are evaluated at each layer:
  1. Speaker Identification (SID): Measured via Speaker Identification Error Rate (SIDER).
  2. Automatic Speech Recognition (ASR): Measured via Word Error Rate (WER).
• Datasets:
  • Sonos Voice Control Bias Assessment Dataset: Contains metadata for gender, age, dialect, and native language.
  • Meta's Fair-speech corpus: Contains metadata for ethnicity, age, gender, and socioeconomic background.
• Models Analyzed: A diverse set of State-of-the-Art (SOTA) pretrained S3Ms (e.g., WavLM, Wav2Vec 2.0, XLS-R, BEST-RQ) varying in architecture (Transformer vs. Conformer), size, and pretraining data (monolingual vs. multilingual, English vs. French). They also include Whisper-medium as a non-SSL baseline (end-to-end trained).
• Fairness Metric: The authors define Relative Error Rate ($Err_{rel}$) as the difference between the error rate of a specific SG and the average error rate of the general population. They aggregate this into a DV-bias metric (mean absolute relative error across all SGs for a demographic variable).

3. Key Contributions

1. First Layerwise Fairness Analysis: This is the first study to systematically probe every embedding layer of pretrained S3Ms to map the evolution of bias for both SID and ASR.
2. Discovery of Divergent Bias-Performance Relationships: The paper identifies a fundamental, opposing relationship between performance and bias for SID versus ASR.
3. Pretraining vs. Fine-tuning Dynamics: The study demonstrates that SG-level bias is "baked in" during pretraining and is largely immutable by standard fine-tuning or fairness-enhancing algorithms (like Adversarial Training).

4. Key Results

A. Emergence of Bias (RQ1)

• Immediate Bias: Bias against certain SGs is present starting from the very first latent layer of all S3Ms.
• Affected Groups:
  • ASR: Non-native speakers and children suffer the most bias.
  • SID: Children and women suffer the most bias.
• Universality: These patterns hold true across models trained on English, French, and multilingual data, suggesting the bias is not solely due to unbalanced pretraining corpora but may be inherent to the modeling process.

B. Layerwise Bias vs. Performance (RQ2)

The study reveals a stark contrast between SID and ASR:

• Speaker Identification (SID):
  • Inverse Relationship: SID performance is highest in the earliest layers, where bias is lowest.
  • As the model moves to deeper layers, SID performance degrades, and bias against specific SGs (women, children) increases.
  • Implication: The features most useful for identifying speakers are learned fairly; unfairness arises in layers that encode less relevant information.
• Automatic Speech Recognition (ASR):
  • Direct Relationship: ASR performance improves (WER decreases) in deeper layers, but bias increases simultaneously.
  • Layers that achieve the best ASR performance (lowest WER) also exhibit the highest SG-level bias.
  • Implication: S3Ms learn ASR-relevant features at the expense of fairness, optimizing for the majority group while sacrificing minority groups.

C. Impact of Fine-tuning (RQ3)

• Ineffectiveness of Fine-tuning: Fine-tuning S3Ms for ASR significantly improves overall performance but has negligible effect on reducing SG-level bias.
• Failure of Fairness Algorithms: Even when using fairness-enhancing techniques like Domain-Enhancing Training (DET) and Adversarial Training (DAT), the bias patterns established during pretraining persist. The bias is "baked in" and difficult to remove via downstream adaptation.

5. Significance and Future Directions

• Theoretical Insight: The findings challenge the assumption that fine-tuning can fix fairness issues. They suggest that the "fairness gap" is a fundamental property of how S3Ms learn to represent speech, particularly for ASR tasks.
• Practical Implication: Simply applying fairness constraints during the fine-tuning stage is insufficient. The community must shift focus toward fairer pretraining techniques to prevent bias from being encoded in the first place.
• Future Work: The authors suggest moving beyond broad SG labels (e.g., "non-native") to analyzing specific acoustic attributes (e.g., dialect density, cadence) to understand the causal mechanisms driving these biases.

In summary, the paper establishes that self-supervised speech models are inherently biased from their first layer, with ASR tasks exhibiting a direct trade-off between performance and fairness, a dynamic that standard fine-tuning cannot resolve.