💬 NLP

Speaker Group Encoding in Self-supervised Speech Recognition Models

This paper investigates how self-supervised speech recognition models encode speaker group information across different training stages, revealing that while speaker identification fine-tuning amplifies phonetic variations and ASR fine-tuning discards them while retaining semantic ones, fairness-enhancing algorithms primarily mitigate phonetic biases, thereby offering insights for designing more equitable ASR systems.

Original authors: Felix Herron, Solange Rossato Alexandre Allauzen, Benoit Favre, François Portet

Published 2026-06-10

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Felix Herron, Solange Rossato Alexandre Allauzen, Benoit Favre, François Portet

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a self-supervised speech model (S3M) as a super-smart, multi-layered translator that listens to a person speaking and tries to understand not just the words, but the person behind the voice. This paper investigates what this translator "learns" about different groups of people (like men vs. women, young vs. old, or people with different accents) as it processes the sound.

Here is a breakdown of their findings using simple analogies:

1. The Translator's "Layers"

Think of the model as a factory with 24 floors (layers).

The Lower Floors: These are like the ears. They catch the raw sounds, the pitch, and the tone.
The Middle Floors: These are like the phonetic analysts. They figure out the specific sounds (like "p" vs. "b").
The Top Floors: These are like the semantic thinkers. They focus on the meaning of the words and the sentence structure.

The researchers found that the model naturally learns to recognize different types of people at different floors.

2. Two Types of "Voice Differences"

The paper discovers that people differ in two main ways, and the model treats them differently:

The "Musical" Differences (Phonetic): These are things like Gender and Age. A child's voice is high-pitched; an older man's is deep. A woman's voice sounds different from a man's. These are like the instrument playing the music.
- The Finding: The model catches these differences early (in the middle floors). If you train the model to be a "Speaker ID" expert (to recognize who is talking), it gets really good at spotting these musical differences.
The "Storytelling" Differences (Semantic): These are things like Dialect, Ethnicity, and Native vs. Non-native status. These are about how the story is told—pausing in different places, using different words for the same thing, or having a specific accent. These are like the lyrics or the style of the song.
- The Finding: The model holds onto these differences all the way to the top floors. Even if you train the model to just transcribe words (ASR), it still remembers these "storytelling" traits.

3. What Happens When You Train the Model?

The researchers tested the model in four different "modes":

Mode A: The Raw Learner (Pretrained): The model learns from pure audio. It picks up on both "Musical" and "Storytelling" differences naturally.
Mode B: The "Who is Talking?" Expert (Speaker ID): When you train the model to identify specific people, it amplifies the "Musical" differences (Gender, Age). It becomes hyper-aware of pitch and tone. However, it doesn't really get better at spotting the "Storytelling" differences (Dialect, Native status).
Mode C: The "What was Said?" Expert (Speech Recognition/ASR): When you train the model to just write down words, it throws away the "Musical" differences. It learns to ignore gender and age because those don't change the meaning of the words. However, it keeps the "Storytelling" differences. It still "hears" the accent or the dialect because those can change the meaning of the sentence.
Mode D: The "Fairness" Expert: The researchers tried special training tricks to make the model "fair" (so it doesn't treat men and women differently).
- The Result: These tricks successfully made the model ignore the "Musical" differences (Gender/Age). The model became "blind" to them in the later layers.
- The Catch: These tricks failed to make the model ignore the "Storytelling" differences (Dialect/Native status). The model still remembered those traits, even when trying to be fair.

4. The "Fairness" Problem

The paper highlights a tricky situation.

We have tools that can make a model ignore Gender and Age (the "Musical" traits).
But we cannot easily make the model ignore Dialect or Native Status (the "Storytelling" traits) using current methods.
Since the biggest fairness problems in speech recognition often come from dialects and non-native speakers, the current "fairness" tools are only fixing half the problem. They are good at making the model blind to who is speaking, but not how they speak.

5. Where is the Information Hiding?

The researchers also looked at where in the model's "brain" this information lives.

They found that information about a speaker isn't just in one place; it's spread out.
Interestingly, the beginning of a sentence (the first second) often holds more information about the speaker's group than the end of the sentence. This is likely because smart speakers usually start with a specific "wake word" (like "Hey Siri"), which gives a consistent starting point for the model to analyze the voice.

Summary

The paper concludes that self-supervised speech models are like a multi-sensory camera.

If you tell it to focus on identity, it zooms in on the voice pitch (Gender/Age).
If you tell it to focus on transcription, it ignores the pitch but keeps the accent (Dialect/Native).
Current "fairness" tools are like a filter that successfully blurs out the pitch, but they leave the accent perfectly clear. The authors suggest we need new tools to blur out the accent if we truly want a fair system.

Technical Summary: Speaker Group Encoding in Self-supervised Speech Recognition Models

Problem Statement

Self-supervised speech encoder models (S3Ms) serve as versatile backbones for downstream tasks like automatic speech recognition (ASR) and speaker identification (SID). However, it is well-documented that these models exhibit performance disparities across different speaker groups (SGs), such as non-native speakers, speakers of non-standard dialects, and speakers of varying ages or genders. While previous research has established that these disparities exist, there is limited understanding of how S3Ms encode speaker group information (SGI) throughout their layer-by-layer processing, how this encoding shifts during fine-tuning, and how fairness-enhancing algorithms alter this representation. This paper investigates the nature, distribution, and persistence of SGI within S3Ms to better understand the mechanisms behind these performance gaps.

Methodology

The authors employ a multi-faceted approach to analyze SGI encoding across various S3M states: pretrained, fine-tuned for SID, fine-tuned for ASR, and fine-tuned with fairness-enhancing algorithms.

1. Models and Data

Models: The study analyzes two primary architectures: Wav2Vec 2.0 (W2V2) and WavLM. Both "base" and "large" configurations are examined, with a focus on models pretrained on LibriVox-derived data (LibriLight, VoxPopuli) and the multilingual W2V2-XLSR-53.
Datasets: Two datasets annotated with precise speaker metadata are used for probing:
- Sonos Voice Control Bias Assessment: Contains 170k utterances labeled with gender, dialect (8 classes), and age (5 classes).
- Meta Fair-speech: Contains 26.5k utterances labeled with gender, ethnicity, socio-economic background, age, and native speaker status.
Fine-tuning Variants: Models are fine-tuned using:
- Vanilla ASR: Using CTC loss.
- SID: Using x-vector based classifiers.
- Fairness Algorithms: Domain Adversarial Training (DAT) to remove speaker information and Domain Enhancing Training (DET) to retain it in intermediate layers.

2. Probing Strategy

To detect SGI without overfitting to specific speaker identities (a common issue in SID vs. SG identification), the authors utilize a two-tiered linear probe:

Projection Layer ( $M_P$ ): Maps the high-dimensional model embedding ( $d_M=1024$ ) to a low-dimensional latent space ( $d_P=5$ ).
Classification Layer ( $M_C$ ): Maps the latent space to the target Speaker Group Category (SGC) classes.
Adversarial Constraint: A reverse gradient layer is applied to a parallel SID branch to ensure the intermediate embeddings remain agnostic to specific speaker identities, forcing the probe to learn general SGI features rather than memorizing specific speakers.

3. Analysis Dimensions

Layer-wise Analysis: Probing is performed at every layer to track the evolution of SGI.
Pooling Strategies: The study compares pooling over the entire sequence, the first/last 50 frames, and single frames to determine temporal distribution.
Dimensional Analysis: Principal Component Analysis (PCA) is used to measure the cosine similarity between the principal components of SGC centroids and speaker ID centroids, assessing whether SGI is stored orthogonally to speaker identity.

Key Results

1. Pretraining and Architecture Impact

WavLM Superiority: WavLM-large captures more SGI than W2V2 models. The authors attribute this to WavLM's reconstructive pretraining objective (inspired by HuBERT), which may encourage the retention of fundamental signal information (including SGI) compared to the contrastive loss used by W2V2.
Multilingualism: Multilingual pretraining (W2V2-XLSR-53) did not significantly increase SGI capture compared to English-only pretraining, suggesting multilingualism alone does not force the learning of these features.

2. Divergence of SGCs During Fine-tuning

The study identifies two distinct classes of Speaker Group Categories (SGCs) based on how they respond to fine-tuning:

Phonetically Variant SGCs (Gender, Age):
- Pretrained: Information peaks in middle layers.
- ASR Fine-tuning: This information is largely discarded in later layers, as it is deemed less relevant for transcription.
- SID Fine-tuning: This information is amplified and retained through later layers.
- Fairness (DAT): DAT effectively removes this information in later layers, rendering the model "blind" to these groups.
Semantically Variant SGCs (Dialect, Ethnicity, Native Status):
- Pretrained: Information is present but less dominant in early layers.
- ASR Fine-tuning: Unlike phonetic features, these are retained and often increase in detectability in later layers. The authors hypothesize this is because dialect and native status involve prosodic and semantic differences (e.g., pauses, grapheme choices) that are relevant to ASR performance.
- Fairness (DAT): DAT has negligible effect on these categories; they remain detectable even in the final layers of fairness-optimized models.

3. Temporal and Dimensional Distribution

Temporal: Contrary to some previous findings on speaker ID, SGI for most categories (except gender) is more strongly captured in the first 50 frames (the "wake phrase" of smart speaker commands) than in the final frames.
Orthogonality: PCA analysis reveals that in DAT-finetuned models, certain SGCs (like gender and native status) are forced to be encoded along axes orthogonal to speaker identity, rather than being completely erased. This suggests DAT changes the representation of SGI rather than eliminating it entirely.

Significance and Claims

The paper does not propose a new fairness algorithm but provides a diagnostic framework to explain the limitations of current approaches.

Mechanism of Fairness Algorithms: The study provides a theoretical explanation for why Domain Adversarial Training (DAT) is effective for phonetically variant groups (gender, age) but ineffective for semantically variant groups (dialect, ethnicity). DAT successfully suppresses phonetic variance but fails to suppress semantic variance, which the model retains because it is useful for the ASR task.
Limitations of Current Fairness: The authors note that the groups exhibiting the greatest fairness divergence in ASR (often semantically variant) are precisely the ones least affected by current DAT/DET methods.
Future Directions: The findings suggest that future fairness algorithms should move beyond simple speaker ID invariance. Instead, they propose experimenting with multi-task fine-tuning that combines multiple SGC classifiers (e.g., using accent labels alongside speaker IDs) to specifically target and suppress the retention of semantically variant SGI in later layers.

In summary, the paper demonstrates that S3Ms encode SGI in a complex, task-dependent manner where "useful" semantic speaker information is preserved even in ASR-finetuned models, while "noise" phonetic information is discarded. This distinction is critical for designing truly fair ASR systems.