🤖 machine learning

Contrastive Regularization for Accent-Robust ASR

This contribution proposes a light supervised contrastive learning strategy (SupCon) for regularization that improves robustness to accents in CTC-finetuned ASR systems by promoting compact encoder representations and achieves a relative WER reduction of up to 29% on the L2-ARCTIC benchmark without requiring architectural changes or explicit accent labels.

Original authors: Van-Phat Thai, Aradhya Dhruv, Duc-Thinh Pham, Sameer Alam

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Van-Phat Thai, Aradhya Dhruv, Duc-Thinh Pham, Sameer Alam

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Accent Barrier"

Imagine you have a very intelligent robot that can listen to people speak and write down exactly what they say. This robot was trained mainly on "native" speakers – people who speak with the standard accent of the region.

The robot is great at this. But as soon as someone with a different accent (such as a non-native speaker) talks to it, the robot gets confused. It starts making mistakes, swapping words, or misunderstanding the meaning. This is a big problem because the world is full of different accents, and we want our speech technology to work for everyone, not just for a specific group.

The Solution: A "Group Hug" for Similar Sentences

The researchers in this paper tried to teach the robot a new trick so it would be less confused by accents. They neither changed the robot's brain (its architecture) nor hired a new teacher to teach it specific accents. Instead, they added a special "exercise" called Supervised Contrastive Learning (SupCon).

Here is how it works, using a metaphor:

Imagine you are organizing a library.

The old way (Standard Training): You tell the robot: "Read this sentence and write it down." If the robot gets it right, it gets a gold star. If it gets it wrong, it gets a red mark. The robot learns to recognize the words, but it does not necessarily learn that the same sentence, spoken by a French person, a Japanese person, and a Brazilian, is all the same "thing" beneath the accent.
The new way (SupCon): The researchers added a second rule. They told the robot: "Look at these two different recordings. One is spoken by a French person, the other by a Japanese person. They are saying exactly the same sentence. Hug them!"

In technical terms, the robot is forced to recognize that although the sounds (accents) are different, the meaning (the transcript) is the same. It learns to bring these different versions of the same sentence closer together in its "brain" (mathematical space), while keeping different sentences further apart.

How They Tested It

They tested this on a famous test set called L2-ARCTIC, which contains English spoken by people with six different native language backgrounds (such as Arabic, Mandarin, Hindi, etc.).

They set up two tough challenges:

The "New Speaker" Test: The robot had to understand people it had never heard before, but who spoke an accent it had already heard.
The "New Accent" Test: The robot had to understand a completely new accent it had never heard during training.

The Results: A Big Win for the "New Accent" Test

The results were impressive, especially for the most difficult challenge:

Standard Training: When the robot faced a brand-new accent it had never heard, it made about 10% errors.
With the "Group Hug" (SupCon): The error rate dropped to about 7.4%.

This is an improvement of 25% to 29% compared to the old method. This means the robot became much more robust. It did not just memorize the specific accents it had seen; it learned the shape of the sentences so well that it could handle accents it had never encountered before.

Why It Works: The "Compact Sphere" Theory

The researchers looked inside the robot's "brain" to see what had changed. They found that without the new training, the robot's understanding of the same sentence spoken by different people was scattered everywhere (like a messy pile of books).

With the new training, the robot's understanding of the same sentence became a solid, compact lump. No matter who spoke it or what accent they used, the robot recognized it as the same object. This "solidity" made the robot much more stable and accurate.

The Bottom Line

This paper shows that you do not need to build a huge, complex new system to fix accent problems. You can take an existing, powerful speech system and add a simple, lightweight "exercise" that teaches it to group similar sentences together, regardless of accent.

No new hardware required.
No need to explicitly label every accent.
It works better on accents the robot has never seen before.

It is like teaching a student not just to memorize answers, but to understand the concept so well that they can answer the question even if it is asked in a different language or dialect.

Technical Conclusion: Contrastive Regularization for Accent-Robust ASR

Problem Statement
While modern Automatic Speech Recognition (ASR) systems, which leverage self-supervised acoustic pretraining and fine-tuning with Connectionist Temporal Classification (CTC), achieve strong performance on native speakers, they remain highly sensitive to accentual variability. This sensitivity leads to significant performance degradation in scenarios involving non-native speakers, particularly in resource-constrained or globally deployed environments where systematic pronunciation deviations from native norms occur. Existing solutions often rely on explicit accent monitoring (e.g., accent embeddings or classifiers) or data augmentation, which may not generalize well to unseen accents or require architectural modifications.

Methodology
The authors propose integrating Supervised Contrastive Learning (SupCon) as a lightweight, model-agnostic auxiliary objective during the CTC fine-tuning of self-supervised acoustic encoders. The framework operates as follows:

Architecture: A standard, self-supervised pre-trained acoustic encoder (e.g., wav2vec 2.0 or WavLM) processes input waveforms to generate frame-based representations. These representations are shared for both the primary ASR task and the auxiliary contrastive task.
Primary Objective: The model is trained using the standard CTC loss ( $L_{CTC}$ ) to map speech to transcripts.
Auxiliary Objective (SupCon):
- Representation Aggregation: Frame-based encoder outputs are aggregated via masked mean pooling into fixed-dimensional utterance-based representations ( $u_i$ ) to exclude padded frames.
- Projection: These representations are passed through a lightweight projection head to generate normalized embeddings ( $z_i$ ).
- Contrastive Pairs: Positive pairs are defined based on transcript identity ( $c_i$ ). Utterances sharing the same transcript but spoken by different speakers (and thus potentially different accents) are treated as positive pairs. No explicit accent labels or additional annotations are required; the transcript itself serves as the supervision signal.
- Loss Function: A supervised contrastive loss ( $L_{SupCon}$ ) is applied to minimize the distance between embeddings of the same transcript and maximize the distance between embeddings of different transcripts.
Combined Training: The total loss is a weighted sum: $L = L_{CTC} + \lambda_t L_{SupCon}$ . A loss scheduling strategy is employed where the contrastive weight $\lambda_t$ is gradually increased during early fine-tuning to stabilize optimization.
Inference: The projection head and contrastive loss are discarded during inference; the system relies solely on the CTC branch for decoding.

Main Contributions

Novel Regularization Strategy: The introduction of SupCon as an auxiliary loss for fine-tuning CTC-based ASR systems, offering a model-agnostic method to improve accent robustness without modifying the underlying encoder architecture or requiring explicit accent monitoring.
State-of-the-Art Performance: Demonstration that a standard CTC pipeline augmented with SupCon achieves state-of-the-art results on the L2-ARCTIC benchmark for multi-accent speech recognition.
Geometric Analysis: A systematic analysis using intra-transcript cosine dispersion to demonstrate how contrastive regularization alters the geometry of encoder representations. The study shows that SupCon fosters more compact and stable representation clusters for identical linguistic content across different speakers and accents.

Experimental Results
Experiments were conducted on the L2-ARCTIC benchmark, which includes English speech from non-native speakers across six L1 backgrounds (Arabic, Mandarin, Hindi, Korean, Spanish, Vietnamese). Evaluation was performed under two settings: Unseen-Transcript (UT) and Unseen-Accent (UA).

Performance Improvements: The proposed method consistently reduced the Word Error Rate (WER) across multiple pre-trained encoders (wav2vec 2.0 and WavLM in Base and Large variants).
- Under the Unseen-Accent (UA) setting, the most significant improvements were observed. For the wav2vec 2.0 Large model, WER decreased from 9.98% (CTC only) to 7.41% (with SupCon), representing a relative reduction of 25.8%.
- Under the Unseen-Transcript (UT) setting, the wav2vec 2.0 Large model recorded a reduction from 10.47% to 9.14% (12.7% relative improvement).
Ablation Studies: The benefits of SupCon were consistent across various model sizes and decoding strategies (greedy vs. 4-gram LM decoding). Gains were generally more pronounced for wav2vec 2.0 than for WavLM, suggesting that the stronger pretraining objectives of WavLM may already provide implicit regularization.
Representation Analysis: Quantitative analysis of intra-transcript dispersion showed a relative reduction of 17% in mean cosine dispersion (from 0.0518 to 0.0430) when using SupCon. This indicates that the model learns more content-invariant representations and clusters utterances of the same transcript more tightly regardless of the speaker's accent.

Significance and Claims
The work claims that SupCon offers an effective and model-agnostic regularization strategy for improving accent robustness in ASR. Unlike accent-specific adaptation methods that lose performance on unseen accents, the proposed accent-invariant approach achieves its greatest gains in the Unseen-Accent setting. The authors emphasize that these improvements are achieved without architectural changes or explicit accent labels, but rather rely on the geometric regularization of encoder representations. The work suggests that fostering a compact representation geometry for identical transcripts is a viable path toward robust, generalizable multi-accent ASR systems.