💬 NLP

Debiased Automatic Speech Recognition for Dysarthric Speech via Sample Reweighting with Sample Affinity Test

This paper proposes Re-SAT, a novel sample reweighting approach that utilizes sample affinity tests to systematically measure and mitigate bias in automatic speech recognition systems, thereby improving performance for dysarthric speakers without degrading results for healthy speakers.

Original authors: Eungbeom Kim, Yunkee Chae, Jaeheon Sim, Kyogu Lee

Published 2026-06-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Eungbeom Kim, Yunkee Chae, Jaeheon Sim, Kyogu Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to understand human speech. Usually, you feed it thousands of hours of recordings from healthy people. The robot learns quickly because healthy speech is clear and consistent. However, when you ask this same robot to listen to people with dysarthria (a condition that makes speech slurred or difficult to understand), it often fails miserably.

The robot isn't "mean"; it's just lazy. It has learned to take shortcuts, focusing on the easy, clear voices and ignoring the difficult ones. This creates an unfair system where the robot works great for some people but is useless for others.

This paper introduces a new training method called Re-SAT (Sample Reweighting with Sample Affinity Test) to fix this imbalance. Here is how it works, using simple analogies:

The Problem: The "Average" Trap

Standard training (called ERM) is like a teacher who grades a class based on the average score of the whole room. If 90% of the students are geniuses and 10% are struggling, the teacher might think the class is doing fine overall. The struggling students get left behind because the teacher doesn't realize they need extra attention. In speech recognition, this means the system gets good at hearing healthy voices but terrible at hearing dysarthric voices.

The Solution: Re-SAT

The authors propose a smarter way to train the robot. Instead of treating every voice recording equally, Re-SAT acts like a discerning coach who decides which practice sessions are actually helpful and which are just noise.

The process happens in four steps:

1. Finding the "Struggling" Samples

First, the system looks at a batch of voice recordings and identifies the ones the robot is currently failing to understand. These are the "hard" samples.

The Catch: Just because a sample is hard (high error) doesn't mean it's helpful to focus on. Sometimes, a sample is just weird or broken (an "outlier"). If you focus too much on broken samples, the robot gets confused.

2. The "Affinity Test" (The Magic Filter)

This is the paper's big innovation. The system runs a quick, one-step "what-if" simulation.

The Analogy: Imagine you have a group of students. You ask, "If I spend 5 minutes teaching Student A right now, will it help Student B understand better?"
If teaching Student A helps the whole group (especially the struggling ones), that sample is a "Bias-Blocking" sample. It's a good teacher.
If teaching Student A makes the group worse or doesn't help, that sample is a "Bias-Accelerating" sample. It's a bad teacher.

The system uses this test to filter out the "bad teachers" (outliers) even if they are hard to understand. It only keeps the samples that genuinely help the robot learn to be fair.

3. Ranking and Reweighting

Once the system knows which samples are "good teachers" and which are "bad teachers," it sorts them.

It gives extra weight (more attention) to the "good teachers."
It gives less weight (less attention) to the "bad teachers."
Think of it like a volume knob: the robot turns up the volume on the helpful lessons and turns down the volume on the confusing ones.

4. Training with the New Weights

Finally, the robot trains using this new, balanced mix of lessons. It learns to pay attention to the difficult dysarthric voices without forgetting how to listen to healthy voices.

The Results: Fairness for Everyone

The researchers tested this on a dataset containing voices from people with varying levels of speech difficulty (from "very low" intelligibility to "high") and healthy speakers.

The Old Way (ERM): The robot was great at healthy voices but terrible at dysarthric voices.
The New Way (Re-SAT): The robot got significantly better at understanding dysarthric voices.
The Surprise: Unlike other methods that tried to fix the problem but accidentally made the robot worse at understanding healthy voices, Re-SAT improved the dysarthric voices without hurting the healthy ones.

In fact, the robot actually got slightly better at understanding healthy voices too, proving that by learning to handle the "hard" cases, it became a more robust listener overall.

Summary

The paper argues that to make speech recognition fair, we can't just throw more data at the problem. We need a smart filter (Re-SAT) that figures out which difficult examples are actually useful for learning and which are just distractions. By doing this, the system becomes inclusive, working well for everyone, regardless of how clear their speech is.

Technical Summary: Debiased Automatic Speech Recognition for Dysarthric Speech via Sample Reweighting with Sample Affinity Test

Problem Statement
Automatic Speech Recognition (ASR) systems based on deep learning are typically trained under Empirical Risk Minimization (ERM), which optimizes for average performance across all data samples. This approach often fails to account for performance disparities between different groups, such as healthy speakers versus those with dysarthria (a motor speech disorder). Consequently, ASR systems exhibit severe bias, performing significantly worse on dysarthric speech due to its low intelligibility and the tendency of deep learning models to fit "shortcuts" rather than robust features. While previous research has focused on improving overall dysarthric speech recognition, few studies have addressed the issue from a debiasing perspective to ensure group robustness without degrading performance on healthy speech. Furthermore, existing debiasing methods often rely on the assumption that samples with high loss (bias-conflicting samples) are inherently helpful for debiasing, an assumption that fails in real-world scenarios where outliers or unhelpful samples may also exhibit high loss.

Methodology: Re-SAT
The authors propose a novel approach called Sample Reweighting with Sample Affinity Test (Re-SAT) to mitigate bias in dysarthric speech recognition. Re-SAT operates under a group label-free environment and consists of four sequential components:

Bias-Conflicting Sample Estimation: The method initially identifies potential bias-conflicting samples within a mini-batch by selecting the $K$ samples with the largest loss values. Unlike static methods, Re-SAT dynamically re-estimates these samples throughout the training process to reflect the model's current state.
Sample Affinity Test (SAT): To distinguish between samples that genuinely help debiasing and those that merely have high loss, the authors introduce SAT. Inspired by task affinity, SAT measures the "sample affinity," defined as the training effect of a given sample on the loss of the estimated bias-conflicting samples.
- The method performs a single-step lookahead update on a given sample $x_i$ to obtain a temporary model $\theta^{LA}_{x_i}$ .
- It then calculates the loss shift on the bias-conflicting set $\{\hat{b}_1, ..., \hat{b}_K\}$ between the original model and the lookahead model.
- The affinity score is computed as the average reduction in loss on the bias-conflicting set. This allows the system to identify bias-blocking samples (which mitigate bias) versus bias-accelerating samples (which worsen bias), even if the latter have high initial loss.
Normalized Reweighting: Instead of directly mapping affinity scores to weights, Re-SAT normalizes them by ranking samples within the batch in descending order of affinity. A softmax-based function assigns weights based on these ranks, stabilizing the learning rate and preventing unwanted shifts.
Training: The model is trained using the reweighted samples, where the total loss is a weighted average favoring bias-blocking samples and penalizing bias-accelerating ones.

Key Contributions

First Debiasing Work for Dysarthric ASR: To the authors' knowledge, this is the first study to approach dysarthric speech recognition specifically through the lens of debiasing to improve group robustness between healthy and dysarthric speakers.
Sample Affinity Test (SAT): The paper introduces a novel metric to directly measure the "debiasing helpfulness" of a sample. This addresses the limitation of loss-based methods by filtering out unhelpful bias-accelerating samples that might otherwise be upweighted due to high loss.
Re-SAT Framework: A complete sample reweighting pipeline that operates without requiring explicit group annotations during training, making it suitable for real-world applications where group labels may be unavailable.

Experimental Results
The method was evaluated on the UASpeech corpus, using the Whisper-tiny model as the backbone. The dataset included 15 dysarthric speakers (categorized by intelligibility levels: Very Low, Low, Mid, High) and 13 healthy control speakers.

Performance on Dysarthric Speech: Re-SAT significantly reduced the Word Error Rate (WER) across all intelligibility groups compared to the ERM baseline. Specifically, it achieved relative WER reductions of 12.02% (Very Low), 11.10% (Low), 7.22% (Mid), and 13.06% (High).
Performance on Healthy Speech: Crucially, Re-SAT also improved performance on healthy speakers (reducing WER from 3.81% to 3.40%), demonstrating that the debiasing process did not come at the cost of general performance.
Comparison with Baselines: Re-SAT outperformed both ERM and the "Just Train Twice" (JTT) debiasing method. While JTT improved results for the "Very Low" group, it degraded performance for other groups and healthy speakers. Re-SAT provided consistent improvements across all groups.
Ablation Study: A variant called "Re-Loss," which used loss-based reweighting instead of SAT, performed better than ERM and JTT but was still outperformed by Re-SAT. This confirms the importance of the Sample Affinity Test in accurately identifying helpful samples.
Hyperparameter Sensitivity: The method showed stable improvements for $K$ (number of bias-conflicting samples) values of 2, 4, and 8. However, setting $K=16$ degraded performance, suggesting that a smaller, dynamically updated set of conflicting samples is more effective for accurate estimation.

Significance and Claims
The paper claims that Re-SAT offers a robust solution for creating fair and inclusive ASR systems for dysarthric speakers. By moving beyond simple loss-based reweighting to a mechanism that directly measures the debiasing utility of samples, the method successfully narrows the performance gap between healthy and dysarthric speakers without sacrificing overall accuracy. The authors position this work as a foundational step toward group-robust ASR, highlighting that their approach is the first to apply debiasing techniques specifically to the dysarthric speech domain. They suggest that future work could integrate Re-SAT with other dysarthric speech recognition systems, leveraging its versatile structure.