Adapting Foundation ASR Models to Dysarthric Speech: A Case Study

Technical Summary: Adapting Foundation ASR Models to Dysarthric Speech

Problem Statement
Automatic Speech Recognition (ASR) systems, particularly large foundation models, have achieved near-human performance on standard speech benchmarks. However, they perform poorly when confronted with dysarthric speech, rendering them ineffective for users with motor speech disorders. This study focuses on a specific user affected by Progressive Multifocal Leukoencephalopathy (PML) caused by the JC-virus, resulting in significant dysarthria. Standard ASR models failed completely on this user's speech, exhibiting a baseline Word Error Rate (WER) of 128.4%, which effectively made off-the-shelf technology unusable for everyday communication.

Methodology
The authors developed a personalized ASR pipeline centered on adapting foundation models to speaker-specific data. The methodology involved three primary components:

1. Data Collection:

   • Initial Dataset: Using the TEQST tool, the user recorded 92 hours of read speech from home. This dataset was split into training (89.8 hours), development (1.1 hours), and test (1.1 hours) sets. The text content was drawn from the fairy tale domain.
   • Continuous Data Collection: A mobile application was deployed for real-world use. The user could correct transcription errors in real-time. These corrections, paired with the original audio, generated an additional 8.8 hours of high-quality, speaker-specific training data.
   • Total Data: The final dataset comprised approximately 100.8 hours of dysarthric speech.

2. Model Adaptation Strategies:

   • Foundation Models: The study compared two foundation models: Whisper (specifically whisper-large-v3) and Qwen3-ASR-1.7B.
   • Adaptation Techniques: The authors evaluated full fine-tuning against Parameter-Efficient Fine-Tuning (PEFT) using Low-Rank Adaptation (LoRA).
   • Experimental Design: Models were fine-tuned with varying amounts of adaptation data (1.4 hours, 22.5 hours, and the full dataset) to analyze the relationship between data volume and performance.

3. System Deployment:

   • The adapted model was deployed via a mobile application (initially Android, later iOS) designed specifically for users with limited fine motor control.
   • The system utilizes a server-side architecture (hosted via faster-whisper on CTranslate2) to offload computational demands, ensuring fast real-time inference.
   • The interface features a high-contrast, single-button design with multiple recording modes (tap-to-start, press-and-hold, single-tap) to accommodate motor impairments. It includes a feedback loop where user corrections are sent back to the server to refine the model.

Key Results
The experiments demonstrated that personalized fine-tuning significantly improves ASR performance for dysarthric speech:

• Baseline vs. Fine-Tuned: Starting from a baseline WER of 128.4%, fine-tuning Whisper with only 1.4 hours of data reduced the WER to 15.8%.
• Data Scaling: Performance improved steadily with more data. Using 22.5 hours of data yielded a 10.7% WER. The best result was achieved using the full dataset (including user corrections), reaching a WER of 9.7%.
• Fine-Tuning vs. LoRA: Full fine-tuning consistently outperformed LoRA adaptation. LoRA performed substantially worse (15% to 39% relative increase in error) for the same data amounts. The authors attribute this to the significant acoustic mismatch between standard speech and dysarthric speech, which required the full model capacity to learn effectively.
• Model Comparison: Whisper proved to be a stronger foundation model than Qwen3-ASR for this specific user. Qwen3-ASR achieved a WER between 14% and 16% even with full data, whereas Whisper reached 9.7%.

Significance and Claims
The paper claims that personalized fine-tuning can transform a foundation ASR model that fails on dysarthric speech into a system accurate and robust enough for practical, real-world deployment. The study highlights that:

• Effectiveness: A substantial reduction in WER (from >128% to <10%) is achievable with a relatively small amount of adaptation data (as little as 1.4 hours for significant gains, ~22.5 hours for a favorable trade-off).
• Practical Utility: The deployed mobile application serves as a functional communication aid, enabling natural face-to-face interaction and providing a fallback for moments when speech alone is insufficient.
• User Impact: Beyond technical metrics, the user reported increased confidence in speaking and an improvement in quality of life, noting that the ability to use the app as a fallback encouraged more frequent speech attempts.

Limitations and Future Work
The authors note that the current system relies on a remote server, requiring an internet connection. Local on-device deployment remains challenging due to computational constraints, though quantization or smaller models might be necessary for future iterations. Furthermore, the system is currently personalized to a single speaker; the data requirements for generalizing this approach to other speakers with different dysarthria profiles remain an open question. Scaling to multiple speakers simultaneously via a single adaptive model is identified as a direction for future research.