📄 medicine

An Automated Listener-Effort Measure for ALS Generalizes across Diverse Platforms and Protocols

This study demonstrates that a machine learning model utilizing speaking rate and transcription confidence can accurately and reliably predict Speech-Language Pathologist-rated Listener Effort across diverse ALS datasets, offering a scalable and efficient solution for remote speech monitoring and clinical trial endpoints.

Original authors: Esteban G. Roitberg, Marcos A. Trevisan, Julian Peller, Federico Sevlever, Felipe Aguirre, Diego E. Shalom, Alan Taitz, Gaston Bujia, Joel Schwartz, Jason Osik, Ryan Shewcraft, Anahita Kyani, Guofa Sh

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Esteban G. Roitberg, Marcos A. Trevisan, Julian Peller, Federico Sevlever, Felipe Aguirre, Diego E. Shalom, Alan Taitz, Gaston Bujia, Joel Schwartz, Jason Osik, Ryan Shewcraft, Anahita Kyani, Guofa Shou, Nader Naghavi, Henk Schuring, Vinni Bijanki, Lindsay Heyd, Brittney Harkey, Taylor K. Stimpson, Marianne Chase, Hong Yu, Alexander V. Sherman, Jeremy Shefner, Sabrina Paganoni, Merit E. Cudkowicz, Eric Macklin, Jordan R. Green, Oren Levy, Ernest Fraenkel, Lyle W. Ostrow, Indu Navar Bingham, James D. Berry

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to listen to a friend who is struggling to speak because of a disease called ALS. Sometimes their words are clear, but other times they sound like a radio station caught in a heavy storm of static. To understand them, your brain has to work overtime, squinting your ears and focusing hard. In the medical world, doctors call this "Listener Effort." It's a score from 0 to 100 that measures how much mental energy a person needs just to understand what the patient is saying.

For years, getting this score has been like hiring a team of super-listeners (called Speech-Language Pathologists) to sit down, put on headphones, and manually grade every single sentence a patient speaks. It's accurate, but it's slow, expensive, and hard to do for everyone.

The Big Idea: A Digital "Ear" That Learns
This paper introduces a new, automated way to guess that Listener Effort score using a computer model. Think of it as teaching a robot to listen to the "static" in the voice and guess how hard a human would have to work to understand it. The researchers didn't build a complex, black-box AI; they built a simple, transparent tool that looks at just two clues:

Speaking Rate: How fast is the person talking?
Whisper Confidence: This is a score from a famous speech-recognition program called "Whisper." It tells us how sure the computer is that it correctly transcribed the words. If the computer is confused, it means the speech is hard to understand, which usually means the listener has to work harder.

The Great Test: Does It Work Everywhere?
The real magic of this paper isn't just that the robot can guess the score; it's that the robot didn't just learn to guess for one group of people. The researchers tested their model on three completely different groups of patients, collected in three different ways:

The Austen Study: 105 people recorded on one platform.
The HEALEY Trial: 266 people recorded on a different platform, with a different set of sentences, and generally sicker patients.
The Radcliff Study: 57 people recorded on yet another setup.

These groups were like different neighborhoods with different accents, different recording equipment, and different levels of illness. Usually, a computer model trained in one neighborhood fails miserably when you take it to another. But here, the model held its ground.

The Results: A Strong Match
When the researchers compared the robot's guesses to the human experts' scores, the match was surprisingly strong.

For the HEALEY group, the model's predictions matched the human scores with a correlation of 0.81 ± 0.02.
For the Radcliff group, the match was 0.82 ± 0.03.

To put that in perspective, if you were betting on whether the robot would get the score right, it would win most of the time. Even more impressively, when they watched how the scores changed over time (longitudinal analysis), the robot tracked the patients' decline just as well as the humans did. The "slopes" of their progress (how fast they were getting worse) were highly correlated: 0.80 for the HEALEY group and 0.93 for the Radcliff group.

What the Paper Rules Out (and What It Doesn't)
It's important to know what this tool isn't.

It is not a magic fix for the worst cases. The paper explicitly notes that the model struggles with the most severe cases. About 10% of participants with the highest Listener Effort scores (between 80 and 100) were actually filtered out because the computer couldn't transcribe their speech at all. The model suggests it works best on speech that is still somewhat decipherable.
It is not a universal translator yet. The paper rules out the idea that this works for everyone right now. The data was almost entirely from white, native English speakers. The authors explicitly state that the model has not been tested on non-English speakers or diverse accents, and they warn that speech recognition systems often struggle with different dialects.
It is not a replacement for humans (yet). The paper argues that while the robot is great for scaling up and saving time, human experts are still essential for validation. The model is a tool to extend human reach, not to erase the need for human judgment.

How Sure Are We?
The authors are confident in their numbers but careful in their claims. They didn't just simulate this on a computer; they measured it on real data from 1,656 recordings in the Austen study, 4,050 in HEALEY, and 1,488 in Radcliff. They used rigorous statistical methods, like "bootstrapping" (a fancy way of resampling data to check for stability), to show that their results weren't just luck.

They suggest that this automated measure could be a "sensitive endpoint" for clinical trials, meaning it might help drug companies see if a medicine is working faster than current methods. However, they stop short of calling it a solved problem. They suggest that future work needs to include spontaneous, natural speech (not just reading sentences) and diverse populations to make the tool truly robust.

In short, this paper suggests that a simple computer model, using just speed and transcription confidence, can reliably guess how hard it is to listen to an ALS patient across many different settings. It's a promising step toward making speech monitoring scalable and practical, but it's a tool that still needs to be tested on more diverse voices and used alongside human experts.

Technical Summary: An Automated Listener-Effort Measure for ALS

Problem Statement
Monitoring the progression of Amyotrophic Lateral Sclerosis (ALS) through speech is complicated by the multifaceted nature of dysarthria, which affects multiple speech subsystems. While Listener Effort (LE)—a subjective rating from 0 to 100 quantifying the cognitive effort required to understand dysarthric speech—is a clinically meaningful metric strongly correlated with disease progression, its assessment relies on manual evaluation by Speech-Language Pathologists (SLPs). This process is labor-intensive, costly, and difficult to scale for remote monitoring or large clinical trials. Furthermore, machine learning models trained on single datasets often fail to generalize to new data collected under different protocols, platforms, or demographic conditions, necessitating rigorous external validation.

Methodology
The authors developed and evaluated a machine learning model designed to predict SLP-rated Listener Effort using remotely collected speech recordings. The study utilized three independent datasets: the Austen Study (training set), the HEALEY ALS Platform Trial, and the Radcliff Study. These datasets differed significantly in recording platforms (Modality.ai vs. AuralAnalytics), speech protocols, session frequencies, and participant demographics.

Model Architecture: The predictive model is a linear regression algorithm utilizing two acoustic features extracted from each speech sample:
1. Speaking Rate: Calculated using the Whisper Large-v2 model.
2. Whisper Confidence: A transcription confidence score (0–1) derived from the Whisper Base model.
Ground Truth: The target variable was the LE score assigned by trained SLPs. SLP reliability was established through intra-rater and inter-rater Intraclass Correlation Coefficients (ICCs), which exceeded 0.90 across all studies.
Validation Strategy: The study adhered to the TRIPOD+AI framework.
- Internal Validation: The model was initially trained on the Austen dataset using 5-fold cross-validation.
- External Validation: The Austen-trained model was tested on the HEALEY and Radcliff datasets without retraining.
- Model Update: A second model was trained on a combined dataset of all three studies using shuffled cross-validation to assess generalizability improvements.
Quality Control: Speech samples underwent filtering based on SLP notes (excluding noise/errors) and transcription accuracy (requiring ≥20% overlap with the original sentence). This filter excluded recordings with severe dysarthria (SLP-rated LE 80–100) where automatic transcription failed, introducing a known bias toward more transcribable speech.
Longitudinal Analysis: Linear Mixed-Effects Models (LMM) were employed to compare the slopes of disease progression (rate of change in LE over time) between SLP-rated and ML-predicted scores at both population and individual levels.

Key Results

External Validation Performance: The model trained on the Austen dataset demonstrated strong generalizability. It achieved an $R^2$ of $0.81 \pm 0.02$ on the HEALEY dataset and $0.82 \pm 0.03$ on the Radcliff dataset. Root Mean Square Error (RMSE) values remained below 10.
Model Update: Retraining the model on the combined dataset resulted in a trade-off: while performance on the original training set (Austen) slightly decreased, prediction accuracy (RMSE) improved on the HEALEY dataset, suggesting better balance across diverse populations.
Longitudinal Agreement: Longitudinal analyses revealed strong agreement between SLP-rated and ML-predicted progression slopes. The Concordance Correlation Coefficients ( $\rho_c$ ) for individual slopes were 0.80 for HEALEY and 0.93 for Radcliff. Population-level slope estimates showed no statistically significant differences between the two measures.
Data Variability: Bootstrap analysis indicated that performance deviations were negligible across the 0–90 LE range but showed greater dispersion in the 90–100 range, consistent with the scarcity of data in this severe dysarthria range due to quality filters.

Significance and Claims
The paper claims that a simple, interpretable machine learning model based on speaking rate and transcription confidence can reliably estimate Listener Effort across diverse, independently collected ALS cohorts. The study demonstrates that this automated measure:

Generalizes Robustly: It maintains high accuracy across different recording platforms, speech protocols, and participant characteristics, addressing a common failure point in clinical machine learning.
Scales Efficiently: By automating the LE assessment, the model offers a scalable alternative to manual SLP ratings, reducing the burden on clinical resources.
Tracks Progression: The model mirrors SLP-rated trajectories over time, providing a sensitive endpoint for clinical trials that can capture subtle changes in speech impairment.

The authors emphasize that while the model reduces reliance on manual ratings, SLP assessments remain essential for validation. They acknowledge limitations, including the exclusion of the most severe dysarthria cases due to transcription filters and the lack of validation in non-English speaking populations. The work positions ML-predicted LE not as a replacement for clinical judgment, but as a scalable tool to extend the utility of speech-based measures in ALS research and care.

Technical Summary: An Automated Listener-Effort Measure for ALS

More like this