Technical Summary: SpeechDx: A Multi-Task Benchmark for Clinical Speech AI

Problem Statement

Clinical speech AI has advanced through isolated, condition-specific studies, resulting in a fragmented field where models are trained and evaluated on individual datasets under inconsistent protocols. This fragmentation hinders the comparison of results and the assessment of generalization. Current models often fail to generalize to unseen data or different conditions because they learn spurious correlations from confounding factors (e.g., recording conditions, demographics, hardware) rather than the underlying clinical signal. The field currently lacks a standardized evaluation framework to quantify progress, assess cross-dataset generalization, or identify robust modeling approaches capable of supporting real-world clinical deployment.

Methodology

The SpeechDx Benchmark

The authors introduce SpeechDx, a large-scale benchmark comprising 12 publicly available datasets and 27 downstream tasks spanning nine health and affective conditions. To organize these tasks meaningfully, SpeechDx adopts the framework proposed by Berisha and Liss, classifying conditions based on the stage of speech production they disrupt:

1. Conceptualization: Disruptions in forming communicative intent (e.g., depression, emotional dysregulation). Tasks include depression detection/severity and emotion recognition.
2. Formulation: Disruptions in encoding intent into linguistic structure (e.g., aphasia, Alzheimer's disease). Tasks include detecting aphasia and Alzheimer's severity.
3. Articulation: Disruptions in executing spoken output, divided into:
   • Neuromuscular: Imprecise motor control (e.g., dysarthria, Parkinson's disease, disfluency).
   • Phonatory/Respiratory: Disruptions in airflow and vocal fold vibration (e.g., COVID-19, vocal pathologies, respiratory symptoms).

Evaluation Protocol

The benchmark evaluates 12 state-of-the-art audio and speech encoders representing different representation learning paradigms:

• General Speech Models: wav2vec 2.0, HuBERT, WavLM, MMS, Qwen3-TTS-Tokenizer, Whisper.
• General Audio Models: AudioMAE, WavJEPA, AST, CLAP.
• Domain-Specific Models: emotion2vec+ (emotional speech), OPERA (respiratory audio).

Probing Strategy: The authors employ a linear probing protocol where encoder weights are frozen, and a single linear layer is trained on top of the mean-pooled encoder output. This approach is computationally efficient and reduces overfitting risks associated with small clinical datasets.

• Metrics: Area Under the Curve (AUC) for classification and Mean Absolute Error (MAE) for regression. Performance is also aggregated using Mean Reciprocal Rank (MRR).
• Generalization Tests:
  1. Within-dataset evaluation: Standard train/validation/test splits or 5-fold cross-validation.
  2. Zero-shot cross-condition transfer: A linear probe trained on a source dataset is evaluated directly on a disjoint target dataset without exposure to target data. This tests whether representations capture shared clinical mechanisms versus dataset-specific artifacts.

Key Results

1. Task Difficulty and Speech Production Stages

The speech-production taxonomy serves as a meaningful predictor of task difficulty:

• Conceptualization tasks (e.g., depression detection) are consistently difficult, with AUCs ranging between 0.40 and 0.65 across models.
• Formulation and Neuromuscular Articulation tasks are more tractable. Aphasia detection reached an AUC of 0.97, and top models exceeded 0.82 on neuromuscular classification tasks.
• Phonatory/Respiratory tasks showed modest performance (best AUC ~0.79 for COVID-19 detection) despite theoretical direct measurability. The authors attribute this to heterogeneous recording conditions and label noise in crowdsourced respiratory datasets, which obscure the clinical signal.

2. Model Performance

• No Universal Generalization: No single encoder achieves uniformly strong performance across all conditions.
• Large-Scale Models: Large-scale general-purpose speech models (Whisper, Qwen3-TTS-Tokenizer, WavLM) provide the strongest overall baselines (highest aggregate MRR).
• Domain-Specific Limitations: Domain-specific pretraining yields selective gains but fails to generalize. For instance, emotion2vec+ dominates emotion tasks but performs poorly on depression detection (a clinically proximate conceptualization task). Similarly, OPERA, pretrained on respiratory audio, performs weakly on respiratory tasks but better on formulation tasks.

3. Zero-Shot Transfer Analysis

• Within-Category Transfer: Transfer often approaches within-dataset performance (e.g., a probe trained on DementiaBank achieving 0.94 AUC on aphasia detection). However, phonatory/respiratory tasks remain the most resistant to cross-dataset generalization.
• Cross-Category Asymmetry: Transfer reveals a clear asymmetry. Representations trained on phonatory/respiratory data transfer effectively to conceptualization and formulation tasks (AUCs up to 0.88), but the reverse direction does not exceed 0.60.
• Model Ranking Discrepancy: Models that excel in within-dataset settings do not necessarily lead in zero-shot transfer. For example, Qwen3 leads within-dataset phonatory tasks but fails to transfer across categories, while emotion2vec+ leads cross-category transfer despite mid-tier within-dataset performance.

Significance and Claims

The paper claims that SpeechDx establishes the first shared evaluation framework specifically designed to advance clinical speech AI. Its primary significance lies in:

1. Standardization: Providing a reproducible baseline for comparing performance across diverse clinical domains and conditions.
2. Identifying Gaps: Demonstrating that current representations do not generalize reliably across the clinical speech landscape, highlighting a critical need for models that maintain performance across regimes rather than excelling in single datasets.
3. Catalyzing Progress: Serving as a foundation for developing general-purpose health audio representations that can transfer across clinical tasks and populations, moving the field toward real-world clinical deployment.

The authors conclude that while large-scale speech models currently offer the most robust baselines, the field must address the lack of reliable cross-condition generalization to realize the full diagnostic potential of speech-based health assessment.