Technical Summary: AP-GRPO for Pathological Speech Reconstruction

1. Problem Statement

Pathological speech reconstruction (PSR) aims to recover a patient's intended textual content from acoustic recordings that are often distorted and linguistically fragmented due to neurodegenerative and neuromotor disorders (e.g., ALS, Parkinson's disease, cerebral palsy, dementia).

Current approaches face two primary limitations:

1. Uniform Degradation Assumption: Standard supervised fine-tuning (SFT) treats all token positions equally, failing to leverage the fact that pathological speech degradation is rarely uniform. Even in severe cases, some words or phrases remain intelligible ("audible anchors"), while others are corrupted.
2. Misaligned Reward Signals: Existing reinforcement learning (RL) methods, such as sentence-level GRPO using BERTScore, reward semantic fluency over faithfulness. They may reward fluent paraphrases (e.g., substituting "ache" for "pain") even when the patient clearly produced the latter, failing to distinguish faithful reconstruction from hallucination. Furthermore, many generative systems require paired data that is difficult to collect from progressive conditions.

2. Methodology: AP-GRPO

The authors propose Anchor-Gated Phonetic Group Relative Policy Optimization (AP-GRPO), a framework that aligns Speech Language Models (SLMs) with pathological speech signals without requiring reference transcripts during the RL phase. The method operates on the principle that reliable audible fragments can serve as structural boundaries to guide the reconstruction of corrupted regions.

Core Components

AP-GRPO utilizes two distinct reward components within a Group Relative Policy Optimization (GRPO) framework:

A. Audible Anchor Extraction and Matching

• Extraction: Using Whisper/Whisper-X, the system extracts timestamped "audible anchors" (words/phrases with high confidence, $c(m) > \tau$) from the pathological recording.
• Monotonic Matching: During RL steps, candidate reconstructions are scanned to match these anchors in temporal order.
• Inter-Anchor Spans: The text segments between matched anchors are defined as "inter-anchor spans," representing the corrupted regions requiring reconstruction.

B. Anchor-Gated Reward ($g_{anchor}$)
This component enforces the preservation of reliable audible evidence. It calculates a confidence-weighted coverage score:
$$g_{anchor}(y_j, A) = \epsilon_g + (1 - \epsilon_g) \frac{\sum c(m) \mathbb{1}[a(m) \text{ matched } y_j]}{\sum c(m)}$$
This reward discourages fluent but unfaithful outputs that contradict the reliable anchors.

C. Inter-Anchor Phonetic Alignment Reward ($R_{inter}$)
To address the corrupted regions, AP-GRPO evaluates whether the recovered text is phonetically supported by the corresponding speech span, rather than relying on semantic similarity.

1. Pathological Styling: The candidate text is converted into a phoneme sequence. Crucially, a Pathological Duration Predictor (based on a pretrained TTDS model) expands this sequence to account for slowed articulation, prolonged phones, and irregular pauses common in dysarthria.
2. Phoneme Posteriorgrams (PPG): The original speech span is converted into a PPG using a frozen wav2vec2 model.
3. Confusion Smoothing: To handle systematic articulatory deviations (e.g., /t/ vs. /d/), the PPG is smoothed using a phoneme confusion matrix, allowing clinically plausible confusions to receive non-zero support.
4. Soft-DTW Alignment: A Soft-DTW algorithm computes the alignment cost between the pathological-styled text phoneme path and the smoothed speech PPG. Lower costs (better alignment) yield higher rewards.

D. Optimization Objective
The total reward combines the anchor gate and the phonetic alignment:
$$R_j = \log g_{anchor} + R_{inter}$$
The policy is updated using GRPO, which optimizes relative advantages within a group of sampled candidates. An adaptive dual variable ($\mu$) enforces a constraint that the expected anchor coverage must remain above a threshold ($\alpha = 0.95$), dynamically balancing anchor preservation against inter-anchor recovery.

3. Key Contributions

1. First GRPO Framework for Pathological Speech: AP-GRPO is the first to apply Group Relative Policy Optimization to align Speech Language Models specifically for pathological speech reconstruction.
2. Novel Reward Architecture: It introduces a dual-reward system combining an anchor-gated reward (for structural integrity) and a pathology-aware phonetic alignment reward (for content fidelity). This includes specific adaptations like pathological duration modeling and phoneme confusion smoothing.
3. Interpretable Disease Profiles: The learned anchor constraint ($\mu$) automatically adapts to the difficulty of specific conditions, revealing interpretable profiles (e.g., severe articulatory degradation requires stronger anchor enforcement, while linguistically impaired conditions rely more on phonetic alignment).

4. Experimental Results

The method was evaluated on four datasets representing different conditions: ALS (TORGO), Cerebral Palsy (UASpeech), Dementia (ADReSSo), and Parkinson's (SJTU).

• Performance Gains: AP-GRPO achieved the lowest Word Error Rate (WER) and Character Error Rate (CER) across all datasets compared to SFT baselines, diffusion models (Diff-DSR), codec language models (Colm-DSR), and other RL methods.
  • Severe Cases (ALS): On the TORGO dataset, AP-GRPO reduced WER from ~0.75 (SFT baseline) to 0.29, transforming barely intelligible output into partially correct text suitable for clinical communication.
  • Mild Cases: Consistent improvements were observed across all conditions, with the method being backbone-agnostic (effective on both Qwen2.5-Omni and Audio-Flamingo 3).
• Hallucination Suppression: The method significantly reduced "hallucinated insertions" (fabricated content words). Decomposition analysis showed that while WER dropped sharply, Content-F1 improvements were driven by the reduction of content-word errors rather than just function word corrections.
• Ablation Studies:
  • Removing either the anchor gate or the phonetic reward degraded performance significantly, confirming their complementary nature.
  • Pathological adaptations (confusion smoothing and pathological styling) were critical; removing them caused WER to jump from ~0.29 to ~0.74.
  • Adaptive Constraints: The adaptive dual variable $\mu$ outperformed fixed constraints, dynamically balancing anchor preservation and content recovery.

5. Significance and Claims

The paper claims that AP-GRPO offers a practical path toward accessible communication for individuals with neurodegenerative speech disorders by:

• Crossing the Intelligibility Threshold: It transforms outputs from "unintelligible" (WER > 0.5) to "partially correct" (WER ~0.3), making them viable for downstream error correction and healthcare communication.
• Data Efficiency: It leverages the patient's own speech signal for alignment, reducing the reliance on large-scale paired reference transcripts during the RL phase.
• Clinical Relevance: The method's ability to automatically adapt to disease-specific impairment profiles (via the learned $\mu$) suggests it can be deployed without per-disease tuning, aligning with clinical needs for robust, generalizable tools.

The authors note limitations, including that the method currently focuses on transcript reconstruction rather than audio restoration, and its performance is dependent on the quality of the initial anchor extraction and the availability of disease-specific duration priors.