⚡ electrical engineering

FiLM-Based Speaker Conditioning of a SpeechLLM for Pathological Speech Recognition

This paper proposes a parameter-efficient approach using Feature-wise Linear Modulation (FiLM) to condition a frozen SpeechLLM on pathological speakers via x-vectors, demonstrating that this method achieves competitive automatic speech recognition performance on Spanish and English pathological data while preserving the model's general conversational capabilities.

Original authors: Fernando López, Santosh Kesiraju, Jordi Luque

Published 2026-06-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Fernando López, Santosh Kesiraju, Jordi Luque

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot librarian who is incredibly good at reading standard, clear books. This robot is so talented that it can transcribe normal speech with almost no mistakes. However, when someone with a neurological condition (like Parkinson's or ALS) tries to speak, their voice might be shaky, quiet, or slurred. The robot gets confused, just like a person trying to read a book written in messy, shaky handwriting.

The researchers in this paper wanted to help this robot understand these specific voices without having to retrain the robot from scratch or risk making it forget how to read the clear books.

Here is the simple breakdown of what they did:

1. The Problem: One Size Doesn't Fit All

Standard speech recognition systems are trained on "healthy" voices. When a person with a speech disorder speaks, the robot struggles because the sound patterns are different. Usually, to fix this, engineers would "fine-tune" the robot's brain. But this is risky: if you tweak the brain too much to understand one specific type of messy voice, the robot might forget how to understand clear voices. It's like teaching a chef to make a specific spicy dish so well that they forget how to make a simple salad.

2. The Solution: The "Smart Glasses" (FiLM)

Instead of rewriting the robot's entire brain, the researchers gave it a pair of smart glasses.

The Robot (The Base Model): They kept the main robot frozen and untouched. It stays exactly as smart as it was before.
The Glasses (FiLM): They added a small, new layer called "FiLM" (Feature-wise Linear Modulation). Think of this as a set of glasses that the robot puts on only when it hears a specific person.
How it works: Before the robot listens to a sentence, a small detector figures out "Who is speaking?" (using a digital voice print called an x-vector). If the speaker has a speech disorder, the glasses adjust the robot's internal hearing to match that specific person's voice. If the speaker is healthy, the glasses turn off (become clear), and the robot listens normally.

This way, the robot can adapt to a specific person's unique voice without changing its core knowledge.

3. The Test: Can it still do other things?

The researchers didn't just test if the robot could transcribe words. They also asked it questions about the speaker, like "Is the speaker male or female?" or "How old are they?"

The Goal: They wanted to make sure that by giving the robot these "glasses" to understand messy speech, they didn't break its ability to answer general questions.
The Result: The robot with the "glasses" got better at understanding the messy speech. Crucially, it also got better at guessing the speaker's gender, even though it wasn't explicitly trained to do that. It seems that by focusing on the speaker's unique voice, the robot became more sensitive to those details.

4. The Catch: It needs a little help

While the "glasses" method worked well, the researchers found that the robot's initial guesses were sometimes a bit "noisy" (like hearing a word but not being 100% sure of the spelling). They had to use a simple "spell-checker" (post-processing) to clean up the final text.

Comparison: Other methods that tried to retrain the whole robot's brain made fewer mistakes initially, but they risked making the robot forget how to handle normal voices. The "glasses" method kept the robot safe and flexible, even if it needed a little extra cleaning at the end.

The Bottom Line

The paper shows that you can help a super-smart speech robot understand difficult, pathological voices by giving it a lightweight, adjustable "adapter" (the FiLM glasses) rather than rebuilding its brain.

It works: It improves understanding of disordered speech.
It's safe: It doesn't ruin the robot's ability to understand normal speech.
It's efficient: It only changes a tiny fraction of the robot's brain (about 1.6%), making it a very efficient way to adapt technology for people with speech disorders.

In short, they found a way to give the robot a "personal translator" for specific voices without breaking its original brain.

Technical Summary: FiLM-Based Speaker Conditioning for Pathological Speech Recognition

Problem Statement
Automatic Speech Recognition (ASR) systems have achieved high accuracy on standard speech but struggle significantly with pathological speech, particularly from individuals with neuromotor disorders like Amyotrophic Lateral Sclerosis (ALS) and Parkinson's Disease (PD). These conditions cause dysarthria, leading to articulatory clarity issues, prosodic alterations, and high inter-speaker variability. This creates an acoustic mismatch with the normative data used to train modern ASR systems. While domain adaptation via fine-tuning is a standard solution, pathological corpora (e.g., TORGO, NeuroVoz) are often small and lack speaker diversity. This scarcity makes data-hungry fine-tuning strategies prone to overfitting and catastrophic forgetting, where the model loses its ability to recognize standard speech. Furthermore, the emergence of Speech Large Language Models (SpeechLLMs), which combine audio encoders with LLMs for tasks beyond transcription (e.g., instruction following), necessitates adaptation strategies that preserve the base model's general knowledge and weights.

Methodology
The authors propose a parameter-efficient adaptation strategy that conditions a frozen SpeechLLM encoder using Feature-wise Linear Modulation (FiLM) driven by speaker information. The core architecture involves:

Frozen Base Model: The entire SpeechLLM (specifically Voxtral-Mini, comprising a Whisper-large-v3 encoder and a Ministral-3B LLM decoder) remains frozen. No base model weights are updated.
Speaker Embedding Extraction: A pretrained x-vector model (SiAmResNet34) extracts speaker embeddings from raw audio. Crucially, for normative (healthy) utterances, the embedding is zeroed out. This ensures the adaptation mechanism does not interfere with healthy speech processing.
FiLM Injection: At every transformer layer of the frozen encoder, a dedicated two-layer Multi-Layer Perceptron (MLP) generates affine modulation parameters ( $\gamma$ $γ$ and $\beta$ $β$ ) and a scalar gate ( $\alpha$ $α$ ) based on the speaker embedding.
- The modulation is applied as a learned residual: $\tilde{H}_\ell = H_\ell + \alpha_\ell ((\gamma_\ell - 1) \odot H_\ell + \beta_\ell)$ .
- The gate $\alpha$ is initialized near zero (via a bias of -2) and trained to open progressively, ensuring the model starts as an identity transform.
Training Scope: Only the FiLM generators and the speaker embedding extractor are trained. For pathological speech, the system adapts internal representations; for healthy speech, the zeroed embedding forces the FiLM layers to act as identity transformations, preserving the original model behavior.

Key Contributions

FiLM-Based Adaptation: The paper introduces a method to condition a frozen SpeechLLM on pathological speech without modifying base weights, thereby preserving generalization to standard speech.
Systematic Comparison: The study benchmarks four adaptation strategies—Full Fine-Tuning (FFT), Encoder-only Fine-Tuning (EFT), LoRA (Full and Encoder-only), and the proposed Speaker Conditioning—across low-resource pathological domains.
Paralinguistic Evaluation: Beyond Word Error Rate (WER), the authors evaluate whether adaptation degrades the model's ability to answer speech-related questions (Multiple Choice Question Answering or MCQA) regarding speaker attributes (sex and age), a critical metric for SpeechLLMs.

Experimental Results
The approach was evaluated on the TORGO (English) and NeuroVoz (Spanish) datasets.

ASR Performance (WER):
- Fine-Tuning: FFT and F-LoRA achieved the lowest WERs, significantly outperforming the base model (e.g., reducing TORGO WER from ~25% to ~11-12%).
- Speaker Conditioning: The FiLM-based method showed marginal raw improvement on TORGO (25.15% to 23.24%) but achieved a competitive 16.36% WER when combined with rule-based post-processing. This suggests the method captures acoustic content well but produces noisier hypotheses than fully fine-tuned models.
- NeuroVoz: All methods showed slight improvements, with F-LoRA performing best (4.07%).
MCQA Performance:
- Base Model: The unmodified model struggled with age prediction but performed above random chance on sex prediction (53.5%).
- Fine-Tuning: FFT and EFT improved sex prediction accuracy (up to 64.9%), suggesting adaptation sharpened speaker feature sensitivity. However, F-LoRA collapsed to near-random performance (8.4%), indicating severe degradation in instruction-following capabilities.
- Speaker Conditioning: The proposed method achieved competitive sex accuracy (60.7%), trailing only EFT by 4.2 percentage points while updating only ~1.6% of the total parameters. Crucially, it maintained the base model's behavior for healthy speech by design.

Significance and Claims
The paper claims that FiLM-based speaker conditioning is a viable, parameter-efficient alternative to traditional fine-tuning for pathological speech recognition. Its primary significance lies in its ability to adapt to individual pathological acoustic profiles while strictly preserving the base model's weights and general knowledge. This is particularly critical for SpeechLLMs, where maintaining the ability to perform non-transcription tasks (like answering questions) is as important as transcription accuracy. The authors conclude that while fully fine-tuned models may yield slightly better raw WER, the speaker-conditioned approach offers a favorable trade-off between efficiency, generalization, and the preservation of broader audio understanding capabilities, provided that post-processing is utilized to refine the output.