💬 NLP

Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR

This paper demonstrates that incorporating a small amount of paired speech-text data (as little as 10% of the target domain) into the mixed-batching adaptation strategy effectively bridges the modality gap in LLM-based ASR, achieving performance comparable to full-data fine-tuning while leveraging the efficiency of text-only adaptation.

Original authors: Thibault Bañeras-Roux, Sergio Burdisso, Esaú Villatoro-Tello, Dairazalia Sánchez-Cortés, Shiran Liu, Severin Baroudi, Shashi Kumar, Hasindri Watawana, Manjunath K E, Kadri Hacioglu, Petr Motlicek, And

Published 2026-04-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Thibault Bañeras-Roux, Sergio Burdisso, Esaú Villatoro-Tello, Dairazalia Sánchez-Cortés, Shiran Liu, Severin Baroudi, Shashi Kumar, Hasindri Watawana, Manjunath K E, Kadri Hacioglu, Petr Motlicek, Andreas Stolcke

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, world-class translator named LLM (Large Language Model). This translator is amazing at understanding written text in any language, from poetry to legal contracts. However, this translator has never actually heard a human voice before; they only know what words look like on a page.

Now, you want to teach this translator to listen to speech and turn it into text (this is called Automatic Speech Recognition, or ASR). To do this, you hire a "bridge builder" (a projection module) who translates the messy, noisy sound waves of a human voice into the clean, structured language the translator understands.

The Problem: The "Silent Library" Trap

Usually, to teach this system a new topic (like banking or farming), you need a huge library of recordings (audio) paired with their transcripts (text). But recordings are expensive and hard to get. Text, on the other hand, is everywhere.

So, people tried a shortcut: Text-Only Adaptation. They just fed the translator new text about banking.

The Result: The translator got really good at banking words, but it forgot how to listen to voices.
The Analogy: Imagine teaching a chef to cook a new dish by only giving them a recipe book, but never letting them taste the ingredients or see the stove. They know the names of the ingredients, but if you hand them a real, messy, hot pan, they don't know how to handle it. The "voice" (audio) and the "text" (words) stopped matching up. This is called the Modality Gap.

The Solution: The "Mixed Batch" Strategy

The researchers asked: "What if we give the translator just a tiny taste of the real audio, mixed in with all that easy text?"

They tested three approaches:

Text Only: Just the recipe book (fails to handle the real voice).
Full Audio: Only real recordings (expensive and hard to get).
Mixed Batching (The Winner): A cocktail party where 90% of the time, the translator reads text, but 10% of the time, they listen to a real voice recording.

How It Works (The Magic Recipe)

The researchers didn't just mix audio and text randomly. They created a clever "training camp" with three types of exercises:

Real Audio: Listening to actual banking conversations.
"Noisy" Text: They took clean text and intentionally scrambled it (like adding typos or random noise) to mimic how a voice sounds when it's converted to text by a machine. This teaches the translator to be flexible.
Old Knowledge: They kept feeding the translator old, familiar data so they didn't forget how to speak the original language.

The Surprising Results

The study found something incredible: You don't need much audio at all.

The "10% Rule": Using just 10% of the available audio (less than 4 hours of recording) mixed with a mountain of text worked better than using 100% of the audio.
Why? The text taught the translator the vocabulary of the new field (e.g., "mortgage," "interest rate"). The tiny bit of audio taught the translator how to listen to the specific accent and noise of that field. Together, they created a perfect match.
No Amnesia: When they used only audio, the translator forgot how to handle the old topics (Catastrophic Forgetting). But with the "Mixed Batch," the translator learned the new skill without forgetting the old ones.

The Big Picture

Think of it like learning to drive a new car.

Text-only is like reading the manual: You know the rules, but you can't drive.
Full Audio is like driving the car for 1,000 hours: You get good, but it takes forever and costs a fortune.
Mixed Batching is like reading the manual and taking a few short, focused driving lessons. You get the best of both worlds: you know the theory, and you have just enough practice to handle the real road.

In short: If you want to teach an AI to understand a new topic via voice, don't worry if you don't have thousands of hours of recordings. Just give it a massive amount of text and a tiny sprinkle of audio, and it will learn faster and better than if you tried to force it to learn from audio alone.

1. Problem Statement

The integration of Large Language Models (LLMs) into Automatic Speech Recognition (ASR) systems has created a new architectural paradigm: a pretrained speech encoder connected to a pretrained LLM via a trainable projection module. While this allows for efficient domain adaptation using text-only data, it introduces a critical modality gap.

The Modality Gap: When an LLM is fine-tuned exclusively on clean, in-domain text transcripts, it loses exposure to the "noisy" or imperfect representations generated by the speech projector. Consequently, the model's ability to align acoustic features with text degrades, leading to suboptimal performance compared to models trained on paired speech-text data.
The Resource Bottleneck: In specialized domains (e.g., healthcare, finance, agriculture), labeled audio data is scarce and expensive to collect, whereas domain-specific text is often abundant. Standard text-only adaptation fails to bridge the modality gap, while full speech-text adaptation is often infeasible due to data scarcity.

2. Methodology

The authors propose a Mixed Batching (MB) strategy to mitigate the modality gap while leveraging abundant text and limited audio.

Architecture

The system follows the SLAM-ASR framework:

Speech Encoder: A frozen, pretrained model (WavLM-Large) extracts acoustic features.
Speech Projector: A trainable linear layer maps acoustic features to the LLM embedding space.
LLM: A pretrained LLM (Llama-3.2-3B-Instruct) generates transcripts autoregressively.
Adaptation Mechanism: During adaptation, the encoder and projector are frozen; only LoRA adapters inserted into the LLM are updated.

The Mixed Batching Strategy

Instead of training on pure text or pure speech-text pairs, the training batch is composed of a mixture of five distinct data segments to balance domain specialization and modality alignment:

Paired Source Speech-Text: Preserves the original audio-text alignment learned during base training.
Projector-Induced Noisy Tokens: Discrete sequences obtained by mapping speech projections to nearest LLM tokens (simulating the projector's output) paired with clean transcripts.
Synthetically Corrupted Source Transcripts: Clean transcripts with random character-level perturbations (simulating projector irregularities) paired with the original clean text.
Paired Target Speech-Text: Real target-domain audio and transcripts to induce acoustic adaptation.
Corrupted Target Transcripts: Perturbed target-domain text paired with clean text for domain-specific linguistic adaptation.

Key Hyperparameters:

$\tau_a$ : Proportion of target audio in the batch.
$\tau_t$ : Proportion of target text in the batch.
The authors found that setting the total target data proportion ( $\tau = \tau_a + \tau_t$ ) to 50% yields optimal results, balancing specialization and source-domain retention.

3. Key Contributions

Analysis of Target Data Proportion: The study quantifies the trade-off between target-domain specialization and source-domain performance, identifying an optimal batch composition ratio.
Hybrid Adaptation Strategy: The proposal of the Mixed Batching strategy, which integrates limited target-domain paired speech-text into a predominantly text-based adaptation setting. This directly addresses the modality gap caused by text-only fine-tuning.
Mitigation of Catastrophic Forgetting: Demonstration that the MB strategy prevents the loss of source-domain capabilities (a common issue in standard fine-tuning) while simultaneously improving target-domain recognition.

4. Experimental Results

Experiments were conducted across diverse domains: Banking (in-domain), Agriculture, and Musical Instruments (out-of-domain).

Performance with Limited Audio:
- Using Mixed Batching with only 10% of the target-domain audio (less than 4 hours of data) achieved Word Error Rates (WER) comparable to, or better than, conventional ASR fine-tuning using the full speech dataset.
- Example: In the Banking domain, 10% audio + MB achieved performance similar to 100% audio standard adaptation.
Comparison of Strategies:
- Text-Only Adaptation: Suffered from a significant modality gap (e.g., 6.38% WER vs. 4.55% for speech-text on Banking).
- Standard Speech-Text Adaptation: Outperformed text-only but suffered from catastrophic forgetting of the source domain.
- Mixed Batching: Consistently outperformed standard speech-text adaptation in low-resource scenarios. It achieved peak performance when the batch contained approximately 60% paired speech-text examples, after which gains plateaued.
Catastrophic Forgetting Analysis:
- Standard fine-tuning on 20% target audio caused a significant WER increase on the source domain (from 12.8% to 17.4%).
- Mixed Batching maintained source-domain performance (WER ~11.3%) while significantly improving target-domain recognition.

5. Significance and Implications

Data Efficiency: The paper proves that small amounts of high-quality paired audio (e.g., <4 hours) combined with abundant text can outperform models trained on full audio datasets. This drastically reduces the cost and effort required for domain adaptation in specialized fields.
Robustness: The strategy effectively bridges the modality gap, ensuring the LLM remains robust to the noisy representations produced by the speech projector.
Practical Deployment: For practitioners with access to domain text but scarce audio, this method offers a scalable solution to adapt LLM-based ASR systems without sacrificing the model's general capabilities or requiring massive audio datasets.

In conclusion, the paper establishes that limited audio is sufficient to realign the modality gap in LLM-based ASR, provided it is integrated via a carefully constructed mixed-batching strategy that preserves both acoustic alignment and linguistic specialization.