⚡ electrical engineering

Closing the Gap Between Text and Speech Understanding in LLMs

The paper introduces SALAD, a data-efficient method that combines cross-modal distillation with targeted synthetic data to address the text-speech understanding gap in speech-adapted Large Language Models by mitigating capability forgetting and improving cross-modal alignment, achieving competitive performance on public corpora with significantly less training data.

Original authors: Santiago Cuervo, Skyler Seto, Maureen de Seyssel, Richard He Bai, Zijin Gu, Tatiana Likhomanenko, Navdeep Jaitly, Zakaria Aldeneh

Published 2026-02-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Santiago Cuervo, Skyler Seto, Maureen de Seyssel, Richard He Bai, Zijin Gu, Tatiana Likhomanenko, Navdeep Jaitly, Zakaria Aldeneh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, world-class librarian (the Large Language Model or LLM). This librarian has read almost every book in the library and can answer any question, solve complex riddles, and tell great stories. They are a genius at reading text.

Now, imagine you want this librarian to also listen to people talking to them, not just read their notes. You want them to be a great listener as well as a great reader.

The Problem: The "Muffled Ear" Effect

When researchers tried to teach this librarian to listen, they hit a snag. Even after training, the librarian became much worse at understanding the conversation when it was spoken, compared to when it was written.

The Text Version: "What is the capital of France?" -> The librarian answers instantly and correctly.
The Speech Version: Someone asks, "What is the capital of France?" -> The librarian hesitates, gets confused, or gives a wrong answer.

The authors call this the "Text-Speech Understanding Gap." It's like the librarian suddenly developed a case of "muffled ears." They can still read the notes perfectly, but the moment you speak to them, their brain seems to fog up.

Why Did This Happen?

The paper digs into two main reasons why the librarian gets confused:

Forgetting the Basics (Amnesia): When you force the librarian to focus entirely on listening, they start to forget the things they already knew about reading. It's like if you tried to learn to play the guitar by only practicing with your left hand; eventually, your right hand (the reading skills) gets rusty.
The "Translation" Glitch: The librarian hears a sound (speech) and tries to match it to a meaning (text). But the connection is weak. It's like trying to translate a song into a poem, but the translator keeps mixing up the words because the rhythm of the song doesn't match the rhythm of the poem.

The Old Solutions (Too Expensive or Impossible)

Previously, people tried to fix this in two ways:

The "Synthetic Voice" Method: They used computers to turn millions of text books into fake speech recordings and trained the librarian on those. This worked a little, but it was incredibly expensive, and the computer voices sounded a bit robotic, so the librarian learned to understand robots, not humans.
The "Secret Library" Method: Some big companies used massive, secret collections of real human conversations (millions of hours) to train their models. This worked well, but nobody else could copy their work because they couldn't see the data.

The New Solution: SALAD

The authors created a new method called SALAD (Sample-efficient Alignment with Learning through Active selection and cross-modal Distillation). That's a mouthful, so let's break it down with a cooking analogy.

Imagine you are trying to teach a chef (the AI) to cook a specific dish (understanding speech) using a recipe they already know (text knowledge).

Step 1: The "Taste Test" (Cross-Modal Distillation)
Instead of just letting the chef taste random ingredients, you have the chef taste the real dish while simultaneously looking at the perfect recipe. You tell them: "When you hear this sound, it means this word."

The Magic: The chef learns to match the sound to the word without forgetting how to read the recipe. This stops the "amnesia" and fixes the "translation glitch."

Step 2: The "Smart Shopping" (Active Selection)
Here is the clever part. The chef still doesn't know how to cook every type of cuisine (like advanced biology or obscure history) because real human conversations usually stick to everyday topics (weather, food, gossip).

The Problem: If you just feed the chef random real conversations, they never learn the hard stuff.
The SALAD Fix: The system acts like a smart shopping assistant. It looks at what the chef is bad at (e.g., "I don't understand the word 'mitochondria' when spoken"). It then goes to the library, finds a text about mitochondria, and synthesizes just that one sentence into speech.
The Result: Instead of buying 1,000 pounds of generic ingredients (massive datasets), the chef only buys the specific missing ingredients they need. This is "Active Selection."

Why Is This a Big Deal?

Efficiency: The old methods needed millions of hours of data. SALAD achieved similar (or better) results using 10 times less data. It's like getting a Michelin-star meal by cooking with a tiny, perfectly chosen basket of ingredients instead of a whole warehouse.
No Amnesia: Because they used the "Taste Test" method (distillation), the librarian didn't forget how to read. They kept their original genius while gaining the ability to listen.
Open Source: Unlike the "Secret Library" methods, this approach uses public data and smart algorithms, so anyone can build these better listening AI systems.

The Bottom Line

The paper shows that you don't need to throw massive amounts of money and data at the problem to make AI understand speech. You just need to be smarter about how you teach it. By carefully matching what the AI hears with what it already knows, and only filling in the specific gaps it has, you can create an AI that is just as smart when you talk to it as when you type to it.

In short: SALAD is the "personal tutor" that helps the AI learn to listen without forgetting how to read, using a tiny, targeted diet of data instead of a giant buffet.

1. Problem Statement: The Text–Speech Understanding Gap

Large Language Models (LLMs) excel at text-based reasoning and knowledge tasks. While they can be adapted to process speech inputs (Speech-Adapted LLMs), these models consistently underperform their text-based counterparts and even cascaded pipelines (ASR + Text LLM) on language understanding tasks.

The authors define this shortfall as the Text–Speech Understanding Gap: the performance drop observed when a speech-adapted LLM processes spoken inputs compared to the original text-based LLM processing equivalent text.

Current Limitations: Existing approaches to close this gap rely on either:
1. Massive Synthetic Data: Generating millions of hours of synthetic speech from text corpora, which is computationally expensive and lacks natural paralinguistic cues.
2. Proprietary Datasets: Using massive, non-reproducible proprietary speech datasets (e.g., millions of hours), which are inaccessible to the broader research community.
Core Hypothesis: The gap is driven by two specific factors:
1. Forgetting: The loss of pre-trained text capabilities during the adaptation process due to domain shift.
2. Cross-Modal Misalignment: The divergence in model outputs when given semantically equivalent speech vs. text inputs.

2. Methodology: SALAD

The authors propose SALAD (Sample-efficient Alignment with Learning through Active selection and cross-modal Distillation). This is a two-stage training strategy designed to maximize performance while minimizing reliance on synthetic data.

Stage I: Distillation on Natural Speech

Objective: Instead of standard Maximum Likelihood Estimation (NLL), SALAD uses Cross-Modal Knowledge Distillation.
Mechanism: The text-based LLM backbone ( $Q_\phi$ ) acts as a "teacher." The speech-adapted model ( $P_\theta$ ) is trained to match the teacher's output distribution when given speech inputs.
Loss Function: The training objective interpolates between NLL and Distillation ( $L = \alpha L_{DIST} + (1-\alpha) L_{NLL}$ ). The authors found that $\alpha=1$ (pure distillation) significantly reduces misalignment and mitigates forgetting compared to standard NLL training.
Data: Trained on natural speech corpora (LibriHeavy and Emilia).

Stage II: Active Selection for Domain Expansion

Motivation: Natural speech datasets are narrow in domain coverage compared to broad-domain text pretraining corpora (e.g., FineWeb-Edu). Simply synthesizing massive amounts of speech is inefficient.
Mechanism: The authors introduce an Active Learning algorithm to identify and fill specific domain gaps.
1. Clustering: The broad-domain text corpus is partitioned into clusters using sentence embeddings.
2. Misalignment Scoring: The model measures the divergence (misalignment) between its predictions and the teacher's predictions for each cluster.
3. Importance Sampling: Clusters with high misalignment (indicating the model struggles with those specific domains) are upweighted.
4. Targeted Synthesis: Only text samples from these high-misalignment clusters are synthesized into speech and added to the training set.
Efficiency: This approach allows the model to learn from a "missing domain" using a tiny fraction of synthetic data (e.g., 1% of the natural speech dataset size) rather than synthesizing the entire corpus.

3. Key Contributions

Quantification of the Gap: The authors formally quantify Forgetting and Cross-Modal Misalignment as statistical distances (KL divergence) between the speech-adapted model and the text backbone. They demonstrate these metrics are highly predictive of downstream benchmark performance.
Analysis of Training Dynamics: They show that standard NLL training on narrow-domain speech data increases misalignment as training scales. Conversely, cross-modal distillation improves alignment and reduces forgetting, even on narrow data.
SALAD Framework: A novel, sample-efficient method combining distillation and active selection. It achieves state-of-the-art results without requiring massive proprietary datasets or full-scale synthetic data generation.
Reproducibility: The method is fully open-weight and relies on public corpora, contrasting with recent "black-box" models that require millions of hours of proprietary data.

4. Experimental Results

The authors evaluated SALAD on 3B and 7B parameter models (based on Qwen2.5) across six broad-domain benchmarks: StoryCloze, MMSU, OpenBookQA, HellaSwag, ARC-Challenge, and PIQA.

Performance vs. SOTA:
- SALAD-3B outperforms larger end-to-end baselines (like DiVA-8B and GLM-4-Voice-9B) on spoken language understanding.
- SALAD-7B achieves performance competitive with the strongest closed-source model, Qwen2.5-Omni-7B, while using over an order of magnitude less training speech data.
Gap Reduction:
- SALAD significantly reduces the text-speech understanding gap. For example, on the 7B model, it narrows the gap by ~11.7% compared to the next-best open-weight model.
- It approaches the performance of cascaded pipelines (ASR + Text LLM) while retaining the benefits of end-to-end modeling (preserving paralinguistic cues).
Preservation of Text Capabilities:
- Unlike other speech-adapted models that suffer from significant "forgetting" (dropping text-only performance), SALAD models maintain or even slightly improve their original text capabilities.
Active Selection Efficacy:
- Ablation studies confirm that the active selection strategy (Stage II) specifically boosts performance on technical and scientific domains (e.g., Biology, Chemistry in MMSU) where natural speech data is scarce.

5. Significance and Impact

Data Efficiency: SALAD proves that massive, expensive synthetic datasets are not strictly necessary to close the text-speech gap. By intelligently selecting which data to synthesize, researchers can achieve high performance with minimal resources.
Democratization: By relying on public datasets and open-weight models, SALAD provides a reproducible path for the community to build high-quality speech-adapted LLMs, bypassing the need for proprietary data hoarding.
Theoretical Insight: The paper shifts the focus from "more data" to "better alignment objectives." It establishes that cross-modal distillation is the superior objective for maintaining text capabilities while learning speech, and that active learning is the key to bridging domain distribution gaps.
Future Direction: This work paves the way for low-latency, streaming-friendly speech LLMs that do not require massive non-causal encoders or proprietary data, making natural spoken interaction more accessible.