💬 NLP

Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Text

This paper identifies that despite strong downstream performance, current Spoken Language Models suffer from weak alignment between speech and text representations due to structural differences, and proposes a framework that decouples length mismatch from semantic alignment to bridge this gap and improve instruction-following and generalization.

Original authors: Hyeonyu Kim, Hwayeon Kim, Youngwon Choi, Myeongkyun Cho, Huu-Kim Nguyen

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hyeonyu Kim, Hwayeon Kim, Youngwon Choi, Myeongkyun Cho, Huu-Kim Nguyen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quiet hum of modern technology, a new kind of intelligence is learning to listen. For years, computers have been excellent at reading words and even better at recognizing them when spoken aloud, but they have struggled to do both at once in a single, fluid thought. Imagine a machine that hears a voice and immediately understands the meaning, tone, and intent behind it, just as a human does, without first converting the sound into a written transcript. This is the promise of spoken language models. These systems are designed to take the raw, continuous flow of human speech—where pauses, speed, and emotion vary constantly—and translate it directly into intelligent text responses. The challenge has always been that speech and text are fundamentally different things. Speech is a wave that changes over time, stretching and compressing with the speaker's breath, while text is a series of fixed, discrete blocks. Bridging this gap has been the central hurdle in teaching machines to truly hear as well as they read.

A team of researchers recently set out to investigate why these models, despite showing promise, still stumble when asked to follow complex instructions or generalize to new tasks. They asked a simple, probing question: when a spoken language model processes a sentence, does it actually hear the speech the way it reads the text? To find the answer, they looked inside the "mind" of these models, examining how the computer represents a spoken word compared to how it represents the same word written down. They discovered that even when the models performed well on standard tests, the internal maps they built for speech and text were still weakly connected. The speech signals remained structurally different from the text, as if the model was hearing the sound but not quite understanding its shape in the same way it understood the written word.

The researchers found that the core issue was not just about getting the meaning right, but about how the model handled the length and structure of the information. Speech is a long, continuous stream, while text is short and choppy. Previous attempts to fix this involved forcing the long speech signals to shrink down to match the short text, a process that often blurred important details or lost the rhythm of the voice. The team proposed a different approach. Instead of trying to force the two to look the same immediately, they created a system that first matches the length of the speech to the length of the text, and then focuses on aligning the meaning. They did this by dynamically adjusting how many "tokens," or units of information, the model used to represent the speech, ensuring it matched the target text exactly during the learning phase. This allowed the model to separate the problem of length from the problem of meaning, giving it a clearer path to learn the true connection between sound and word.

To test this idea, the researchers trained their model using approximately 69,000 hours of paired speech–text data, which included diverse speech-specific attributes. They taught the model not just to repeat what it heard, but to follow instructions given in speech, such as answering questions or summarizing stories, just as it would if reading the text. Crucially, they added a second layer of training that encouraged the model to make the internal representation of the speech match the internal representation of the text at every single step, not just at the final answer. This fine-grained alignment helped the model understand that a specific sound corresponds to a specific word, even when the timing was different. The results were striking. On a series of rigorous tests designed to measure how well these models understand and reason with speech, the new approach performed as well as, and in some cases better than, the most advanced systems available, including those from major technology companies.

However, the study also revealed a subtle but important limit to how much alignment is helpful. The researchers found that while matching the speech and text representations improved performance, pushing this alignment too hard actually made the model worse at certain tasks. When the model was forced to make the speech look exactly like the text, it began to lose the unique qualities of the voice, such as emotion, hesitation, or the specific tone that carries meaning beyond the words. The best results came from a balance: enough alignment to understand the words, but enough separation to keep the richness of the human voice. This suggests that for a machine to truly hear, it must learn to respect the differences between sound and text, rather than trying to erase them.

The implications of this work extend beyond just making better chatbots. By showing that explicitly addressing the structural differences between speech and text leads to better performance, the researchers have provided a new blueprint for how these systems should be built. They demonstrated that the key to unlocking the full potential of spoken language models lies in treating speech not as a flawed version of text, but as a distinct signal that requires its own careful handling. The study confirms that when we stop trying to force speech to fit the mold of text, and instead build bridges that respect the unique nature of both, machines can finally learn to listen with the same depth and nuance that they read.

Technical Summary: Do Spoken Language Models Hear Speech as They Read Text?

Problem Statement

Spoken Language Models (SLMs) aim to generate textual responses directly from speech inputs, bypassing the error propagation inherent in cascaded Automatic Speech Recognition (ASR) and Large Language Model (LLM) systems. Despite recent advances, existing SLMs exhibit weaker instruction-following capabilities and limited generalization across diverse tasks compared to text-based LLMs.

The paper identifies a fundamental discrepancy: while SLMs may achieve strong downstream performance, their internal representations of speech remain weakly aligned with text representations. This misalignment persists despite strong semantic correspondence between speech and its transcription. The authors argue that this is due to structural differences between the two modalities:

Speech is a continuous, temporally varying signal, resulting in longer, structurally distinct feature sequences.
Text is discrete and symbolic.

Prior approaches often attempt to reduce the length of mapped speech features (via downsampling or fixed-rate compression) or enforce behavioral alignment (matching output responses). However, these methods often leave the internal alignment of representations implicit or force the modality adapter to perform length matching and semantic alignment simultaneously, which the authors suggest is suboptimal.

Methodology

The authors propose a framework that explicitly addresses structural differences by decoupling length mismatch from semantic alignment and introducing token-level internal alignment.

1. Dynamic Query Allocation (Decoupling Length and Semantics)

The core architectural innovation is a modality adapter based on a windowed Q-former. Unlike prior works that use fixed query allocation rates (e.g., 3 tokens per second of audio), this framework employs Dynamic Query Allocation (DQA):

Training: The model leverages speech-transcription pairs $(s, t)$ . It dynamically allocates the number of query tokens ( $L_t$ ) to exactly match the length of the target text embeddings. This explicitly separates the task of matching sequence lengths from the task of learning semantic correspondence.
Inference: A lightweight speech rate predictor estimates the target token length from the input speech to allocate queries accordingly.

2. Token-Level Internal Alignment

In addition to standard Behavior Alignment (training the model to generate the same response $y$ from speech as it would from a text description of that speech), the authors introduce a Token-Level Internal Alignment Loss ( $L_{internal}$ ):

This loss enforces cosine similarity between the mapped speech features and the corresponding text embeddings at specific hidden layers of the LLM (input layer and layers $N/4, N/2, 3N/4, N$ ).
Because DQA ensures the speech and text sequences have the same length during training, the loss can be computed over corresponding token positions.
The final objective is $L = L_{behavior} + \lambda L_{internal}$ , where $\lambda$ controls the strength of the internal alignment.

Key Contributions

Analysis of Internal Representations: The authors demonstrate that current SLMs exhibit weak alignment between speech and text internal representations, evidenced by low Centered Kernel Alignment (CKA) scores and inconsistent token-wise similarity maps, even when downstream performance is acceptable.
Decoupled Framework: They propose a simple framework that separates length matching from semantic alignment via Dynamic Query Allocation, allowing the model to learn meaningful semantic correspondences without being constrained by fixed compression rates.
Token-Level Alignment: They incorporate a token-level internal alignment loss to encourage fine-grained correspondence between speech and text features, moving beyond mere behavioral alignment.
Empirical Validation: The approach achieves competitive performance against strong baselines (including closed-source models like Gemini-1.5-Pro and Gemini-2.0-Flash) across multiple benchmarks, validating the importance of explicitly addressing structural differences.

Experimental Results

The model was evaluated on four major benchmarks: AIR-Bench (instruction-following), SpeechR (reasoning), MMSU (perception and reasoning), and Speech-IFEval (instruction-following constraints).

Performance: The proposed model ("Ours") achieved competitive results, outperforming strong baselines like SALMONN and Qwen2-Audio-Instruct on several subsets. Notably, it showed significant improvements in the "forgetting rate" on Speech-IFEval compared to SALMONN.
Representation Similarity: The model achieved a significantly higher CKA score (0.6399) compared to existing SLMs (e.g., 0.3570 for BLSP-emo, 0.4113 for Qwen2-Audio-Instruct), indicating stronger structural similarity between speech and text representations.
Ablation Studies:
- Removing either behavior alignment or internal alignment caused performance degradation of over 10%.
- Alignment Strength: Increasing the weight ( $\lambda$ ) of internal alignment improved CKA scores but, if set too high, degraded downstream performance. This suggests that overly strong alignment can over-constrain speech representations, potentially losing speech-specific information (e.g., paralinguistic cues).
- Query Allocation: Replacing Dynamic Query Allocation with Fixed Query Allocation (FQA) or using a joint length-matching/semantic adapter (CIF-based) resulted in substantial performance drops, confirming the benefit of decoupling these tasks.
Speech-Specific Information: Attention analysis on emotional speech (IEMOCAP) showed that despite strong alignment with text, the model retains sensitivity to speech-specific cues (e.g., emotional expression regions), which are absent in neutralized or text-only inputs.

Significance and Claims

The paper argues that the persistent performance gap between SLMs and text-only LLMs is partly due to unaddressed structural differences between continuous speech and discrete text. The authors claim that:

Explicitly addressing structural differences (specifically length mismatch) is crucial for effective SLM training.
Decoupling length matching from semantic alignment allows models to learn better representations without forcing the adapter to solve both problems simultaneously.
Internal alignment should be balanced: While token-level alignment improves representational similarity, it should complement, not replace, behavior alignment. Over-maximizing alignment can hinder the model's ability to capture speech-specific information (prosody, emotion) that is not present in text transcriptions.

The work concludes that future SLMs must carefully balance linguistic correspondence with the preservation of modality-specific cues, rather than treating alignment as a quantity to be maximized indiscriminately.