💬 NLP

Represented but Ignored: A Causal Account of Prosodic Underuse in Audio-Language Models

This paper introduces a causal probe ladder to demonstrate that audio-language models typically preserve and correctly represent prosodic information in their internal states but fail to utilize it in their final responses, a bottleneck that can be overcome through targeted hidden-state interventions.

Original authors: Linkai Peng, Baorian Nuchged

Published 2026-08-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Linkai Peng, Baorian Nuchged

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Human speech is far more than a sequence of words. When we speak, we carry meaning not just in our vocabulary, but in the shape of our voice. A simple sentence like "He is on the lilo" can become a question or a statement, or convey happiness or anger, solely through changes in pitch, timing, and volume. These musical qualities of speech, known as prosody, are essential for understanding how a person feels or what they truly mean. As artificial intelligence systems have grown more sophisticated, capable of listening to and understanding human speech, researchers have begun to ask a critical question: do these machines truly hear the music of speech, or do they only hear the lyrics?

The field of audio-language models has advanced rapidly, creating systems that can transcribe speech and answer questions about what was said. However, these systems often stumble when the meaning depends entirely on how something was said. A machine might correctly identify the words in a sentence but fail to recognize that the speaker was asking a question rather than making a statement, simply because it missed the rising tone at the end. Until now, when a model made such a mistake, researchers could only see the final wrong answer. They could not tell if the error happened because the machine failed to hear the sound, because it heard the sound but misunderstood it, or because it heard and understood the sound but simply chose to ignore it in its final response.

A team of researchers set out to solve this mystery by building a diagnostic tool to trace the journey of a sound as it moves through an artificial intelligence model. They treated the model not as a black box, but as a series of stages, much like a relay race. In the first stage, the audio is converted into a digital representation. In the second, the model processes this information internally. In the final stage, the model generates an answer. The researchers wanted to know exactly where the prosodic information gets lost. They tested four different advanced audio-language models using carefully controlled experiments where the words were identical, but the tone of voice changed to signal different meanings, such as a happy tone versus a sad one, or a question versus a statement.

The investigation revealed a surprising pattern. In most cases, the models were not failing to hear the sound, nor were they misinterpreting it. The information about the tone of voice was successfully captured by the audio sensors and remained clearly visible deep inside the model's internal processing layers. The researchers could look inside the model at specific points in its processing and see that the correct emotional or linguistic category was present and distinct. The signal was there, waiting to be used. The failure occurred at the very last step: the model possessed the correct understanding but did not let that understanding influence its final answer. It was as if the model had heard the question clearly, knew it was a question, but then decided to answer as if it were a statement.

To confirm that this internal signal was actually causing the model's behavior, the researchers performed a delicate experiment. They reached into the model's internal state at the precise moment before the answer was generated and made a tiny, targeted adjustment. By nudging the internal representation slightly in the direction of the correct tone, they were able to force the model to change its answer. In many instances, a single, small edit was enough to make the model switch from a wrong answer to the right one. This proved that the information was not just a passive byproduct of the model's thinking; it was a functional part of the system that, if properly activated, could drive the correct decision.

The study also looked at the specific features within the model that carried this information. They found that the ability to recover the correct answer relied on a very small, sparse set of internal connections. On the emotional tasks, these specific connections aligned with known acoustic cues, such as changes in energy or volume that humans use to express arousal. This suggests that the models are not inventing new ways to understand emotion, but are instead utilizing the same physical cues that humans use, yet failing to prioritize them when speaking.

The researchers concluded that the primary bottleneck for these advanced systems is not a lack of hearing or a failure of comprehension. The machines can hear the prosody, and they can represent it correctly inside their minds. The problem is that they are not using what they know. The recurring failure mode is one of underuse, where a model holds the correct information but fails to express it in its final output. This finding shifts the focus for future improvements. Rather than building better microphones or more complex audio encoders, the path forward may lie in training methods that encourage these models to trust and act upon the rich, expressive signals they are already capable of detecting. The machines are listening; they just need to be taught to speak up.

Technical Summary: Represented but Ignored: A Causal Account of Prosodic Underuse in Audio-Language Models

Problem Statement

Large Audio-Language Models (Audio-LLMs) are increasingly expected to interpret prosody—the intonational and paralinguistic cues in speech that convey questions, statements, and emotions. However, standard behavioral benchmarks evaluate only the final output (e.g., "Is this a question?"), rendering the cause of failure ambiguous. A model might fail because:

F1 (Perception Failure): The audio pathway loses the prosodic contrast before it reaches the language model.
F2 (Interpretation Failure): The model internally represents the prosodic cue incorrectly.
F3 (Underuse Failure): The model correctly represents the prosodic cue internally but fails to utilize it in the final decision.

Current evaluation methods cannot distinguish between these mechanistically distinct failure modes. This paper argues that behavioral errors alone are insufficient for diagnosing whether a model "hears" prosody but fails to "use" it.

Methodology

The authors introduce a stage-specific probe ladder to localize prosodic failures across the Audio-LLM stack. This framework combines three diagnostic tools:

Audio-Path Probes (Diagnosing F1): Linear probes are trained on the frozen audio tower and projector outputs to determine if prosodic information is preserved in the representation passed to the LLM.
Logit-Lens Readouts (Diagnosing F2): A layer-wise logit-lens (using the model's unembedding matrix) reads the residual stream at the answer position to determine if the correct prosodic category is decodable from the internal states. The layer with the highest Area Under the Curve (AUC) is denoted as $L^*$ .
Behavioral Readouts (Diagnosing F3): A "three-condition ladder" compares model performance across:
- Text-no-cue baseline: Transcript only (measures lexical prior).
- Audio condition: Full audio input.
- Text+cue reference: Transcript with an explicit text label of the prosody (measures the model's upper bound for using prosodic cues).
- F3 Signature: The "capability gap" where the audio condition performs better than the baseline but significantly worse than the reference, despite high internal decodability (high Logit-Lens AUC).

Causal Interventions: To verify that the internal signal is causally responsible for the output (rather than a mere correlate), the authors perform two interventions at $L^*$ :

Direction Injection: Adding a class-mean prosodic direction vector ( $\mu_+ - \mu_-$ ) to the answer-position residual stream.
Activation Patching: Replacing the answer-position state of a recipient clip with the corresponding state from a matched donor clip of the opposite class.
Sparse Feature Intervention: Using Sparse Autoencoders (SAEs) trained on $L^*$ activations. Features are ranked by causal attribution (AtP*), and a small subset ( $S_{0.95}$ ) covering 95% of attribution is clamped to drive the model toward a target class.

Models and Data: The study evaluates four understanding-only Audio-LLMs: Qwen2.5-Omni-7B, Phi-4-multimodal-instruct, Audio-Flamingo-3, and DeSTA2.5-Audio. Experiments use matched-content contrasts from IViE (question/statement), CREMA-D, and VESUS (emotion), ensuring lexical content is controlled so distinctions rely solely on prosody.

Key Results

1. Diagnosis of Failure Modes

Across 11 clean model-contrast cells:

F1 (Perception): Generally disfavored. Three Whisper-tower models (Qwen, Audio-Flamingo-3, DeSTA) preserve both intonational and emotional prosody through the audio path. The exception is Phi-4 on emotion tasks, where the conformer encoder attenuates the signal (F1).
F2 (Interpretation): Generally disfavored. In 8 of 11 cells, the correct prosodic category is decodable from the late-layer LLM states (Logit-Lens AUC $\ge$ 0.81).
F3 (Underuse): Identified as the dominant recurring pattern in 7 of 11 cells. Even when the prosodic code is decodable internally (high AUC), the behavioral output recovers only a fraction of the available range (e.g., 30–59% recovery on emotion tasks, 0% on DeSTA's IViE task).

2. Causal Validation

Direction Injection: Shifting the $L^*$ state along the prosodic direction consistently shifts the answer-token log-odds in the predicted direction across all 18 clean cells.
Activation Patching: Swapping naturally occurring states between donor and recipient clips produces bidirectional shifts in the output distribution, confirming that $L^*$ states causally influence the decision.
Recovery: A single linear edit at $L^*$ is sufficient to drive the model toward the suppressed prosodic class in most cases, though the effect acts as a directional bias rather than a perfect selective restoration.

3. Feature-Level Localization

Sparse Subspace: The recoverable signal can be expressed through a very small subspace of SAE features (33–102 features, <0.5% of the dictionary). Clamping these attribution-selected features recovers the gated class in 17 of 18 cells.
Acoustic Correlates: On emotion contrasts, top-attribution features show strong correlations ( $|r| \approx 0.51\text{--}0.73$ ) with RMS energy (vocal arousal). Intonation features show weaker correlations with F0 slope but still form clear categorical clusters.

Significance and Claims

The paper claims to provide the first causal account of why Audio-LLMs fail on prosodic tasks despite having the necessary internal representations.

Primary Conclusion: The recurring bottleneck in expressive speech understanding is not a failure to perceive prosody (F1) or a failure to interpret it internally (F2), but a failure to use an available internal representation (F3).
Mechanistic Insight: Models "hear" and correctly represent prosodic cues but fail to express them in their final answers. This "capability gap" is defined by the difference between a lens-readable internal signal and behavioral performance.
Implications: Progress in this domain may depend less on improving audio encoders and more on training and decoding strategies that promote the translation of existing internal representations into behavior.
Limitations: The authors note that F3 is the dominant but not universal pattern (e.g., Phi-4's emotion failures are upstream perception issues). Furthermore, the interventions induce a directional bias rather than a perfect class restoration, and the findings are currently limited to dense-transformer architectures.

The work establishes a diagnostic framework (the probe ladder) and causal evidence (interventions) to distinguish between perception, interpretation, and utilization failures in multimodal models.