Entity Binding Failures in Speech LLM Reasoning: Diagnosis and Chain-of-Thought Intervention

Technical Summary: Entity Binding Failures in Speech LLM Reasoning

Problem Statement
Speech Large Language Models (SLLMs) consistently underperform their text-based counterparts on complex reasoning benchmarks. Previous research has largely attributed this speech-to-text (S2T) versus text-to-text (T2T) modality gap to macroscopic factors such as generalized information dilution or modality-level representation divergence. This paper challenges the view that this gap represents a uniform cognitive deficit. Instead, the authors identify a specific, localized failure mode: entity binding failure. While SLLMs perform comparably to text models on spatial, syntactic, and factual tasks, their accuracy collapses to chance levels on logical reasoning tasks that require tracking entities and their changing properties (e.g., the "web of lies" task). The authors diagnose this as a structural bottleneck where continuous speech features, processed via temporal pooling and downsampling, blur the discrete token boundaries necessary for implicit entity-property association.

Methodology
The authors conduct a fine-grained, task-specific evaluation of two architecturally diverse SLLMs: Qwen2.5-Omni and Phi-4-Multimodal. They evaluate these models on 1,000 items from the VoiceBench BBH split, categorized into four reasoning types: hyperbaton, navigate, sports understanding, and web of lies. Each item is presented in both synthesized speech and plain text to enable paired accuracy comparisons.

To address the diagnosed entity binding failure, the authors introduce Entity-Aware Chain-of-Thought (EA-CoT), a lightweight inference-time intervention. Unlike generic CoT prompts, EA-CoT forces the SLLM to explicitly:

1. Enumerate entities: List all people mentioned in the input.
2. Record claims: Write down statements linking each person to a specific property or state.
3. Reason step-by-step: Resolve claims in sequence.
4. Extract the answer: Produce the final response.

This process externalizes fragile implicit acoustic tracking into explicit textual anchoring. To isolate the effect of entity binding from general reasoning benefits, the authors deploy isomorphic structured control prompts for non-entity-tracking tasks (e.g., listing adjectives for hyperbaton). Additionally, they implement a token budget control experiment to disentangle performance gains driven by instruction content from those driven by increased generation length (256 vs. 1,024 tokens).

Key Contributions

1. Task-Specific Gap Diagnosis: The paper demonstrates that the S2T/T2T modality gap is not uniform but is heavily concentrated in entity-tracking logical reasoning tasks. Excluding these tasks, S2T reasoning performance is largely comparable to T2T.
2. Mechanistic Explanation: The authors link SLLM logical failures to the loss of acoustic details through encoder pooling. They argue that blurred token boundaries disrupt implicit entity binding, while global semantic information remains preserved.
3. Effective Targeted Intervention: EA-CoT is introduced as a causal probe and solution. It uniquely bridges the modality gap in logical reasoning (yielding up to a 24.4 percentage-point accuracy gain) and remains effective even when entity names are phonetically misrecognized, proving the failure is structural rather than purely perceptual.

Results

• Baseline Performance: On the web of lies task, S2T accuracy for both models plummeted to chance levels (~50–53%), while T2T baselines ranged from 86.8% to 91.6%. In contrast, other tasks showed minimal gaps (<8 percentage points), with some even favoring speech input.
• EA-CoT Efficacy: EA-CoT significantly improved S2T accuracy on web of lies, recovering +16.4 percentage points (pp) for Qwen and +24.4 pp for Phi-4. This recovery scaled with the model’s underlying T2T capability.
• Instruction vs. Token Budget: Decomposition analysis confirmed that gains were driven by the structured instructions, not the increased token budget. Increasing the token limit without EA-CoT instructions yielded negligible improvement (<0.2 pp) for speech inputs.
• Ablation Study: On Qwen’s S2T web of lies performance, explicit entity enumeration contributed 59% of the total EA-CoT effect, confirming it as the critical component.
• Robustness to Misrecognition: EA-CoT maintained logical integrity even when the model transcribed names inconsistently (e.g., mapping "Ka" to "Cass"), as long as a consistent textual anchor was established.
• Specificity: The intervention did not improve performance on the MMSU benchmark, which relies on acoustic cues rather than semantic binding, confirming that EA-CoT specifically repairs semantic structural binding rather than acting as a generic reasoning enhancer.

Significance and Claims
The paper claims to reframe the S2T/T2T modality gap in logical reasoning from a missing capability to an elicitation failure. By demonstrating that SLLMs possess the underlying reasoning capability but fail to activate it reliably under speech input due to entity binding issues, the authors argue that explicit semantic binding via EA-CoT can bridge this gap. The work highlights that the critical failure in SLLM logical reasoning is a structural semantic binding failure caused by acoustic detail loss, rather than a superficial speech recognition error. The authors note that while EA-CoT is effective, it increases inference latency, suggesting future work should explore representation-level alignments to stabilize entity tracking without autoregressive overhead.