Technical Summary: Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs

1. Problem Statement

Integrating continuous acoustic signals into frozen Large Language Models (LLMs) presents a fundamental design challenge. Existing speech-to-LLM interfaces typically operate at two extremes, both of which suffer from significant limitations:

1. Discrete Token Alignment: Methods enforcing near-discrete token alignment (e.g., via CTC or text alignment objectives) benefit Automatic Speech Recognition (ASR) but collapse representations into near one-hot distributions. This effectively discards paralinguistic information such as prosody and emotion.
2. Unconstrained Continuous Representations: Methods using learned continuous tokens or latent embeddings offer flexibility but often drift away from the LLM's pretrained input embedding manifold. This misalignment leads to instability during autoregressive decoding and degrades performance.

The core issue is that neither approach properly aligns the geometry of speech representations with the LLM's input space. The former over-constrains the output to a purely lexical channel, while the latter under-constrains it, allowing the representation to leave the trained manifold.

2. Methodology: Convex Gate (C-Gate)

The authors propose Convex Gate (C-Gate), a bridge architecture that explicitly enforces geometric alignment with the LLM's input embedding space. Instead of mapping speech to discrete tokens or unconstrained vectors, C-Gate represents each time step as a convex combination of the LLM's own input token embeddings.

Architectural Components

• Geometric Constraint: Every speech representation $\tilde{e}_t$ is constrained to lie within the convex hull of the LLM's input embedding table $E$. Mathematically, $\tilde{e}_t = \sum_{v \in S_t} \alpha_{t,v} E_v$, where $\alpha_{t,v} \geq 0$ and $\sum \alpha_{t,v} = 1$. This ensures the representation remains strictly within the geometry the LLM was trained to interpret, preventing "basis drift" while preserving continuous expressivity.
• Scoring and Selection:
  1. Downsampling: A frozen speech encoder (Whisper-large-v3) produces hidden states, which are downsampled via stride-$k$ mean pooling.
  2. Query Projection: Pooled acoustic states are linearly mapped to queries ($q_t$) to score against the frozen vocabulary embedding table ($E$).
  3. Top-K Selection: A full-vocabulary cross-attention mechanism scores all tokens. The top-16 tokens ($S_t$) are selected deterministically.
  4. Convex Mixture: The final bridge vector is a renormalized weighted sum of the top-16 embeddings. Crucially, no value projection or post-codebook MLP is applied to the embeddings; the values remain the raw frozen LLM rows.
• Training Regime:
  • Frozen Components: The LLM (Qwen2.5-7B-Instruct), the speech encoder (Whisper), the embedding table, and the LM head remain frozen.
  • Trainable Parameters: Only the bridge similarity scorer (query/key projections, temperature) and the LLM's self-attention projections ($W_Q, W_K, W_V, W_O$) are trained.
  • Objective: A multi-task cross-entropy loss with dynamic loss reweighting (DR) is used to jointly optimize ASR, emotion recognition, and spoken reasoning tasks.

3. Key Contributions

1. Geometry-Constrained Interface: C-Gate introduces a convex gate that resolves the trade-off between lexical lock-in and basis drift by enforcing that all speech inputs lie within the LLM's trained embedding manifold.
2. Improved Information Transfer: The authors demonstrate a controlled transfer regime where joint training improves ASR performance (up to 48.7% relative reduction in WER) while preserving or enhancing paralinguistic (emotion) classification accuracy.
3. Mechanistic Insight: The paper identifies that information is not carried by discrete token identities or static routing weights, but by time-resolved trajectories through the embedding space. The working channel is the ordered sequence of selected supports.
4. Controlled Evaluation: The study establishes a stricter reporting protocol for multimodal integration, using causal interventions and stress tests to audit audio grounding rather than relying solely on multiple-choice benchmarks.

4. Experimental Results

The model was evaluated on LibriSpeech (ASR), RAVDESS (Emotion), and various reasoning benchmarks (VoiceBench-BBH, BBH-HO, MMSU, MMAU, SpMMLU).

• ASR Performance:
  • C-Gate-2T (ASR + Emotion) reduced autoregressive WER on LibriSpeech from 7.76% to 4.78% (38.4% relative improvement) compared to an ASR-only baseline, while simultaneously improving emotion accuracy from 96.2% to 97.1%.
  • C-Gate-3T (ASR + Emotion + Reasoning) further reduced WER to 3.98% (48.7% relative improvement over the ASR-only baseline), with a manageable drop in emotion accuracy (6.6 percentage points).
• Emotion Recognition: The model achieved 97.1% accuracy on RAVDESS, outperforming single-task baselines. Analysis showed gains were class-symmetric and not driven by a collapse onto frequent classes.
• Reasoning Probes: C-Gate-3T showed uniform improvements across all reasoning benchmarks compared to single-task reasoning baselines, serving as a boundary measurement for frozen-LLM capabilities under joint supervision.
• Scale Calibration: While C-Gate-3T does not match the absolute performance of massive-scale foundation models (e.g., Qwen2.5-Omni, Kimi-Audio) due to differences in training data volume and parameter count, it remains competitive with academic, public-data speech LLMs.

5. Mechanistic Analysis and Causal Interventions

The authors conducted causal interventions on C-Gate-3T to validate their hypotheses:

• Trajectory vs. Set: Shuffling the bridge trajectory in time caused a collapse in both ASR and emotion performance, confirming that the time-ordered trajectory is the critical information carrier, not just the set of selected tokens.
• Manifold Alignment: Replacing the trained embedding table with a Gaussian or row-permuted table erased performance gains, even though codebook size and dimensionality were preserved. This isolates the trained input manifold as the essential factor.
• Acoustic Content: Replacing audio with noise or zeros collapsed performance, confirming the reliance on real acoustic signals.
• Routing Diffuseness: Analysis of the routing posterior showed high entropy (diffuse routing), indicating that the model does not retrieve specific symbolic tokens per frame. Instead, information is encoded in the continuous mixture weights and the temporal evolution of the support set.

6. Significance and Claims

The paper argues that geometry, rather than token discreteness, is the fundamental design factor in speech-LLM interfaces. By constraining speech representations to the convex hull of the LLM's embedding space, C-Gate enables a frozen LLM to interpret continuous acoustic signals without retraining the backbone.

The authors claim that this approach provides a "controlled regime" for studying multimodal integration. They emphasize that their reasoning results should be viewed as boundary measurements of what is achievable with a frozen backbone, rather than claims of fully grounded spoken reasoning. The work suggests that future interfaces should focus on preserving the geometric structure of the embedding manifold rather than forcing discrete tokenization or unconstrained vector drift.

Limitations: The authors note that the strongest alternative hypothesis is that success comes simply from adding trainable capacity near the LLM, rather than the specific convex codebook organization. They acknowledge that their evaluation is limited to a single encoder-LLM pair, a single training seed, and acted emotion datasets, and they caution against deploying the model for high-stakes speaker inferences.