Technical Summary: Minimizing Modality Gap from the Input Side with TextPro-SLM

1. Problem Statement

Speech Large Language Models (SLMs) are predominantly constructed by continually training from Text Large Language Model (TLM) checkpoints to extend reasoning and language understanding capabilities to spoken interaction. Despite sharing a backbone with powerful TLMs, SLMs suffer from a significant modality gap, where speech-based question answering (QA) performance lags substantially behind text-based QA performance (e.g., up to a 20% drop in benchmarks like GLM-4-Voice).

Prior research has primarily attempted to bridge this gap from the output side, evolving from end-to-end speech token generation to "thinker-talker" architectures where the LLM generates text tokens and a separate module synthesizes speech. While these approaches improve performance by aligning the generation process with TLM paradigms, the modality gap persists. The authors argue that the remaining bottleneck lies on the input side: unlike TLMs, SLMs must infer both linguistic content and paralinguistic cues (prosody, emotion, speaker identity) directly from speech representations, which are often not presented to the LLM in a format it can fully exploit.

2. Methodology

The paper proposes TextPro-SLM, a framework designed to minimize the modality gap by transforming spoken input into a representation that closely resembles that of a prosody-aware text LLM. The core philosophy is to separate "what is said" (semantic content) from "how it is said" (paralinguistic information) at the input stage, rather than compressing speech into a single latent stream.

2.1 WhisperPro: A Unified Speech Encoder

The first component is WhisperPro, a speech encoder based on Whisper-large-v3. Unlike standard ASR models that prioritize lexical content, WhisperPro is trained to produce two synchronized outputs:

1. Text Tokens: Accurate transcriptions of the utterance content.
2. Prosody Embeddings: Token-aligned embeddings encoding paralinguistic information (speaking style, emotion, timbre).

Architecture & Training:

• Encoder: Uses the 32-layer Transformer encoder-decoder of Whisper-large-v3.
• Extraction: Prosody embeddings are extracted from the hidden states of the 16th decoder layer, aligned with the generated text tokens.
• Reconstruction Objective: To ensure the encoder retains paralinguistic details, a Mel-reconstructor (decoder) is attached. It attempts to reconstruct the original log-Mel spectrogram from the paired text tokens and prosody embeddings.
• Loss Function: The model is optimized with a multi-task objective:
  $$L = L_{ASR} + \lambda L_{mel}$$
  where $L_{ASR}$ is the cross-entropy loss for transcription and $L_{mel}$ is the Mean Squared Error (MSE) loss for spectrogram reconstruction. $\lambda$ is set to 1.0.
• Data: Trained on 1,000 hours of audio (LibriSpeech and CommonsenseQA), utilizing punctuated transcriptions and TTS-synthesized QA data to improve robustness.

2.2 Prosody-Aware LLM Backbone

The second component is an LLM backbone (Qwen2.5-3B or 7B-Instruct) trained to preserve semantic capabilities while learning paralinguistic understanding.

Input Format:
The LLM receives the text transcript as the primary input, with prosody information injected via two strategies:

1. Global Prepending (Default): Prosody embeddings are averaged into a single utterance-level vector, projected via an MLP, and prepended to the text sequence.
2. Interleaving: Compressed prosody embeddings are inserted at fixed intervals among text tokens to capture finer-grained prosodic variations.

Training Pipeline:
The LLM is trained using a two-stage pipeline with two coupled objectives:

1. Semantic Intelligence Preservation (Knowledge Distillation): The model is distilled from teacher TLMs using text-only datasets (UltraChat, CommonsenseQA) converted to speech via TTS. The student receives WhisperPro's text+prosody input but learns to mimic the teacher's text-only responses.
2. Paralinguistic Understanding: The model is trained on tasks involving speaker emotion (SER), gender, age, and accent detection using datasets like IEMOCAP, CREMA-D, and Common Voice.

3. Key Contributions

1. Input-Side Modality Gap Reduction: The paper shifts the focus from output-side generation strategies to input-side representation, arguing that making speech input resemble a "prosody-aware text LLM" is the key to reducing the gap.
2. WhisperPro Encoder: Introduces a unified encoder that simultaneously outputs accurate transcripts and rich, token-aligned prosody embeddings, trained via a reconstruction objective to preserve acoustic details without sacrificing ASR accuracy.
3. Data Efficiency: Demonstrates that significant gains can be achieved with only ~1,000 hours of audio training data for the LLM, contrasting with the massive datasets often required for end-to-end speech models.
4. Dual-Stream Representation: Proposes a design where semantic and paralinguistic information are explicitly separated and synchronized, allowing the LLM to retain its native text-processing strengths while gaining prosodic awareness.

4. Experimental Results

Experiments were conducted on 3B and 7B parameter scales, comparing TextPro-SLM against leading SLMs (e.g., GLM-4-Voice, Qwen2.5-Omni, Kimi-Audio) and cascaded baselines.

• Modality Gap: TextPro-SLM achieved the lowest average modality gap among all compared SLMs.
  • TextPro-SLM-7B achieved an average gap of 0.7%, significantly outperforming SALAD-7B (7.1%) and Qwen2.5-Omni-7B (3.1%).
  • On reasoning-heavy benchmarks (MMSU, OBQA), TextPro-SLM reduced the gap to near-zero (e.g., 0.7% on OBQA), whereas other models struggled with gaps exceeding 10-30%.
• Paralinguistic Understanding: TextPro-SLM achieved state-of-the-art performance on paralinguistic tasks (Emotion, Gender, Age, Accent) among the baselines, with the 7B model reaching 64.8% average accuracy.
• Ablation Studies:
  • Removing Knowledge Distillation (KD) caused a substantial performance drop, confirming the necessity of preserving TLM semantic capabilities.
  • Removing the reconstruction objective in WhisperPro degraded paralinguistic performance, validating the importance of the reconstruction loss for encoding prosody.
  • Using the interleaved input (5:1 ratio) yielded comparable results to global prepending, suggesting that a compact global prosody vector is often sufficient.

5. Significance and Claims

The paper claims that TextPro-SLM validates the hypothesis that the most effective route to stronger Speech LLMs is to transform them into prosody-aware text LLMs. By aligning speech inputs with text-style representations while explicitly preserving paralinguistic cues, the model achieves:

• Minimal Modality Gap: Speech-based performance approaches text-based performance, particularly in reasoning tasks.
• High Data Efficiency: The approach requires significantly less audio data (~1,000 hours) compared to prior methods, suggesting that the bottleneck is architectural alignment rather than data volume.
• Practical Usability: The method maintains the strong reasoning capabilities of the underlying TLM, making SLMs more reliable for general tasks without the catastrophic forgetting often seen in continual training on speech data.

The authors note limitations, including the current lack of a speech synthesis module (evaluations are text-output based), the use of non-streaming ASR (introducing latency), and the focus on a specific subset of paralinguistic attributes rather than the full spectrum of spoken cues. However, they posit that this input-side alignment strategy offers a promising and efficient direction for future SLM development.