Technical Summary: Unlocking Speech–Text Compositional Powers via SPEECHCOMBINE

Problem Statement

Instruction tuning for Speech Language Models (SLMs) faces significantly greater challenges than for text-based Large Language Models (LLMs). While text LLMs follow a standard training paradigm involving pre-training and instruction tuning, SLMs must master a new modality (speech) alongside a broader spectrum of instructions, including speech understanding (e.g., emotion detection) and speech generation (e.g., expressive synthesis).

Current SLM training approaches largely replicate the text LLM pipeline: pre-training on massive speech corpora followed by supervised fine-tuning (SFT) or reinforcement learning (RL) on speech instruction datasets. This strategy suffers from two critical bottlenecks:

1. Data Inflation: Speech sequences are significantly longer than text sequences (e.g., a 5-token text sentence expands to 60–200 speech tokens), making it difficult to scale effective dataset sizes.
2. Catastrophic Forgetting: Extensive training on speech tokens often causes the model to forget the knowledge and reasoning capabilities of its text LLM base, forcing the model to re-learn these skills in a harder-to-learn speech representation.

Existing attempts to mitigate this, such as freezing the text LLM and training only adapters, often fail to support speech-specific instructions (understanding and generation), limiting the model to text-oriented tasks.

Methodology: SPEECHCOMBINE

The paper proposes SPEECHCOMBINE, an instruction-following SLM trained without iterative instruction tuning. The core hypothesis is that instruction-following capabilities encoded in a text LLM can be directly transferred to a speech-adapted model via weight combination, inspired by recent advances in model merging.

1. Framework and Weight Combination

The method operates in three stages within the parameter space:

• Base Model ($\theta_{base}$): A pre-trained text LLM.
• Instruction-Tuned Model ($\theta_{inst}$): The text LLM fine-tuned for instruction following.
• Speech-Adapted Model ($\theta_{speech}$): The base model ($\theta_{base}$) subjected to a single round of continuous pre-training on speech data.

The final model parameters ($\theta_{SC}$) are constructed by linearly combining the weight differences:
$$\theta_{SC} = \theta_{base} + \lambda \Delta\theta_{speech} + \Delta\theta_{inst}$$
Where:

• $\Delta\theta_{inst} = \theta_{inst} - \theta_{base}$ encodes instruction-following behavior.
• $\Delta\theta_{speech} = \theta_{speech} - \theta_{base}$ captures speech modality knowledge.
• $\lambda \in (0, 1]$ is a soft weight balancing the integration of speech knowledge with the preservation of text capabilities.

Crucial Distinction: The speech pre-training must start from $\theta_{base}$, not $\theta_{inst}$. Starting from the instruction-tuned model disrupts the compositional structure required for successful merging.

2. Pre-Training Data Structure

To achieve the dual goals of familiarizing the model with speech token sequences and understanding speech information (content and prosody), the authors design a specific pre-training data structure:
[S1 cap] [S1 text] [S1 speech] [S1 cap] ...

• [cap]: Natural language descriptions of speech attributes (content, pitch, rate, emphasis, emotion).
• [text]: The exact transcription of the speech.
• [speech]: Discrete prosody tokens representing the speech utterance.

The inclusion of [text] alongside [speech] is essential to anchor the transfer of text-based instruction-following capabilities. The [cap] section is randomized (appearing before, after, or omitted) to teach generation, understanding, and multi-turn patterns.

3. Tokenization and Inference

• Tokenization: The authors use a prosody tokenization scheme (based on Qian et al., 2025) that encodes only prosodic features (pitch, duration, energy) rather than full waveforms, significantly reducing token length and redundancy.
• Inference: The model follows a template similar to the text LLM but includes [speech] sections. An external ASR system transcribes input speech into the [text] section. "Format forcing" techniques are employed to stabilize the generation of speech tokens and ensure the model adheres to the required output structure.
• Long Thinking: The model inherits "long thinking" (Chain of Thought) capabilities from the text LLM simply by modifying the answer template to include a <thought> section, without additional training.

Key Contributions

1. Instruction-Free SLM Training: Demonstrates that an SLM with strong instruction-following capabilities across text, speech understanding, and speech generation can be built using only a single round of pre-training (30k hours of data) and weight combination, eliminating the need for massive speech instruction-tuning datasets.
2. Compositional Generalization: Shows that instruction-following capabilities ($\Delta\theta_{inst}$) generalize to the speech domain when combined with speech adaptation ($\Delta\theta_{speech}$), enabling the model to handle unseen speech-specific tasks (e.g., emotion detection, emphasis generation) without explicit training on those tasks.
3. Preservation of Text Capabilities: The strategy successfully preserves the original knowledge and reasoning skills of the text LLM, avoiding the catastrophic forgetting typical of standard SLM training pipelines.
4. Efficiency: Achieves state-of-the-art or competitive performance using less than 1% of the training data required by existing SLMs (e.g., FUN-AUDIO-CHAT trained on millions of hours).

Experimental Results

The authors evaluate SPEECHCOMBINE on two categories of tasks: Shallow Combination (text-oriented tasks in speech modality) and Deep Combination (speech-specific tasks).

• Text-Oriented Tasks (QA, Reasoning): SPEECHCOMBINE performs on par with or better than the "ASR + Text LLM" baseline (which suffers no forgetting) and significantly outperforms other SLMs trained on massive datasets (e.g., GLM-4-Voice, Step-Audio). It achieves top or second-best results across datasets like OpenbookQA, GSM8k, and MLC.
• Speech Understanding (Emotion, Emphasis):
  • Emotion: Achieves moderate performance, limited by the scarcity of emotion captions in the pre-training data.
  • Emphasis Detection: Achieves state-of-the-art performance with substantial margins over baselines, demonstrating the model's ability to infer emphasis from prosodic cues.
• Speech Generation (Expressive, Emphasis):
  • Expressive Generation: Achieves the second-best performance on emotion generation.
  • Emphasis Generation: Achieves the best performance among comparable models.
  • Notably, these capabilities emerge without task-specific training, relying solely on the compositional transfer of skills.
• Ablation Studies:
  • $\lambda$ Weight: A sweet spot (e.g., $\lambda=0.85$) balances text preservation and speech acquisition.
  • Data Structure: Removing [cap] or [text] significantly degrades performance, confirming their necessity for knowledge transfer and anchoring.
  • Long Thinking: Enabling the thinking mode improves performance on complex reasoning and speech tasks.

Significance and Claims

The paper claims that SPEECHCOMBINE offers a new, scalable direction for SLM training that avoids the "undue ordeal of inefficient data piling." By leveraging model merging, the approach allows SLMs to rapidly inherit emerging capabilities from evolving text LLMs without requiring re-training on massive speech datasets.

The authors acknowledge limitations, including the reliance on external ASR for inference, the lack of timbre/voice quality encoding in the current tokenization scheme, and the need for format forcing to ensure output stability. However, they posit that this paradigm provides a robust foundation for building powerful, instruction-following speech models that are both data-efficient and capable of complex reasoning.