Technical Summary: SpeechMapper

Problem Statement

Current approaches to integrating speech foundation models (SFM) with Large Language Models (LLMs) face significant computational and generalization challenges. State-of-the-art methods typically involve jointly instruction-tuning (IT) both the SFM encoder and the LLM on large-scale speech instruction data. This strategy is computationally intensive, requiring high-end GPUs and massive datasets, and is prone to overfitting specific tasks or prompts, thereby reducing the model's ability to generalize to unseen scenarios. Alternative methods, such as cascading ASR with an LLM, discard valuable acoustic cues and introduce latency, while mapping speech to discrete tokens suffers from quantization sensitivity and speed issues. There is a need for a resource-efficient method to bridge SFMs and LLMs that preserves acoustic information, minimizes overfitting, and scales effectively without requiring full-scale joint training.

Methodology

The authors propose SpeechMapper, a cost-efficient, two-stage training approach that maps speech embeddings directly into the target LLM's embedding space. This method decouples the heavy lifting of learning the speech-to-text mapping from the LLM's forward computations.

Architecture

The SpeechMapper projector sits between a frozen SFM (specifically seamless-m4t-v2-large) and a frozen LLM (e.g., Llama-3.1-8B-Instruct or EuroLLM-9B-Instruct).

• Input: Speech is encoded by the frozen SFM.
• Processing: The projector consists of two sequential blocks. Each block contains a convolutional layer (kernel size 6, stride 2) to compress the sequence, followed by six Transformer attention layers and a feed-forward layer.
• Output: The architecture progressively compresses the long speech sequence and maps it into the higher-dimensional LLM embedding space (4096 dimensions).

Training Procedure

The training is divided into two distinct stages:

Stage 1: Pretraining (Task-Agnostic)

• Objective: Learn to map SFM embeddings to LLM text embeddings using only Automatic Speech Recognition (ASR) data (LibriSpeech).
• Mechanism: The LLM's forward pass is frozen and not executed. Instead, the model minimizes the Mean Squared Error (MSE) between the projector's output and pre-computed LLM text embeddings of the target transcripts.
• Handling Length Mismatch: Since speech sequences are longer than text, the target text embeddings are padded with a special pad token to match the projector's output length.
• Loss Function: The loss is a weighted combination of MSE for words and the first pad token ($MSE_{word}$) versus the remaining pads ($MSE_{pad}$), plus a cosine embedding loss to emphasize critical dimensions. This forces the model to shift semantic information to the beginning of the sequence.
• Hardware: This stage can be run on affordable hardware (e.g., V100 GPUs) as it does not require LLM inference.

Stage 2: Rapid Adaptation (Instruction Tuning)

• Objective: Refine the pretrained projector for specific tasks (Speech Translation, Spoken Question Answering) or maintain zero-shot capabilities.
• Mechanism: The pretrained projector is attached to the full, frozen LLM. The system undergoes a brief instruction tuning (IT) stage (1,000 steps) on a single A100 GPU.
• Loss Function: A combination of Cross-Entropy (CE) loss (standard LLM generation loss) and the MSE loss defined in Stage 1.
  • $L_{stage2} = (1 - \sigma)L_{CE} + \sigma L_{MSE}$
• Adaptation Strategies:
  • Task-Specific: $\sigma = 0$ (Standard CE only).
  • Task-Agnostic (Zero-Shot): $\sigma > 0.8$ (High weight on MSE). This prevents the projector from overfitting to specific prompts or tasks, enabling the model to generalize to unseen tasks without further training.

Key Contributions

1. Decoupled Training: SpeechMapper separates the learning of the speech-to-text mapping from the LLM's forward computation, allowing the expensive pretraining phase to run on cheaper hardware and scale independently of LLM size.
2. Mitigation of Overfitting: By utilizing a high-weight MSE term during the brief adaptation stage, the model avoids the prompt and task overfitting common in jointly trained speech LLMs, resulting in more robust zero-shot generalization.
3. Efficiency: The approach achieves competitive performance with only 1,000 steps of adaptation (approx. 1.5 hours on an A100) and significantly less data compared to state-of-the-art baselines.
4. Versatility: A single pretrained block can serve as a foundation for both zero-shot speech LLMs and task-specialized models, demonstrating flexibility across Speech Translation (ST) and Spoken Question Answering (SQA).

Experimental Results

The authors evaluated SpeechMapper on Speech Translation (EuroParlST, CoVoST2) and Spoken Question Answering (SpokenSQUAD, LibriSQA), comparing it against BEST-IWSLT25-IF, a strong baseline from the IWSLT 2025 competition trained on 2,000 hours of data.

• Speech Translation (ST):

  • Zero-Shot: The Stage 1 pretrained model alone achieved strong performance. The Stage 2 zero-shot model (ASR CE+MSE) outperformed the Seamless ST baseline on several language pairs (e.g., en-fr, en-de) despite never being trained on ST data.
  • In-Domain: With minimal task-specific adaptation, SpeechMapper closed the gap with BEST-IWSLT25-IF, achieving comparable COMET scores with far fewer resources.
  • Language Adherence: The CE+MSE objective significantly improved target language adherence compared to CE-only, reducing unintended English outputs in zero-shot settings.

• Spoken Question Answering (SQA):

  • Zero-Shot: The best zero-shot SpeechMapper (EuroLLM backbone) rivaled the performance of the specialist BEST-IWSLT25-IF model across test sets, even outperforming it on LibriSQA Part II.
  • In-Domain: With 1,000 steps of task-specific IT, SpeechMapper surpassed the specialist baseline across all test sets and reached performance levels comparable to pipeline systems (ASR + LLM).

• Resource Efficiency: SpeechMapper achieved these results using only 960 hours of LibriSpeech for pretraining and a tiny subset of task data for adaptation, contrasting with the 2,000+ hours and multi-GPU training required by the baseline.

Significance and Claims

The paper claims that SpeechMapper offers a practical and scalable path to integrating speech capabilities into LLMs. By avoiding the computational bottleneck of joint training and the overfitting risks of standard instruction tuning, it enables the creation of robust, generalizable speech-LLMs with minimal data and compute requirements. The authors position this as a cost-effective solution that rivals specialized, resource-heavy models, making advanced speech-LLM integration accessible without the need for large-scale instruction tuning datasets or high-end GPU clusters.