Technical Summary: Reducing Prompt Sensitivity in LLM-Based Speech Recognition Through Learnable Projection

1. Problem Statement

While Large Language Model (LLM)-based Automatic Speech Recognition (ASR) has emerged as a promising architecture—connecting speech foundation models to LLMs via a speech-to-LLM projector—current implementations rely heavily on fixed, manually defined prompts for both training and inference.

The paper identifies a critical, underexplored vulnerability in this design: prompt sensitivity. The authors demonstrate that:

• The choice of prompt significantly impacts ASR performance (measured in Word Error Rate, WER).
• No single prompt performs optimally across all datasets or scenarios.
• Minor variations in wording or the placement of speech embeddings within the prompt can lead to drastic differences in transcription quality.
• In some cases, specific prompts perform worse than using no prompt at all (an "empty" prompt), introducing instability and limiting the robustness of practical ASR systems.

2. Methodology

2.1. Base Architecture

The study utilizes the SLAM-ASR architecture as a foundation. This model consists of:

• Speech Encoder: WavLM-large (trained on 94k hours of unlabeled data).
• Speech Projector: A lightweight module ($sp(\cdot)$) that maps downsampled audio features to the LLM input space.
• LLM: Vicuna-7B (fine-tuned from Llama).
• Training Strategy: The speech encoder and LLM are frozen; only the speech projector is trained.

2.2. Proposed Solution: The Prompt Projector

To address prompt sensitivity without modifying the underlying LLM or speech encoder, the authors propose a Prompt Projector module, denoted as $pp(\cdot)$.

• Concept: Inspired by the success of the speech projector, the prompt projector learns a common projection function to transform fixed, manually defined prompt embeddings into a more effective region of the LLM input space.
• Architecture: It shares the same architecture as the speech projector (a two-layer linear network with ReLU activation) but operates on LLM token embeddings rather than speech features.
• Training Protocol:
  1. Train the base "vanilla" model (frozen encoder/LLM, learned speech projector).
  2. Freeze all components of the trained base model.
  3. Train only the new prompt projector ($pp(\cdot)$) on top of the frozen model.
  • Note: The authors explicitly tested unfreezing the underlying models but found it led to unstable training and degraded performance.

2.3. Experimental Setup

• Datasets: Four diverse datasets were used:
  • LibriSpeech (LS-C/LS-O): Read English audiobooks.
  • CallHome (CH): Spontaneous telephone conversations.
  • AMI: Multi-modal meeting recordings.
  • ContactCenter (CC): Proprietary domain-specific conversations (health/finance).
• Prompt Variations: Ten distinct prompt templates were evaluated, ranging from the standard "base" prompt to variations in wording and embedding placement, plus an "empty" prompt (containing only speech embeddings).
• Baselines: Comparisons included vanilla models with different prompts, models with LoRA fine-tuning, and recent state-of-the-art LLM-based ASR systems.

3. Key Results

3.1. Impact of Manual Prompts

The evaluation of the vanilla model across ten prompts revealed high variability:

• Performance Variance: Different prompts yielded significantly different WERs even under identical model and data conditions. For example, on the ContactCenter dataset, the WER ranged from 11.81% to 13.00% depending solely on the prompt.
• Suboptimal Defaults: The standard "base" prompt often underperformed compared to other manually crafted prompts and, in some cases (e.g., CallHome), performed worse than the "empty" prompt.
• Dataset Dependency: A prompt that excelled on one dataset (e.g., CallHome) often underperformed on another (e.g., AMI), confirming that no single prompt is universally optimal.

3.2. Effectiveness of the Prompt Projector

The addition of the prompt projector ($pp(\cdot)$) yielded consistent improvements:

• Robustness: The projector significantly reduced performance variance. It mitigated the negative impact of suboptimal prompts, ensuring that even the "worst" manual prompt, when augmented with $pp(\cdot)$, outperformed the best manual prompt used without the projector.
• Performance Gains: Across all four datasets, the $pp(\cdot)$ extension consistently lowered WER compared to the vanilla baseline.
  • Example: On LibriSpeech-Clean, the best manual prompt achieved 2.72% WER, while the "base" prompt achieved 3.09%. With $pp(\cdot)$, the "base" prompt improved to 2.34%, and the best prompt improved to 2.28%.
• Statistical Significance: Paired statistical tests confirmed that the improvements were statistically significant ($p < 0.05$) across all datasets.
• Comparison to SOTA: The proposed method, particularly when combined with LoRA fine-tuning, achieved competitive or superior results compared to recent state-of-the-art systems (e.g., Qwen-Audio, SALMONN) on specific benchmarks.

4. Significance and Contributions

The paper claims the following contributions and significance:

1. First Systematic Analysis: It provides the first comprehensive study of prompt sensitivity in LLM-based ASR, revealing that prompt choice is a critical, often overlooked factor in system performance.
2. Novel Architecture: It introduces the Prompt Projector, a simple, model-agnostic extension that learns to align prompt embeddings with the LLM's input space without requiring changes to the underlying LLM or speech encoder.
3. Robustness without Complexity: The approach improves robustness and reduces performance variance without introducing complex prompt-engineering parameters or tokens. It functions as a "drop-in" extension to existing architectures.
4. Diagnostic Insight: The findings suggest that future research should include an "empty prompt" baseline as a diagnostic tool to detect poorly performing manual prompts early in development.

5. Limitations and Future Work

The authors maintain a modest stance regarding the scope of their work:

• Initial Exploration: This is presented as a first step toward understanding prompt robustness.
• Comparison Gap: The paper does not directly compare the Prompt Projector with soft-prompt learning methods (which learn task-specific tokens), noting that a fair comparison would require extensive hyperparameter tuning across multiple prompts and datasets, which is beyond the current scope.
• Generalization: Open questions remain regarding whether a single projector can generalize across different tasks, languages, or LLM architectures.
• Scope: The current work focuses strictly on transcription; applicability to other downstream tasks is a subject for future investigation.