In-Context Learning in Speech Language Models: Analyzing the Role of Acoustic Features, Linguistic Structure, and Induction Heads

1. Problem Statement

While In-Context Learning (ICL) is well-studied in text-only Large Language Models (LLMs), its mechanisms in Speech Language Models (SpeechLMs) remain largely unexplored. SpeechLMs must process both acoustic and linguistic information simultaneously. The core research questions are:

• How do acoustic features (e.g., speaking rate, pitch, intensity) and linguistic features (e.g., syntax, lexical overlap) affect ICL performance in SpeechLMs?
• To what extent do SpeechLMs mimic the acoustic characteristics of demonstration speech in their output?
• Do induction heads (attention mechanisms responsible for copying/prefix-matching in text-based ICL) play a causal role in speech-based ICL, and are there modality-specific induction heads?

2. Methodology

Model and Task

• Model: The study uses SPIRITLM, a model based on Llama-2-7B, continuously pre-trained on interleaved sequences of speech (discrete tokens from HUBERT) and text.
• Task: Text-to-Speech (TTS). The model is prompted with text-speech pairs (demonstrations) and asked to generate speech for a target text. This allows for dual evaluation:
  1. Task Accuracy: How well the model infers the correct spoken content (transcription accuracy).
  2. Acoustic Mimicry: How well the model replicates the acoustic style (rate, pitch, intensity) of the demonstrations.

Experimental Design

The authors constructed prompts using the PRIME-LM corpus, which provides controlled pairs of demonstration ("prime") and target sentences. They systematically manipulated four factors:

1. Number of Demonstrations: 0 to 5 shots.
2. Syntactic Structure: Congruent (Active/Active, Passive/Passive) vs. Incongruent (Active/Passive, Passive/Active).
3. Linguistic Overlap: Controlled overlap between demo and target, ranging from "Core" (no overlap) to "Lexical Overlap" (nouns, verbs, function words, all words) and "Semantic Similarity."
4. Acoustic Manipulations:
   • Speaking Rate: Fast (2x), Slow (0.5x), Regular.
   • Pitch Range: Flattened vs. Regular.
   • Intensity: Low (30 dB) vs. Regular (~67 dB).

Evaluation Metrics

• Task Performance: Measured via Word Error Rate (WER) and Content Word Recall (percentage of correctly predicted content words), using an external ASR (Whisper) for transcription.
• Acoustic Mimicry: Measured by extracting speaking rate, pitch statistics, and intensity from the generated audio using Praat.
• Mechanistic Analysis:
  • Induction Head Identification: Computed "prefix-matching scores" for all attention heads on both speech and text tokens.
  • Ablation Study: Systematically zeroed out the output vectors of specific head groups (Top-k speech-prefix heads, text-prefix heads, unique heads, random heads) to measure causal impact on ICL performance.

3. Key Results

A. Linguistic and Acoustic Effects on Performance

1. Demonstration Count: A single demonstration is sufficient to trigger ICL; performance plateaus after 2 demonstrations.
2. Linguistic Overlap: Lexical overlap (especially function words and all words) significantly boosts performance. Semantic similarity alone does not improve performance, contrasting with findings in text-only ICL.
3. Syntax: Syntactic congruence between demo and target has little effect. However, passive targets consistently yield better performance than active targets, primarily due to more accurate generation of function words.
4. Acoustic Impact:
   • Speaking Rate: Fast speaking rate in demonstrations significantly degrades ICL performance (likely due to compressed information). Slow speaking rate has a neutral or slightly positive effect.
   • Pitch & Intensity: Variations in pitch range and intensity have no significant impact on task performance.

B. Acoustic Mimicry

• Speaking Rate: The model successfully mimics the speaking rate of demonstrations (outputs are faster after fast demos, slower after slow demos), though the effect is stronger with slow rates and requires more shots for fast rates to be clear.
• Pitch & Intensity: The model fails to consistently mimic flattened pitch or low intensity. These features show little to no transfer from demo to output.

C. Mechanistic Role of Induction Heads

• Identification: The study identified distinct groups of heads: those specializing in speech tokens, those in text tokens, and general-purpose heads. There is significant overlap, but ~25 of the top 50 heads are unique to each modality.
• Ablation Findings:
  • Ablating Speech-Prefix Heads (Top-k) causes a drastic drop in ICL performance.
  • Ablating Text-Prefix Heads also degrades performance, but the drop is less severe than for speech heads in this specific TTS task.
  • Ablating Unique Speech-Prefix Heads (those not shared with text) causes a massive performance collapse, proving their causal necessity.
  • Ablating Random Heads or Non-Prefix Heads has negligible or even slightly positive effects (suggesting non-prefix heads may introduce noise).

4. Key Contributions

1. First Systematic Analysis of Speech ICL: This is the first study to disentangle the effects of acoustic vs. linguistic features on ICL in SpeechLMs, specifically for TTS.
2. Acoustic Feature Hierarchy: Establishes that temporal features (speaking rate) are critical for both task performance and mimicry, while frequency/amplitude features (pitch, intensity) are largely ignored by the model in the ICL context.
3. Mechanistic Validation in Speech: Demonstrates that induction heads are the causal mechanism for ICL in SpeechLMs, mirroring findings in text-only LLMs.
4. Modality Specialization: Reveals that while many induction heads are modality-agnostic, a significant subset specializes in speech tokens, and these specific heads are critical for speech-based ICL.

5. Significance and Implications

• Model Design: The findings suggest that future SpeechLMs should prioritize mechanisms that handle temporal token distribution (speaking rate) and lexical repetition, as these are the primary drivers of ICL success.
• Interpretability: The confirmation that induction heads drive speech ICL validates the application of mechanistic interpretability tools (like ablation studies) to multimodal models.
• Limitations & Future Work: The study highlights that current SpeechLMs struggle to mimic complex acoustic prosody (pitch/intensity) via ICL, suggesting a gap in how these models represent acoustic style. Future work should investigate how induction heads handle longer spans of tokens and compare the strength of these heads between text-only and speech-text models to explain performance gaps in reasoning tasks.

In summary, the paper provides a rigorous empirical and mechanistic explanation of how SpeechLMs learn from context, identifying speaking rate and lexical overlap as the primary drivers of success, and induction heads (specifically those specialized for speech) as the underlying functional circuit.