Where Do Backdoors Live? A Component-Level Analysis of Backdoor Propagation in Speech Language Models

1. Problem Statement

Speech Language Models (SLMs) are composite systems typically built by cascading a pretrained audio encoder, a projection connector, and a pretrained Large Language Model (LM). While backdoor attacks in unimodal settings are well-studied, the behavior of backdoors in these multimodal, multitask pipelines remains obscure.
The paper addresses three critical questions:

1. Vulnerability: Can audio backdoors propagate effectively through the entire SLM pipeline across different tasks (ASR, emotion, age, gender)?
2. Component Role: Which specific components (encoder, connector, LM) are responsible for learning, sustaining, or erasing the backdoor?
3. Defensibility: Do standard defense mechanisms relying on the separability assumption (that poisoned samples form distinct clusters from benign ones in embedding space) hold true in multitask SLMs?

2. Methodology

Experimental Setup

• Architecture: The authors use a modified SpeechLLM pipeline: Audio Encoder $\rightarrow$ Connector (CNN) $\rightarrow$ LM (TinyLlama with LoRA).
• Tasks: Four distinct tasks were evaluated:
  • Automatic Speech Recognition (ASR)
  • Emotion Recognition
  • Speaker Gender Prediction
  • Speaker Age Prediction
• Datasets: LibriSpeech (ASR), CREMA-D (Emotion), and VoxCeleb2-AE (Gender/Age).
• Attack Strategy: A dirty-label poisoning strategy using a natural audio trigger (a 220ms typewriter click).
  • For ASR, the trigger is repeated throughout the audio.
  • For other tasks, a single trigger is applied.
  • Poisoning rates: 5% for ASR, 7.5% for others.

Analysis Framework

The authors designed a Component-Level Analysis to isolate the role of each module. They manipulated the training state (Train vs. Freeze) and the data source (Clean vs. Poisoned) for each component:

1. Single-Frozen Component Attack: Freezing one component (using clean weights) while training the others on poisoned data to see if the backdoor persists.
2. Single-Training Component Attack: Training only one component on poisoned data while freezing the others (using clean weights) to see if a single module can sustain the backdoor.
3. Propagation Attack: Using a previously poisoned component (frozen) in an otherwise clean pipeline to test if a "contaminated" pretrained model can infect a new system.
4. Embedding Analysis: Using Activation Clustering (AC) and t-SNE/PCA to visualize whether poisoned and clean samples form separable clusters in the embedding space.

3. Key Contributions & Results

A. Universal Vulnerability and Component Roles

• Full Pipeline Vulnerability: SLMs are highly susceptible to backdoors across all four tasks and various audio encoders (WavLM, HuBERT, Whisper, wav2vec).
• The Audio Encoder is the "Pillar":
  • The Audio Encoder is the most critical component. It is the only component capable of independently sustaining a backdoor when trained in isolation (Single-Training Attack).
  • If the encoder is poisoned, the backdoor persists even if the connector and LM are trained on clean data.
• Connector vs. LM:
  • The Connector plays a secondary role; it can encode triggers but is less robust than the encoder.
  • The LM cannot independently learn a backdoor. If the encoder and connector are clean, poisoning the LM alone fails to produce a backdoor.
• Propagation Risk: A previously poisoned Audio Encoder can successfully propagate a backdoor into a clean pipeline (specifically for the Emotion task, achieving ~63.5% Attack Effectiveness Rate). This highlights a severe security risk in "plug-and-play" reuse of pretrained components.

B. Task-Dependent Robustness

The persistence of the backdoor varies significantly by task:

• Emotion Recognition: Highly robust. The backdoor survives even when the encoder is poisoned and the rest of the pipeline is retrained on clean data. This is attributed to the "soft" nature of emotion labels (multiple valid interpretations for the same audio).
• ASR, Gender, and Age: These tasks are less robust to "clean retraining." When a poisoned encoder is used but the connector is retrained on clean data, the backdoor is erased.
  • Reasoning: The target labels for ASR ("This is a malicious sentence") and static attributes (Gender/Age) strongly conflict with the ground truth of the clean data. The connector learns to override the encoder's malicious signal to minimize loss on the clean distribution.

C. Failure of Separability Assumption

The paper challenges a common defense assumption: that poisoned samples are linearly separable from benign samples in the embedding space.

• Multitask Dominance: In multitask embeddings, task-specific features (e.g., gender) dominate the representation space.
• Clustering Failure: Standard Activation Clustering (AC) with $k=2$ failed to separate poisoned samples. Instead, samples clustered by gender (the dominant feature), effectively hiding the backdoor.
• Complexity: While increasing clusters to $k=4$ could isolate poisoned samples, this is impractical for real-world defenses as the optimal $k$ is unknown. Furthermore, poisoned samples often do not form a distinct cluster but are intermingled with benign samples of the same task class.

4. Significance and Implications

1. Rethinking SLM Security: SLMs cannot be treated as simple extensions of unimodal models. Their composite nature creates unique vulnerabilities where the Audio Encoder acts as the primary vector for backdoor injection.
2. Supply Chain Risks: The "Propagation Attack" results demonstrate that reusing a poisoned pretrained encoder (a common practice in the industry) can compromise an entire downstream system, even if the subsequent components are trained on clean data.
3. Defense Limitations: Defenses relying on data filtering (removing outliers based on embedding separability) are likely ineffective for multitask SLMs. The backdoor signal is diluted and masked by other strong task features (like speaker identity), making it indistinguishable from benign data without prior knowledge of the attack structure.
4. Task Sensitivity: Security measures must be task-aware. Emotion recognition is inherently more vulnerable to persistent backdoors than ASR or demographic prediction due to the ambiguity of the output space.

Conclusion

The paper concludes that backdoor behavior in SLMs is defined not just by where the attack lives, but by how it transforms through the pipeline. The Audio Encoder is the critical point of failure, and the multitask nature of these models fundamentally breaks the assumptions required for current separability-based defenses. Future work must focus on component-level auditing and defenses that account for the complex, non-uniform encoding of backdoors in shared embedding spaces.