💻 computer science

Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalization

The paper proposes Speech-FT, a novel two-stage fine-tuning framework that combines representational drift reduction with weight-space interpolation to enhance task-specific performance while preserving and even improving cross-task generalization compared to existing regularization methods.

Original authors: Tzu-Quan Lin, Wei-Ping Huang, Hao Tang, Hung-yi Lee

Published 2026-04-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tzu-Quan Lin, Wei-Ping Huang, Hao Tang, Hung-yi Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Specialist Trap"

Imagine you have a brilliant, well-traveled guide (the Pre-trained Model) who knows a little bit about everything: geography, history, cooking, and sports. This guide was trained on a massive library of books (unlabeled data) and is great at answering general questions.

Now, you want to hire this guide to become a Master Chef (Fine-tuning for a specific task). You give them a cookbook and let them practice.

The Issue: As the guide practices cooking intensely, they start to forget everything else. They become so obsessed with recipes that they forget how to navigate a city or talk about history. If you ask them a question about the weather or a famous landmark, they might give you a terrible answer because their brain has been "rewired" too specifically for cooking.

In the world of AI, this is called Representational Drift. When we fine-tune a speech model to do one thing (like transcribing speech), it often loses its ability to do other things (like recognizing emotions or identifying speakers).

The Old Solutions: Playing it Safe (But Failing)

Researchers tried to fix this by telling the guide: "Don't change your brain too much." They used Weight-Space Regularization, which is like putting a leash on the guide.

The Leash: "You can learn cooking, but you can't move your brain more than 5 inches away from where it started."
The Flaw: The guide learns cooking very poorly because they are too scared to move, and they still forget the other skills because the "leash" doesn't actually stop their brain from changing its thinking style, just its physical location.

The New Solution: Speech-FT (The "Two-Step Dance")

The authors propose a new method called Speech-FT. Think of it as a two-step dance that lets the guide become a great chef without forgetting how to be a guide.

Step 1: The "Stable" Warm-up

First, the guide practices cooking, but with a special rule:

Freeze the Foundation: The guide's basic senses (hearing the sizzle, feeling the heat) are locked in place. These are the "low-level features" that are useful for everything, not just cooking.
Learn the Head: The guide learns the specific recipes (the task-specific part) first without messing up their core senses.
Result: The guide gets better at cooking, but they haven't completely scrambled their brain yet.

Step 2: The "Mixing" (Interpolation)

This is the magic trick. The authors take two versions of the guide:

Version A: The original, well-traveled guide (Pre-trained).
Version B: The guide who just finished the cooking warm-up (Fine-tuned).

Instead of picking one or the other, they blend them together.

Imagine taking 75% of the original guide's brain and mixing it with 25% of the new cooking skills.
The Result: You get a guide who is an expert chef but still remembers how to navigate cities, talk about history, and recognize faces.

Why This Works (The "Feature Similarity" Secret)

The paper discovered something surprising:

Old Method (Leash): Tried to keep the guide's brain in the same physical spot, but the way they thought changed.
New Method (Mixing): Allowed the guide's brain to move a lot, but the mixing step pulled the thinking style back to the original.

It's like taking a photo of a landscape (the original model) and a photo of a sunset (the fine-tuned model). If you just try to keep the camera in the exact same spot, you miss the sunset. But if you take the sunset photo and blend it with the landscape photo, you get a beautiful picture that has both the scenery and the sunset.

The Results: What Did They Find?

The researchers tested this on several famous speech models (like HuBERT and wav2vec 2.0) and found:

Better at Everything: Models using Speech-FT got better at the specific task they were trained on (like speech recognition) and kept their ability to do other tasks (like identifying speakers or emotions).
Beating the Competition: It worked better than the "leash" method (regularization) and better than "early stopping" (just quitting training early).
Cross-Language Magic: They tested it on a model trained in English and then taught it Chinese. Speech-FT allowed the model to learn Chinese without forgetting how to speak English.
Multiple Tasks: They even tested it where the model had to learn multiple things at once (like recognizing phonemes AND speakers). Speech-FT helped the model handle all of them without getting confused.

In a Nutshell

Speech-FT is a clever way to teach a smart AI a new skill without making it forget all its old skills. Instead of forcing the AI to stay rigid or letting it go wild, it lets the AI learn, and then gently blends the new knowledge back with the old wisdom. The result is a model that is both a specialist and a generalist.

1. Problem Statement

Fine-tuning pre-trained speech representation models (e.g., HuBERT, wav2vec 2.0) on specific downstream tasks often leads to a degradation in cross-task generalization. While fine-tuning improves performance on the target task, it frequently causes representational drift, where the model becomes overly specialized and loses the generalizable features learned during self-supervised pre-training.

Existing solutions attempt to mitigate this by:

Weight-space regularization: Penalizing large deviations from pre-trained weights.
Parameter-efficient fine-tuning (PEFT): Using adapters (e.g., LoRA) to limit parameter updates.
Early stopping: Halting training before weights diverge too far.

Limitations: The authors argue that these methods fail because constraining weight space does not necessarily preserve feature space similarity. Models can undergo significant weight changes while retaining feature similarity, or conversely, small weight changes can drastically alter features. Consequently, existing baselines often lose the ability to generalize to unrelated tasks.

2. Methodology: Speech-FT

The authors propose Speech-FT, a novel two-stage framework designed to enhance task-specific performance while restoring cross-task generalization.

Stage 1: Stable Fine-Tuning (Stable-FT)

This stage aims to minimize representational drift during the initial adaptation to a new task.

Freezing the Downsampling Module: The initial convolutional encoder (which extracts low-level frequency features) is frozen to preserve crucial low-level acoustic patterns.
Warm-up Strategy: For the first $\beta\%$ (set to 10%) of training steps, only the task prediction head ( $D$ ) is updated. This allows the model to adapt its output layer without immediately disrupting the pre-trained feature extractor.
Full Update: After the warm-up, the entire model (except the frozen downsampling module) is fine-tuned on the target task.

Stage 2: Weight-Space Interpolation (Model Merging)

After obtaining the fine-tuned model weights ( $\theta'$ ), Speech-FT merges them with the original pre-trained weights ( $\theta_0$ ) using linear interpolation:
$\hat{\theta} = (1 - \alpha) \cdot \theta_0 + \alpha \cdot \theta'$

$\alpha$ (Scaling Factor): A hyperparameter (set to 0.25 in experiments) that balances the pre-trained general knowledge and the task-specific knowledge.
Mechanism: This step acts as a form of model merging. It effectively injects the task-specific "task vector" ( $\tau = \theta' - \theta_0$ ) back into the pre-trained base, but scaled down. Theoretically, this leverages Layerwise Linear Feature Connectivity (LLFC), ensuring that the interpolated model's features remain linearly connected to the pre-trained features, thereby restoring feature similarity.

Extension to Multiple Tasks

The framework supports multiple fine-tuning scenarios:

Multitask Fine-Tuning: Joint training on multiple tasks.
Linear/TIES Merging: Merging multiple independently fine-tuned models before interpolating with the pre-trained model.
Sequential Fine-Tuning: Iteratively applying Speech-FT to a sequence of tasks.

3. Key Contributions

Novel Framework: Introduction of Speech-FT, a two-stage process combining "Stable Fine-Tuning" and "Weight-Space Interpolation" to solve the generalization vs. specialization trade-off.
Feature Similarity Insight: Demonstration that feature similarity is a more critical metric for generalization than weight-space proximity. Speech-FT achieves higher feature similarity to the pre-trained model than regularization-based methods, even though it allows larger weight updates.
Comprehensive Evaluation: Extensive experiments across diverse models (HuBERT, wav2vec 2.0, DeCoAR 2.0, WavLM Base+) and tasks (ASR, Phoneme Recognition, Speaker ID, Emotion Recognition).
Superiority over Baselines: Proving that Speech-FT outperforms weight-space regularization, LoRA, DoRA, and early stopping in maintaining cross-task generalization.

4. Experimental Results

The authors evaluated Speech-FT on the SUPERB benchmark and various specific datasets.

Supervised Fine-Tuning:
- Speech-FT consistently achieved higher SUPERB scores than both the pre-trained baseline and Stable-FT (which represents $\alpha=1.0$ ).
- Example (HuBERT on ASR): Reduced Phone Error Rate (PER) from 5.17% to 3.94% and Word Error Rate (WER) from 6.38% to 5.75%, while simultaneously increasing Speaker Identification accuracy from 81.86% to 84.11%.
- In contrast, standard fine-tuning (Stable-FT) often degraded performance on unrelated tasks (e.g., fine-tuning on Speaker ID caused a massive drop in Phoneme Recognition).
Unsupervised Fine-Tuning (Cross-Lingual):
- When continuing pre-training on Chinese (AISHELL-3) or Spanish data, Stable-FT caused a severe drop in English performance (representational forgetting).
- Speech-FT successfully adapted to the new language while preserving English capabilities, achieving a balanced trade-off.
Comparison with Baselines:
- Weight-Space Regularization: Improved target tasks but degraded unrelated tasks significantly (lower SUPERB scores than pre-trained models).
- LoRA/DoRA: Despite being parameter-efficient, they failed to maintain feature similarity and suffered from poor cross-task generalization.
- Early Checkpoints: Stopping training early (e.g., at 20% steps) was insufficient to prevent generalization loss.
Ablation & Analysis:
- Hyperparameter $\alpha$ : Performance peaked at $\alpha=0.25$ , suggesting that a small injection of task-specific knowledge is optimal.
- LLFC Verification: The interpolated models satisfied Layerwise Linear Feature Connectivity, confirming that the interpolation path preserves the geometry of the pre-trained representations.

5. Significance

Practical Efficiency: Speech-FT requires no additional computational cost compared to standard fine-tuning (the interpolation step is a simple linear operation) and does not require complex architectural changes like adapters.
Paradigm Shift: It challenges the conventional wisdom that limiting weight updates is the only way to preserve generalization. Instead, it shows that post-hoc interpolation is a more effective mechanism for restoring the pre-trained feature space.
Broad Applicability: The method is model-agnostic and effective across different pre-training objectives (contrastive, generative) and languages, offering a robust solution for refining speech representations in real-world scenarios where models must adapt to new domains without forgetting prior knowledge.

In conclusion, Speech-FT provides a simple yet powerful solution for the "catastrophic forgetting" problem in speech representation learning, enabling models to become specialists in specific tasks without sacrificing their generalist capabilities.