💬 NLP

Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic Forgetting in Automatic Speech Recognition

This paper proposes a simple yet effective weight averaging method, optionally enhanced with knowledge distillation, to overcome catastrophic forgetting in end-to-end automatic speech recognition by achieving high performance on both old and new tasks.

Original authors: Steven Vander Eeckt, Hugo Van hamme

Published 2026-01-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Steven Vander Eeckt, Hugo Van hamme

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "One-Task" Brain

Imagine you hire a brilliant speech-recognition assistant (an AI) who is an expert at understanding American English. You are happy with it. Then, you decide you need it to understand Scottish accents. So, you train it on Scottish data.

The Catch: As soon as you teach it Scottish, it starts forgetting how to speak American English. This is called Catastrophic Forgetting. It's like a student who studies for a history exam so intensely that they completely forget everything they learned in math class the week before.

In the world of AI, this is a huge problem. If you want an AI to learn many languages or dialects over time, you usually have to save all the old data (like recording thousands of hours of old conversations) to re-teach it. This is expensive, slow, and sometimes impossible due to privacy rules.

The Solution: The "Blending" Trick

The authors of this paper propose a surprisingly simple solution called Weight Averaging.

Think of the AI's "brain" (its internal settings or "weights") as a recipe.

The Old Recipe: You have a perfect recipe for American English (let's call it Recipe A).
The New Training: You tweak Recipe A to learn Scottish, creating a new version, Recipe B.
The Mistake: If you just use Recipe B, you lose the American flavor. If you just use Recipe A, you can't understand Scottish.
The Fix: Instead of choosing one, you take a spoonful of Recipe A and a spoonful of Recipe B and mix them together to create Recipe C.

By mathematically averaging the "old" brain and the "newly trained" brain, the AI keeps the knowledge of the old task while still being good at the new one.

How They Tested It

The researchers tested this idea in two scenarios:

The Dialect Test (Monolingual): They taught the AI six different English dialects (US, England, Australia, India, Scotland, Ireland) one by one.
- Result: The "Blending" method worked incredibly well. It remembered the first dialect almost perfectly while learning the sixth, beating all other methods that tried to save old data in a memory bank.
The Language Test (Multilingual): They taught it English, then Dutch, Swedish, Polish, and Russian. These are much more different from each other than dialects are.
- Result: Forgetting is usually worse here. However, the "Blending" method still crushed the competition. It learned the new languages without wiping out the old ones, even though it didn't store a single extra sentence of data.

Two Ways to Mix the Soup

The paper suggests two ways to do this averaging:

The 50/50 Mix: Just take equal parts of the old and new model. This works well, but sometimes the AI forgets a tiny bit of the very first task.
The "Equal History" Mix: This is the winner. Imagine you have learned 5 tasks. Instead of just mixing the current one with the very first one, you mix the current model with all the versions of the model you've created so far. This ensures that the first task gets just as much attention as the last one.

The "Teacher" Bonus (Knowledge Distillation)

Sometimes, just mixing the recipes isn't enough, especially when the new task is very different (like switching from English to Russian).

The authors added a "Teacher" step. Before mixing the recipes, they let the "Old AI" (the Teacher) watch the "New AI" learn.
The Teacher says, "Hey, when you hear this sound, remember that it used to mean X, not Y."
This helps the new model hold onto old knowledge better during the learning process. The paper calls this LWFA (Learning Without Forgetting + Averaging). It proved especially helpful when learning very different languages.

The Big Takeaway

Usually, to stop an AI from forgetting, you need a massive "memory bank" of old data to rehearse with. This paper shows that you don't need that memory bank at all.

By simply taking the average of the AI's brain before and after learning a new task, you can keep the old skills alive while mastering the new ones. It's simple, requires no extra storage space, and works better than complex methods that try to save data.

In short: Don't throw away the old recipe when you learn a new one. Just mix them together, and you get the best of both worlds.

Technical Summary: Weight Averaging for Catastrophic Forgetting in ASR

Problem Statement
Adapting trained Automatic Speech Recognition (ASR) models to new tasks—such as new speakers, dialects, or languages—often results in Catastrophic Forgetting (CF), where the model loses performance on previously learned tasks. This severely limits the ability to build continually learning ASR systems or to leverage newly available data without retraining on all historical data, which is computationally and resource-intensive. While Continual Learning (CL) methods exist, many rely on rehearsal (storing samples from past tasks), which raises privacy concerns, or require task-specific parameters that complicate inference. Existing regularization-based methods have shown limited effectiveness in End-to-End (E2E) ASR.

Methodology
The authors propose Weight Averaging, a simple yet effective CL method that operates in two distinct steps after a new task $t$ becomes available:

Adaptation: The model with parameters $\theta_{t-1}$ (trained on previous tasks) is adapted to the new task $t$ . This adaptation can be performed via:
- Naive Fine-Tuning: Standard training on the new task data.
- Learning Without Forgetting (LWF): A regularization-based approach that adds a knowledge distillation loss ( $L_{lwf}$ ) to the standard cross-entropy and CTC losses. This loss forces the new model to mimic the output probabilities of the old model ( $\theta_{t-1}$ ) on the new task's data, thereby preserving old knowledge without storing past samples.
- The resulting adapted parameters are denoted as $\hat{\theta}_t$ .
Weight Averaging: The final parameters $\theta_t$ are computed as a weighted average of the old model ( $\theta_{t-1}$ ) and the adapted model ( $\hat{\theta}_t$ ):
$\theta_t = (1 - \eta)\theta_{t-1} + \eta\hat{\theta}_t$
where $0 \le \eta \le 1$ . The paper investigates two strategies for $\eta$ :
- Fixed $\eta = 0.5$ : An exponential moving average where the contribution of older models decays exponentially.
- Dynamic $\eta = 1/t$ : An arithmetic mean where all adapted models up to task $t$ contribute equally ( $\theta_t = \frac{1}{t} \sum_{i=1}^t \hat{\theta}_i$ ).

If the model architecture includes shared parameters ( $\theta_S$ ) and task-specific parameters ( $\theta_U$ ), the averaging is applied only to the shared components.

Key Contributions

Memory-Free CL: The method achieves high continual learning performance without requiring a memory buffer of past samples, addressing privacy and storage constraints associated with rehearsal-based methods.
Simplicity and Effectiveness: The approach is computationally simple, requiring only a single forward pass of the old model during the adaptation phase (for LWF) and a parameter averaging step.
Dual-Task Architecture Support: The method is validated on both monolingual (shared model) and multilingual (shared encoder, task-specific decoder) ASR architectures.

Experimental Results
The authors evaluated the method on the Common Voice (CV) dataset using an E2E model consisting of a Conformer encoder and Transformer decoder.

Monolingual Experiments (6 English Dialects):
- The proposed methods (FTA: Fine-Tuning with Averaging; LWFA: LWF with Averaging) significantly outperformed all baselines, including Experience Replay (ER) with 2,000 stored utterances.
- The best configuration ( $\eta = 1/t$ ) achieved an Average Word Error Rate (AVG WER) of 13.72%, outperforming the strongest baseline (ER with 2k samples, 14.20%) by 3.4%.
- Backward Transfer (BWT), a measure of forgetting, was reduced to near zero (-0.1%), compared to -2.5% for naive fine-tuning.
Multilingual Experiments (5 Languages):
- Forgetting was more severe in this setting due to task dissimilarity. Naive fine-tuning resulted in catastrophic forgetting (BWT of -22.2%).
- LWFA with $\eta = 1/t$ achieved the best performance with an AVG WER of 19.54%, outperforming the strong baseline (ER with 2k samples, 23.65%) by 17.4%.
- LWFA reduced the forgetting observed in ER by approximately 88.9% while requiring no memory storage.
- In multilingual settings, the addition of the LWF loss during adaptation provided a significant boost over simple fine-tuning, whereas in monolingual settings, the difference between FTA and LWFA was minimal.

Significance and Claims
The paper claims that weight averaging is a "simple yet effective" solution to catastrophic forgetting in E2E ASR. The authors assert that their method:

Eliminates the need for memory: Unlike previous state-of-the-art CL methods for ASR which relied on storing samples, this method achieves superior performance without any memory buffer.
Achieves near-perfect retention: With $\eta = 1/t$ , the method results in "almost zero forgetting" while still learning new tasks effectively.
Generalizes across architectures: The method is robust across both monolingual (fully shared) and multilingual (partially shared) scenarios.

The authors attribute the success of weight averaging to the theoretical finding that averaging weights leads to "wider optima" in the loss landscape, resulting in better generalization, similar to observations in checkpoint averaging and model soups. They conclude that this approach enables high-performance continual learning in ASR without the resource overhead previously deemed necessary.