🤖 machine learning

Continual Learning for Monolingual End-to-End Automatic Speech Recognition

Original authors: Steven Vander Eeckt, Hugo Van hamme

Published 2026-01-22

📖 5 min read🧠 Deep dive

Original authors: Steven Vander Eeckt, Hugo Van hamme

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant speech-recognition assistant, like a very smart secretary who is an expert at understanding Dutch. You train this secretary perfectly on the "main" Dutch dialect. They are amazing at it.

Now, you want to teach them a new skill: understanding a different regional accent or a new topic. You sit them down and start teaching. But here's the problem: as soon as they learn the new skill, they start forgetting how to do the old one perfectly. This is called Catastrophic Forgetting. It's like if you learned to play a new song on the piano, but in doing so, your fingers forgot the muscle memory for the song you were already a master of.

This paper is about finding a way to teach this speech assistant new things without making them forget the old things, and without needing to store every single recording they ever heard (which would take up a massive amount of space).

The Big Problem: The "All-or-Nothing" Dilemma

The researchers tried two main ways to solve this:

The "Hard Stop" Approach (Fine-Tuning): Just teach the new thing.
- Result: The assistant learns the new thing but forgets the old thing. They become bad at everything overall.
The "Library" Approach (Joint Training): Keep all the old recordings and the new recordings in a giant library, and re-teach the assistant from scratch using everything.
- Result: This works perfectly! The assistant knows everything. But it's impossible in real life because you can't keep terabytes of old data forever, and you might not even have permission to keep it (privacy issues).

The Solution: Continual Learning (The "Smart Study" Methods)

The paper tests many different "study techniques" (Continual Learning methods) to see which one helps the assistant learn new accents while keeping the old ones fresh, using only a tiny bit of old data.

They tested two main types of study techniques:

1. The "Mental Gym" (Regularization Methods)

These methods try to change the assistant's brain gently. They put up "guardrails" that say, "You can learn this new thing, but don't move the parts of your brain that are crucial for the old thing."

The Analogy: Imagine trying to learn a new dance move without unlearning the steps of your favorite song. You try to be careful not to mess up the old steps.
The Result: These methods were mostly disappointing. The "guardrails" were too rigid. The assistant either got stuck and couldn't learn the new accent well, or they learned the new thing but still forgot the old one. It's like trying to walk a tightrope while carrying a heavy backpack; you just can't move fast enough to learn anything new.

2. The "Flashcard" Method (Rehearsal Methods)

These methods use a tiny "memory bank" (a small notebook) containing just a few examples (500 sentences) from the old tasks. Every time the assistant learns something new, they also quickly review these flashcards.

The Analogy: Instead of trying to remember the whole library, you keep a small, curated set of "best hits" from your old training. Before you learn a new song, you quickly hum the old hits to keep them fresh in your mind.
The Result: This worked much better.

The Star Performer: Knowledge Distillation (The "Mentor" System)

Among all the flashcard methods, one stood out as the clear winner. It's called Knowledge Distillation (KD).

How it works: Imagine the assistant has an "Old Self" (the model trained on the first task) and a "New Self" (the model learning the new task). When the New Self learns, the Old Self acts as a mentor. The New Self looks at the Old Self's answers on the flashcards and tries to mimic them while learning the new material.
The Magic: The researchers found that by using this "mentor" system with a tiny notebook (containing only 0.6% of the original data), the assistant could close the gap between "forgetting everything" and "knowing everything" by over 40%.
The Storage Win: Usually, keeping a memory bank gets bigger as you learn more tasks. But the researchers showed you can fix the notebook size to be tiny (0.2% of the data) and still get amazing results. This means you don't need a massive hard drive; a small USB stick is enough.

The Bottom Line

If you want to update a speech recognition system to understand new accents or topics:

Don't just retrain it on new data (it will forget the old stuff).
Don't try to keep every single old recording (it's too expensive and impractical).
Do use a "Flashcard" method where you keep a tiny, fixed-size sample of old data.
Best of all: Use the "Mentor" technique (Knowledge Distillation), where the new model learns by mimicking the old model's behavior on those few flashcards.

This approach allows a speech assistant to keep getting smarter and more versatile over time without losing its original skills, all while using a tiny fraction of the data storage space.

Technical Summary: Continual Learning for Monolingual End-to-End Automatic Speech Recognition

Problem Statement

Automatic Speech Recognition (ASR) models, particularly End-to-End (E2E) architectures, suffer from Catastrophic Forgetting (CF) when adapted to new domains, accents, or dialects. Even within a single language, shifting data distributions between an original task and a new task causes the model to degrade in performance on the original task. Standard fine-tuning exacerbates this issue. While Continual Learning (CL) methods have been explored in image classification, their application to E2E ASR remains under-explored. Existing ASR-specific CL studies often focus on HMM-based acoustic models, pre-trained models like wav2vec2, or specific scenarios where subsequent tasks are significantly smaller than the initial one. There is a need to evaluate a broad spectrum of CL methods on a monolingual E2E Hybrid CTC-Transformer model without relying on access to previous tasks' validation sets for hyper-parameter optimization.

Methodology

Model Architecture and Setup

The study utilizes a Hybrid CTC/Transformer model (following [21]) trained on the Corpus Gesproken Nederlands (CGN) dataset. The model employs a combined loss function:
$L(X, y; \theta) = c \cdot L_c(X, y; \theta) + (1-c) \cdot L_d(X, y; \theta)$
where $L_c$ is the CTC loss, $L_d$ is the Decoder Cross-Entropy loss, and $c=0.3$ . The vocabulary consists of 300 word pieces generated by Sentence Piece.

The experimental setup involves four sequential tasks based on dialect and region: NL-main, VL-main, NL-rest, and VL-rest. The goal is to learn these tasks sequentially such that the final model $\theta_T$ minimizes the cumulative loss across all tasks without direct access to previous datasets $D_1, \dots, D_{T-1}$ (except for a small memory buffer in rehearsal-based methods).

Continual Learning Methods Evaluated

The paper implements and compares 11 CL methods categorized into two groups:

Regularization-based Methods: These add a penalty term to the loss function to prevent significant changes to parameters important for previous tasks.
- EWC (Elastic Weight Consolidation): Uses the diagonal of the Fisher Information Matrix to weight L2 regularization.
- MAS (Memory-Aware Synapses): Computes importance weights based on the sensitivity of the output norm to parameter changes.
- CSQN & CSQN-BT: Extends EWC by considering parameter interactions via low-rank Hessian approximations.
- LWF (Learning Without Forgetting): Uses knowledge distillation between the old model (teacher) and new model (student) on the new task's data.
Rehearsal-based Methods: These utilize a small memory buffer of representative samples from previous tasks.
- ER (Experience Replay): Standard mini-batch mixing, with variants ER ( $\lambda$ ) (weighted loss) and BER (Batch-level ER) (merged training set).
- A-GEM (Average-Gradient Episodic Memory): Projects gradients to ensure they do not negatively interfere with gradients from the memory buffer.
- KD (Knowledge Distillation): Applies LWF-style distillation loss on samples drawn from the memory rather than the new task data.

Hyper-parameter Determination

A novel, efficient method is proposed to determine the regularization weight $\lambda$ without requiring validation sets from previous tasks. The method iteratively adjusts $\lambda$ based on the Token Error Rate (TER) improvement on the current task's validation set, aiming to close the gap between the unregularized fine-tuning performance and the initial model performance.

Memory Configuration

For rehearsal-based methods, 500 utterances are sampled per task from the training set, filtered to ensure meaningful sentence length. In a secondary experiment, a fixed memory size of 500 utterances total is tested to simulate constraints where memory cannot grow with the number of tasks.

Key Results

Performance Metrics

The study evaluates methods using Average Word Error Rate (AWER), Backward Transfer (BWT), Forward Transfer (FWT), and Coverage (COV). COV measures how well a method closes the gap between the Fine-Tuning (FT) lower bound and the Continued Joint Training (CJT) upper bound.

Regularization-based Performance: Methods like EWC, MAS, and CSQN generally performed worse than or comparable to standard Fine-Tuning (FT). They successfully reduced forgetting (BWT) but failed to learn new tasks effectively (low FWT), likely because the tasks were too similar, causing a conflict in which parameters to update. LWF was the strongest regularization-based method, significantly outperforming FT in FWT but achieving only a 12.4% COV.
Rehearsal-based Performance: Methods utilizing memory significantly outperformed regularization-only approaches.
- KD (Knowledge Distillation) emerged as the top performer, achieving a COV of 41.7% (increasing to 38.3% with fixed memory). It reduced the forgetting of FT by over 70% while learning new tasks as well as FT.
- ER ( $\lambda$ ) and A-GEM also showed strong performance (COV of 27.2% and 22.0%, respectively).
- Standard ER performed poorly, suffering from overfitting to the memory buffer.

Storage Efficiency

Rehearsal methods require storing the model plus the memory buffer. With a growing memory (500 utterances per task), this equates to ~3.24 models. With a fixed memory (500 utterances total), storage drops to 1.72 models, independent of the number of tasks.
Regularization methods are generally more storage-efficient (storing only the model and potentially importance weights), but LWF was the only one to improve upon the FT baseline.
Joint Training (JT/CJT) requires storing all historical data, equivalent to ~260 models, rendering it impractical for continual adaptation.

Fixed Memory Experiment

When the memory size was fixed at 500 utterances (0.2% of the original data after four tasks), the performance of KD, ER ( $\lambda$ ), and A-GEM remained robust, with negligible deterioration compared to the growing memory setup. This demonstrates that high efficacy can be maintained with a very small, constant memory footprint.

Significance and Claims

The paper claims that Continual Learning is a viable solution for extending monolingual E2E ASR models to new dialects and domains without catastrophic forgetting.

Superiority of Rehearsal: The study concludes that rehearsal-based methods are currently the most practical and effective approach for E2E ASR. Specifically, Knowledge Distillation (KD) on a small memory buffer closes the performance gap between fine-tuning and joint training by over 40%, while requiring access to only 0.6% of the original data (or 0.2% with fixed memory).
Limitations of Regularization: Regularization-based methods, particularly those estimating parameter importance (EWC, MAS), struggle when tasks are highly similar, as they cannot distinguish between parameters that need updating for the new task and those that must be preserved for the old task.
Practicality: The proposed method for determining hyper-parameters ( $\lambda$ ) allows for realistic deployment without needing previous validation sets. Furthermore, the ability to maintain high performance with a fixed, small memory addresses storage and privacy constraints, making these methods feasible for real-world deployment where storing all past data is impossible.

In summary, the paper establishes that with a small, fixed memory buffer, CL methods (especially KD) can effectively mitigate catastrophic forgetting in monolingual E2E ASR, offering a path toward models that can be continually enhanced without retraining from scratch or storing massive historical datasets.